Jobsub ID 108651.12@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108651.12@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 00:59:09
Site	US_UChicago
Entry	Engage_US_MWT2_uiuc_gk02_condce_mcore
Last heartbeat	2024-11-23 02:47:50
From worker node	Hostname	iut2-c398.iu.edu
	cpuinfo	AMD EPYC 7443 24-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	86400 (24 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 01:14:53
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65840069_419_20231122T052552Z_gen_g4_detsim_hitreco__20240507T171304Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (5561s)
	CPU time	1h (5200s = 93%)
Outputting started	2024-11-23 02:47:35
Output files
Finished	2024-11-23 02:47:50
List job events Wrapper job log
Jobscript log (last 10,000 characters)

e
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  22-Nov-2024 21:47:27 EST run: 65840069 subRun: 1 event: 84000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  22-Nov-2024 21:47:27 EST run: 65840069 subRun: 1 event: 84000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  22-Nov-2024 21:47:27 EST run: 65840069 subRun: 1 event: 84000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e pma::Track3D:  PMAlgTrackMaker:pmtracktc@BeginModule  22-Nov-2024 21:47:27 EST run: 65840069 subRun: 1 event: 84000
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrackMaker:pmtracktc@BeginModule  22-Nov-2024 21:47:27 EST run: 65840069 subRun: 1 event: 84000
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrackMaker:pmtracktc@BeginModule  22-Nov-2024 21:47:27 EST run: 65840069 subRun: 1 event: 84000
Hit sorting problem.
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 21:47:28 EST run: 65840069 subRun: 1 event: 84000
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 81, 81, 46
minwire 0: 1332
minwire 1: 1113
minwire 2: 1572
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.00940256, 0.000556646, 0.990041, 
Output 1: 0.934491, 0.0650069, 0.000448789, 5.30008e-05, 
Output 2: 0.988986, 0.0110143, 

378
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1028!
22-Nov-2024 21:47:33 EST  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65840069_419_20231122T052552Z_gen_g4_detsim_hitreco__20240507T171304Z_reco2_reco_data_2024-11-23T_011458Z.root"
22-Nov-2024 21:47:34 EST  Closed input file "root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/fardet-hd/b3/b1/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65840069_419_20231122T052552Z_gen_g4_detsim_hitreco__20240507T171304Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.861549       25.0776       577.137       10.3113       54.7154        200    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0277773     0.135065      0.783096      0.110546      0.101464        200    
reco:linecluster:LineCluster                           0.236758      0.403747       4.6942       0.275285      0.554645        200    
reco:trajcluster:TrajCluster                           0.0771441     0.618138       29.288       0.0899905      2.57035        200    
reco:pandora:StandardPandora                           0.0912592      2.13433       95.5015       1.22732       6.8381         200    
reco:pandoraTrack:LArPandoraTrackCreation             0.000137028    0.0063267     0.325109     0.00164334     0.0243984       200    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000173777    0.0109526     0.341918     0.00333584     0.0279069       200    
reco:pandoracalo:Calorimetry                          0.000111058    0.0179464      1.07387      0.001152      0.0861856       200    
reco:pandorapid:Chi2ParticleID                        3.5977e-05    0.000177771   0.00136102    0.000135149   0.000154059      200    
reco:trkshowersplit:TrackShowerHits                   9.8215e-05      0.02056       1.23227     0.000935108    0.0983653       200    
reco:pmtrack:PMAlgTrackMaker                          0.000995038     1.95724       60.2823      0.206889       6.1262         200    
reco:pmtrackcalo:Calorimetry                          6.8089e-05    0.00843068      0.4551      0.000974343    0.0359081       200    
reco:pmtrackpid:Chi2ParticleID                        2.5368e-05    0.000279068   0.00419887    0.000154154   0.000435609      200    
reco:pmtrajfit:PMAlgTrajFitter                        0.000163347     2.04736       74.8074      0.289554       7.05994        200    
reco:pmtrajfitcalo:Calorimetry                        6.0334e-05     0.0191594      1.12418     0.00158242     0.0904787       200    
reco:pmtrajfitpid:Chi2ParticleID                      2.2923e-05    0.000198223   0.00196619    0.000164955   0.000181413      200    
reco:pmtracktc:PMAlgTrackMaker                        0.000754829     2.54995       78.7662      0.267224       8.42047        200    
reco:pmtracktccalo:Calorimetry                        6.8509e-05     0.0060124     0.288695     0.000915211    0.022764        200    
reco:pmtracktcpid:Chi2ParticleID                      2.5658e-05    0.000267796   0.00357723    0.000169063   0.000370378      200    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000129704     1.37453       36.648       0.289001       3.7048         200    
reco:pmtrajfittccalo:Calorimetry                      5.5965e-05     0.0124799     0.479173      0.0012848     0.045574        200    
reco:pmtrajfittcpid:Chi2ParticleID                    2.1531e-05    0.000350441   0.00359178    0.000175691   0.000536388      200    
reco:blurredcluster:BlurredClustering                 0.00181895     0.963637       6.51847      0.566167       1.16971        200    
reco:emtrkmichelid:EmTrackMichelId                     0.0389387      9.26839       176.998       2.99514       19.3798        200    
reco:emshower:EMShower                                0.000235904    0.0252307     0.880389     0.00376837     0.0731767       200    
reco:cvnmap:CVNMapper                                 1.9186e-05    0.00779143     0.0350311    0.00632655     0.0066394       200    
reco:cvneva:CVNEvaluator                              1.5609e-05     0.693831       1.27703      0.883785      0.424926        200    
reco:energyrecnumu:EnergyReco                         0.000327196    0.0252158     0.987149      0.005385      0.0851706       200    
reco:energyrecnue:EnergyReco                          0.000113173    0.0021012     0.0993606    0.000588837   0.00759085       200    
reco:energyrecnc:EnergyReco                           0.00143838     0.0044075     0.108498     0.00304125    0.00814792       200    
reco:energyrecnumurange:EnergyReco                    0.000162506    0.0225542     0.988504     0.00277155     0.0849127       200    
reco:energyrecnumumcs:EnergyReco                      0.00145359     0.0245352     0.981438     0.00342456     0.0856511       200    
reco:opdec:Deconvolution                               0.0843108     0.390008       1.12907       0.33417      0.179261        200    
reco:ophitspe:OpHitFinderDeco                          0.264859       2.08095       4.16411       2.05055      0.798215        200    
reco:opflash:OpFlashFinder                            0.000190388   0.00141592    0.00336694    0.00137829    0.000574088      200    
reco:rns:RandomNumberSaver                            1.8815e-05    4.22719e-05   0.000298741   4.1143e-05    2.26674e-05      200    
reco:anglereconue:NuAngularReco                       0.000152938   0.00154454     0.0931265    0.000421478   0.00682486       200    
reco:anglereconumu:NuAngularReco                      6.4081e-05    0.00135532     0.0906656    0.000247972   0.00671451       200    
reco:anglereconuepfps:NuAngularReco                   0.000104416   0.00149264     0.0907475    0.000387513   0.00672558       200    
reco:anglereconumupfps:NuAngularReco                  9.3445e-05    0.00143452     0.090491     0.00031396    0.00672299       200    
[art]:TriggerResults:TriggerResultInserter             8.255e-06    1.36062e-05   3.4174e-05    1.29245e-05   3.98373e-06      200    
end_path:out1:RootOutput                               2.885e-06    7.18925e-06   2.4216e-05     7.73e-06     2.94385e-06      200    
end_path:out1:RootOutput(write)                        0.0125069     0.236111       1.31265      0.243129      0.156517        200    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 17317.9 MB
  Peak resident set size usage (VmHWM): 2374.15 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65840069_419_20231122T052552Z_gen_g4_detsim_hitreco__20240507T171304Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65840069_419_20231122T052552Z_gen_g4_detsim_hitreco__20240507T171304Z_reco2_reco_2024-11-23T_011458Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65840069_419_20231122T052552Z_gen_g4_detsim_hitreco__20240507T171304Z_reco2_reco_data_2024-11-23T_011458Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work