Jobsub ID 301520.191@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301520.191@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-22 23:23:47
Site	CA_SFU
Entry	DUNE_CA_SFU_lcg-ce3
Last heartbeat	2024-11-23 00:42:28
From worker node	Hostname	cdr232.int.cedar.computecanada.ca
	cpuinfo	Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	84598 (23 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-22 23:34:07
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419113_325_20231203T145123Z_gen_g4_detsim_hitreco__20240508T053904Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (4087s)
	CPU time	54m (3260s = 79%)
Outputting started	2024-11-23 00:42:14
Output files
Finished	2024-11-23 00:42:28
List job events Wrapper job log
Jobscript log (last 10,000 characters)

IntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 16:41:47 PST run: 6419113 subRun: 1 event: 32600
1st wire C:0 T:16 P:2 W:1099 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 16:41:51 PST run: 6419113 subRun: 1 event: 32600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 16:41:51 PST run: 6419113 subRun: 1 event: 32600
1st wire C:0 T:16 P:2 W:920 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 16:41:51 PST run: 6419113 subRun: 1 event: 32600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 16:41:51 PST run: 6419113 subRun: 1 event: 32600
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 16:41:51 PST run: 6419113 subRun: 1 event: 32600
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 16:41:51 PST run: 6419113 subRun: 1 event: 32600
TuneFullTree failed.
%MSG
Boundary wire vector sizes: 734, 693, 83
minwire 0: 2077
minwire 1: 608
minwire 2: 2317
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0015788, 0.00066927, 0.997752, 
Output 1: 0.0304597, 0.94663, 0.0220226, 0.000887785, 
Output 2: 0.996587, 0.003413, 

538
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2012!
22-Nov-2024 16:42:13 PST  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419113_325_20231203T145123Z_gen_g4_detsim_hitreco__20240508T053904Z_reco2_reco_data_2024-11-22T_233411Z.root"
22-Nov-2024 16:42:13 PST  Closed input file "root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/fardet-hd/53/88/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419113_325_20231203T145123Z_gen_g4_detsim_hitreco__20240508T053904Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.24522       34.1412       921.021       12.9717       95.4472        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0736287     0.157353      0.266585      0.148441      0.0423944       100    
reco:linecluster:LineCluster                           0.434308       1.34106       62.0121      0.702638       6.0985         100    
reco:trajcluster:TrajCluster                           0.217782       1.7725        122.054       0.2303        12.1359        100    
reco:pandora:StandardPandora                           0.234582       2.7231        60.217        1.83727       5.85473        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000175435    0.0062292     0.0888584    0.00161241     0.0137522       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000242406    0.0148334     0.194493      0.0030689     0.0324953       100    
reco:pandoracalo:Calorimetry                          0.000136478    0.0297029      1.97396     0.00107762     0.197076        100    
reco:pandorapid:Chi2ParticleID                        5.4435e-05    0.000187565    0.001641     0.000148495   0.000183683      100    
reco:trkshowersplit:TrackShowerHits                   9.1429e-05     0.0254039      1.47592     0.000727713    0.149324        100    
reco:pmtrack:PMAlgTrackMaker                          0.00171544      2.74497       135.497      0.177192       13.6989        100    
reco:pmtrackcalo:Calorimetry                          0.000117532    0.0124998     0.511585     0.000872731    0.0535114       100    
reco:pmtrackpid:Chi2ParticleID                         4.797e-05    0.00030802    0.00444367    0.000163971   0.000528501      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000278719     3.03702       149.861       0.2793        15.0645        100    
reco:pmtrajfitcalo:Calorimetry                        0.000106811    0.0302519      1.95712     0.00140811     0.195495        100    
reco:pmtrajfitpid:Chi2ParticleID                      4.6124e-05    0.000203264   0.000811133   0.000172587   0.000145103      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00154868      2.67958       115.681      0.191576       11.8988        100    
reco:pmtracktccalo:Calorimetry                         0.000116     0.00754692     0.126077     0.000877103     0.01863        100    
reco:pmtracktcpid:Chi2ParticleID                      4.6794e-05    0.000279366   0.00221991    0.000165717   0.000347085      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000258693     1.65649       45.7502      0.312477       5.00599        100    
reco:pmtrajfittccalo:Calorimetry                      0.000109618    0.0175778     0.789471     0.00119058     0.0815464       100    
reco:pmtrajfittcpid:Chi2ParticleID                     4.706e-05    0.000364257   0.00418741    0.000184989   0.000569371      100    
reco:blurredcluster:BlurredClustering                 9.2349e-05      1.32385       10.2207       0.60555       1.8586         100    
reco:emtrkmichelid:EmTrackMichelId                    0.000157595     12.2894       204.246       3.31568       26.4377        100    
reco:emshower:EMShower                                0.000240958    0.0308198     0.226991     0.00552071     0.0477225       100    
reco:cvnmap:CVNMapper                                 2.8254e-05     0.0130292     0.0533604     0.0126545     0.0118542       100    
reco:cvneva:CVNEvaluator                              2.3076e-05     0.739171       2.69884       1.01007      0.534587        100    
reco:energyrecnumu:EnergyReco                         0.000350112    0.0568787      1.30339      0.003935      0.180036        100    
reco:energyrecnue:EnergyReco                          0.000224427   0.00230028     0.0269718    0.000672673   0.00438729       100    
reco:energyrecnc:EnergyReco                           0.00297016    0.00487806     0.0300959    0.00336356    0.00447721       100    
reco:energyrecnumurange:EnergyReco                    0.000226167    0.0548139      1.30185     0.00316773     0.179405        100    
reco:energyrecnumumcs:EnergyReco                      0.00251105     0.0567119      1.30568     0.00402686     0.179429        100    
reco:opdec:Deconvolution                               0.292875      0.756642       1.16321      0.744717      0.186216        100    
reco:ophitspe:OpHitFinderDeco                          0.652923       2.25982       3.83805       2.24958      0.659603        100    
reco:opflash:OpFlashFinder                            0.000162323   0.00163918    0.00510063    0.00157894    0.000741933      100    
reco:rns:RandomNumberSaver                            2.1253e-05    3.6326e-05    0.000346982   3.07435e-05   3.29372e-05      100    
reco:anglereconue:NuAngularReco                       0.000174265   0.00148633     0.0188108    0.000372143   0.00340594       100    
reco:anglereconumu:NuAngularReco                      0.00011327    0.00135697     0.0180175    0.000266771   0.00337614       100    
reco:anglereconuepfps:NuAngularReco                   0.000172552   0.00149459     0.0179366    0.000407683   0.00334808       100    
reco:anglereconumupfps:NuAngularReco                  0.000168604    0.001452      0.0183944    0.00036794    0.00337321       100    
[art]:TriggerResults:TriggerResultInserter            1.3758e-05    1.61501e-05   5.4548e-05    1.4922e-05    4.49039e-06      100    
end_path:out1:RootOutput                               2.842e-06    6.45868e-06   1.9613e-05    6.0815e-06    3.89533e-06      100    
end_path:out1:RootOutput(write)                        0.0109559     0.285834       1.32168       0.3039       0.199987        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8238.28 MB
  Peak resident set size usage (VmHWM): 2270.76 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419113_325_20231203T145123Z_gen_g4_detsim_hitreco__20240508T053904Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419113_325_20231203T145123Z_gen_g4_detsim_hitreco__20240508T053904Z_reco2_reco_2024-11-22T_233411Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419113_325_20231203T145123Z_gen_g4_detsim_hitreco__20240508T053904Z_reco2_reco_data_2024-11-22T_233411Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work