Jobsub ID 301679.164@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301679.164@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 03:41:13
Site	UK_Durham
Entry	DUNE_UK_SGridDurham_ce4
Last heartbeat	2024-11-23 04:38:19
From worker node	Hostname	n260.dur.scotgrid.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 03:43:34
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406500_837_20231202T130835Z_gen_g4_detsim_hitreco__20240507T222511Z_reco2.root
Jobscript	Exit code	0
	Real time	54m (3267s)
	CPU time	47m (2851s = 87%)
Outputting started	2024-11-23 04:38:02
Output files
Finished	2024-11-23 04:38:19
List job events Wrapper job log
Jobscript log (last 10,000 characters)

06500 subRun: 1 event: 83800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:37:12 GMT run: 6406500 subRun: 1 event: 83800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:37:12 GMT run: 6406500 subRun: 1 event: 83800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:37:12 GMT run: 6406500 subRun: 1 event: 83800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:37:12 GMT run: 6406500 subRun: 1 event: 83800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:37:12 GMT run: 6406500 subRun: 1 event: 83800
1st wire C:0 T:18 P:2 W:549 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:37:12 GMT run: 6406500 subRun: 1 event: 83800
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 1903, 863, 1484
minwire 0: 1300
minwire 1: 865
minwire 2: 1562
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0191891, 0.0237616, 0.957049, 
Output 1: 0.372964, 0.468778, 0.12812, 0.0301368, 
Output 2: 0.0775565, 0.922444, 

659
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2456!
23-Nov-2024 04:38:00 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406500_837_20231202T130835Z_gen_g4_detsim_hitreco__20240507T222511Z_reco2_reco_data_2024-11-23T_034342Z.root"
23-Nov-2024 04:38:00 GMT  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/9f/cc/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406500_837_20231202T130835Z_gen_g4_detsim_hitreco__20240507T222511Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.97821       28.351        497.903       10.3241       65.7657        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0174196     0.0450932     0.0996004     0.0476177     0.0162018       100    
reco:linecluster:LineCluster                           0.172895      0.434524       14.5305      0.202757       1.47556        100    
reco:trajcluster:TrajCluster                           0.0504578     0.878066       47.5722      0.0608889      4.95397        100    
reco:pandora:StandardPandora                           0.927538       2.68937       54.1611       1.39672       7.32834        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000134152    0.0058616     0.0995281    0.00171966     0.0142692       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000187162    0.0122637     0.220062      0.0033027     0.0273278       100    
reco:pandoracalo:Calorimetry                          9.9617e-05     0.0194122     0.615067      0.0011571     0.0803848       100    
reco:pandorapid:Chi2ParticleID                        3.9334e-05    0.000194285   0.00200005    0.000153935   0.00021536       100    
reco:trkshowersplit:TrackShowerHits                   0.000100549    0.0294094      1.32624     0.000840057     0.14482        100    
reco:pmtrack:PMAlgTrackMaker                          0.00106477      2.63997       79.4353      0.229818       10.1307        100    
reco:pmtrackcalo:Calorimetry                          6.9952e-05     0.0123537     0.374502     0.00102239     0.0508116       100    
reco:pmtrackpid:Chi2ParticleID                        2.9676e-05    0.000346212   0.00414343    0.000167366   0.000602263      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000186872     2.03877       44.9663      0.250525       5.88394        100    
reco:pmtrajfitcalo:Calorimetry                        6.0965e-05     0.0186308     0.635702      0.001629      0.0756093       100    
reco:pmtrajfitpid:Chi2ParticleID                      2.7141e-05    0.000218675   0.000989926   0.000177254   0.000162391      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000907991     2.58071       72.039       0.232227       9.31545        100    
reco:pmtracktccalo:Calorimetry                        6.6886e-05    0.00576996     0.116217     0.00115164     0.0152438       100    
reco:pmtracktcpid:Chi2ParticleID                      2.8724e-05    0.000286451   0.00223121    0.000182418   0.000338997      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.00016532      1.57774       23.8398       0.33898       3.82712        100    
reco:pmtrajfittccalo:Calorimetry                      6.1666e-05     0.0149516     0.368775     0.00144625     0.052823        100    
reco:pmtrajfittcpid:Chi2ParticleID                     2.638e-05    0.000418744   0.00425349    0.00019099    0.000689734      100    
reco:blurredcluster:BlurredClustering                 0.00520774     0.972005       6.09446      0.437946       1.29815        100    
reco:emtrkmichelid:EmTrackMichelId                     0.198687       10.8572       144.727       3.28475       22.7464        100    
reco:emshower:EMShower                                0.000274346    0.028954      0.390922     0.00511999     0.0538185       100    
reco:cvnmap:CVNMapper                                 2.9605e-05    0.00961286     0.0544479    0.00752351    0.00921545       100    
reco:cvneva:CVNEvaluator                              1.7553e-05      0.62741       1.23034      0.730256      0.381891        100    
reco:energyrecnumu:EnergyReco                         0.00037748     0.0353619     0.854926     0.00448466     0.111199        100    
reco:energyrecnue:EnergyReco                          0.000148569    0.0018544     0.0290578    0.000595631   0.00400601       100    
reco:energyrecnc:EnergyReco                            0.0018166    0.00434501     0.0334264    0.00346055    0.00436813       100    
reco:energyrecnumurange:EnergyReco                    0.000244209    0.0331598     0.850025     0.00280483     0.110476        100    
reco:energyrecnumumcs:EnergyReco                      0.000923006    0.034595      0.844075     0.00381386     0.109602        100    
reco:opdec:Deconvolution                               0.0813058     0.313876      0.597541      0.314878      0.105372        100    
reco:ophitspe:OpHitFinderDeco                          0.246891       2.15701       4.81954       2.00218       1.09742        100    
reco:opflash:OpFlashFinder                            0.000298071   0.00154752    0.00542741    0.00144669    0.00079222       100    
reco:rns:RandomNumberSaver                            2.3133e-05    4.15253e-05   0.000389394   3.45845e-05   3.71938e-05      100    
reco:anglereconue:NuAngularReco                       0.000168206   0.00134228     0.0229397    0.000421038    0.0031161       100    
reco:anglereconumu:NuAngularReco                      8.2546e-05    0.00114185     0.0227364    0.000269509   0.00308408       100    
reco:anglereconuepfps:NuAngularReco                   0.000128351   0.00129107     0.0227476    0.000398385   0.00307188       100    
reco:anglereconumupfps:NuAngularReco                  0.000105859   0.00121248     0.0228903    0.000333869   0.00309047       100    
[art]:TriggerResults:TriggerResultInserter             9.168e-06    1.56569e-05   9.3576e-05    1.39165e-05   8.77254e-06      100    
end_path:out1:RootOutput                               3.266e-06    7.05515e-06   2.5889e-05    5.5455e-06    3.92247e-06      100    
end_path:out1:RootOutput(write)                        0.0164019     0.262332       1.17845      0.264885       0.19081        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 38194.3 MB
  Peak resident set size usage (VmHWM): 2143.52 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406500_837_20231202T130835Z_gen_g4_detsim_hitreco__20240507T222511Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406500_837_20231202T130835Z_gen_g4_detsim_hitreco__20240507T222511Z_reco2_reco_2024-11-23T_034342Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406500_837_20231202T130835Z_gen_g4_detsim_hitreco__20240507T222511Z_reco2_reco_data_2024-11-23T_034342Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work