Jobsub ID 108725.18@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108725.18@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 03:51:13
Site	UK_Durham
Entry	DUNE_UK_SGridDurham_ce4
Last heartbeat	2024-11-23 06:27:59
From worker node	Hostname	n257.dur.scotgrid.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:02:10
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6210996_70_20231120T135600Z_gen_g4_detsim_hitreco__20240507T181501Z_reco2.root
Jobscript	Exit code	0
	Real time	2h (8732s)
	CPU time	2h (7801s = 89%)
Outputting started	2024-11-23 06:27:43
Output files
Finished	2024-11-23 06:27:59
List job events Wrapper job log
Jobscript log (last 10,000 characters)

itter:pmtrajfittc@BeginModule  23-Nov-2024 06:27:15 GMT run: 6210996 subRun: 1 event: 14200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:27:15 GMT run: 6210996 subRun: 1 event: 14200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:27:16 GMT run: 6210996 subRun: 1 event: 14200
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:27:16 GMT run: 6210996 subRun: 1 event: 14200
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:27:16 GMT run: 6210996 subRun: 1 event: 14200
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:27:16 GMT run: 6210996 subRun: 1 event: 14200
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:27:16 GMT run: 6210996 subRun: 1 event: 14200
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:27:16 GMT run: 6210996 subRun: 1 event: 14200
TuneFullTree failed.
%MSG
Boundary wire vector sizes: 956, 638, 811
minwire 0: 2365
minwire 1: 648
minwire 2: 2268
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0271353, 0.124335, 0.84853, 
Output 1: 0.162186, 0.613087, 0.210849, 0.0138783, 
Output 2: 0.09516, 0.90484, 

650
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2315!
23-Nov-2024 06:27:42 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6210996_70_20231120T135600Z_gen_g4_detsim_hitreco__20240507T181501Z_reco2_reco_data_2024-11-23T_040216Z.root"
23-Nov-2024 06:27:42 GMT  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/26/05/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6210996_70_20231120T135600Z_gen_g4_detsim_hitreco__20240507T181501Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              0.61755       39.1555       613.89        12.6905       87.8685        200    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0176496     0.0892319     0.206289      0.0758004     0.0528664       200    
reco:linecluster:LineCluster                           0.100276      0.718903       44.0116      0.208443       3.42137        200    
reco:trajcluster:TrajCluster                           0.0502373      1.19783       43.8878      0.0664846      5.09869        200    
reco:pandora:StandardPandora                           0.062049       3.18128       131.938       1.39095       10.6959        200    
reco:pandoraTrack:LArPandoraTrackCreation             0.000134485    0.0085076     0.159701     0.00211638     0.0205699       200    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000189469    0.0184158     0.695621     0.00397383     0.0545151       200    
reco:pandoracalo:Calorimetry                          0.000103938    0.0325517      1.30738     0.00164138     0.124344        200    
reco:pandorapid:Chi2ParticleID                        3.8553e-05    0.00020962    0.00196526    0.000158015   0.000217407      200    
reco:trkshowersplit:TrackShowerHits                   0.000127552    0.0416217      1.11558     0.00122508     0.155876        200    
reco:pmtrack:PMAlgTrackMaker                          0.00121326      4.00732       99.6739      0.333668       13.3949        200    
reco:pmtrackcalo:Calorimetry                          6.9953e-05     0.0162881     0.449547     0.00140506     0.0557518       200    
reco:pmtrackpid:Chi2ParticleID                        2.9195e-05    0.000404973   0.00429133    0.000182662   0.000650887      200    
reco:pmtrajfit:PMAlgTrajFitter                        0.000240177     2.92857       57.8972      0.441571       7.58462        200    
reco:pmtrajfitcalo:Calorimetry                        7.1215e-05     0.0334868      1.31652     0.00217272     0.126136        200    
reco:pmtrajfitpid:Chi2ParticleID                      2.8084e-05    0.000241274   0.00151523    0.000188793   0.000216902      200    
reco:pmtracktc:PMAlgTrackMaker                        0.000903768     4.74464       109.24       0.378839       15.335         200    
reco:pmtracktccalo:Calorimetry                        6.7038e-05     0.0107383     0.239101     0.00157337     0.030011        200    
reco:pmtracktcpid:Chi2ParticleID                      2.8374e-05    0.000376673   0.00313037    0.00020304    0.000518881      200    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000174191     2.45341       48.4161       0.42538       6.3425         200    
reco:pmtrajfittccalo:Calorimetry                      6.5755e-05     0.0237665     0.593825     0.00183781     0.0790661       200    
reco:pmtrajfittcpid:Chi2ParticleID                    2.7743e-05    0.000518437    0.0053048    0.000210234   0.000836747      200    
reco:blurredcluster:BlurredClustering                 9.0512e-05      1.32306       9.14894      0.664306       1.78427        200    
reco:emtrkmichelid:EmTrackMichelId                    0.000132772     14.3955       176.537       4.16677       27.705         200    
reco:emshower:EMShower                                0.000206332    0.0385547     0.577915     0.00859852     0.0747598       200    
reco:cvnmap:CVNMapper                                 2.8955e-05     0.0120174     0.0665076     0.0106581     0.0109742       200    
reco:cvneva:CVNEvaluator                              1.8906e-05     0.702944       1.27776      0.912414      0.406617        200    
reco:energyrecnumu:EnergyReco                         0.000346869    0.0543222      1.17073     0.00607832     0.147977        200    
reco:energyrecnue:EnergyReco                          0.000165474   0.00264516     0.0471728    0.000681751   0.00591338       200    
reco:energyrecnc:EnergyReco                           0.00212091    0.00555865     0.0522053    0.00376992    0.00622644       200    
reco:energyrecnumurange:EnergyReco                    0.000154213    0.0519787      1.16173     0.00350757      0.1479         200    
reco:energyrecnumumcs:EnergyReco                       0.0012974     0.0536085      1.14916     0.00443348     0.147455        200    
reco:opdec:Deconvolution                               0.0886766     0.321823      0.819863      0.328923      0.116666        200    
reco:ophitspe:OpHitFinderDeco                          0.154039       2.37662       4.60005       2.42223       1.01093        200    
reco:opflash:OpFlashFinder                            0.00017367    0.00166648    0.00559503     0.0016778    0.00078086       200    
reco:rns:RandomNumberSaver                            2.1962e-05    4.35318e-05   0.000462338   4.17545e-05   3.21641e-05      200    
reco:anglereconue:NuAngularReco                       0.000183699   0.00199843     0.0421998    0.000476365   0.00487016       200    
reco:anglereconumu:NuAngularReco                      8.6714e-05    0.00174922     0.0404105    0.000295872   0.00472598       200    
reco:anglereconuepfps:NuAngularReco                   0.000125569   0.00189989     0.0410955    0.000429851   0.00479205       200    
reco:anglereconumupfps:NuAngularReco                  0.000108687    0.0018396     0.0417218    0.00036575    0.00486339       200    
[art]:TriggerResults:TriggerResultInserter             9.588e-06    1.49431e-05   5.5776e-05    1.44125e-05   4.23099e-06      200    
end_path:out1:RootOutput                               3.156e-06    7.54783e-06   3.0037e-05     7.81e-06     3.62103e-06      200    
end_path:out1:RootOutput(write)                        0.0117435      0.29689       1.28953      0.290376      0.210188        200    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 38670.4 MB
  Peak resident set size usage (VmHWM): 2614.29 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6210996_70_20231120T135600Z_gen_g4_detsim_hitreco__20240507T181501Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6210996_70_20231120T135600Z_gen_g4_detsim_hitreco__20240507T181501Z_reco2_reco_2024-11-23T_040216Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6210996_70_20231120T135600Z_gen_g4_detsim_hitreco__20240507T181501Z_reco2_reco_data_2024-11-23T_040216Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work