Jobsub ID 108701.1@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108701.1@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 03:03:34
Site	UK_Durham
Entry	DUNE_UK_SGridDurham_ce3
Last heartbeat	2024-11-23 04:14:18
From worker node	Hostname	n260.dur.scotgrid.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 03:13:26
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50538503_834_20231201T175945Z_gen_g4_detsim_hitreco__20240507T191849Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (3626s)
	CPU time	53m (3211s = 88%)
Outputting started	2024-11-23 04:13:53
Output files
Finished	2024-11-23 04:14:18
List job events Wrapper job log
Jobscript log (last 10,000 characters)

: Alg0001, LArPreProcessing
> Running Algorithm: Alg0002, LArDLMaster
> Running Algorithm: Alg0003, LArNeutrinoEventValidation
---RAW-MATCHING-OUTPUT--------------------------------------------------------------------------
Failure in algorithm Alg0003, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  23-Nov-2024 04:13:48 GMT run: 50538503 subRun: 1 event: 83500
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  23-Nov-2024 04:13:48 GMT run: 50538503 subRun: 1 event: 83500
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtrack@BeginModule  23-Nov-2024 04:13:48 GMT run: 50538503 subRun: 1 event: 83500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 04:13:48 GMT run: 50538503 subRun: 1 event: 83500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 04:13:48 GMT run: 50538503 subRun: 1 event: 83500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 04:13:48 GMT run: 50538503 subRun: 1 event: 83500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 04:13:48 GMT run: 50538503 subRun: 1 event: 83500
Comparing two wires in the same plane: return failure
%MSG
407
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1071!
23-Nov-2024 04:13:52 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50538503_834_20231201T175945Z_gen_g4_detsim_hitreco__20240507T191849Z_reco2_reco_data_2024-11-23T_031331Z.root"
23-Nov-2024 04:13:52 GMT  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/f2/56/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50538503_834_20231201T175945Z_gen_g4_detsim_hitreco__20240507T191849Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.11986       32.1803       572.26        9.73339       80.7851        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0179431     0.0407366     0.0740305     0.0377895     0.0135268       100    
reco:linecluster:LineCluster                           0.101388      0.514426       19.6873      0.195657       1.9981         100    
reco:trajcluster:TrajCluster                           0.0508107      1.02866       37.8422      0.0619796      4.88828        100    
reco:pandora:StandardPandora                           0.0648489      2.40076       63.7067       1.23099       7.01226        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000117401   0.00706885     0.149608     0.00167955     0.0199886       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000153649    0.0148237     0.306389     0.00319412     0.0401344       100    
reco:pandoracalo:Calorimetry                          9.1352e-05     0.0246427     0.915163     0.00117689     0.103562        100    
reco:pandorapid:Chi2ParticleID                        3.4826e-05    0.000188517   0.00258604    0.000141412   0.00026923       100    
reco:trkshowersplit:TrackShowerHits                   8.8277e-05     0.028186      0.841704     0.000823553     0.11303        100    
reco:pmtrack:PMAlgTrackMaker                          0.000958096     2.97669       84.7865      0.252263       10.2656        100    
reco:pmtrackcalo:Calorimetry                          5.9833e-05     0.0151332     0.755406      0.0010387     0.0778533       100    
reco:pmtrackpid:Chi2ParticleID                        2.5718e-05    0.000317238   0.00477326    0.000174679   0.000547421      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000154471     3.37362       127.555      0.371042       14.0755        100    
reco:pmtrajfitcalo:Calorimetry                        5.7658e-05     0.0254302     0.920426     0.00190391     0.104513        100    
reco:pmtrajfitpid:Chi2ParticleID                      2.3735e-05    0.000218763   0.00128027    0.000174839   0.000181569      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000780942     3.40039       96.0637      0.262573       11.6052        100    
reco:pmtracktccalo:Calorimetry                        5.7569e-05    0.00851779     0.323063     0.00137624     0.0342291       100    
reco:pmtracktcpid:Chi2ParticleID                      2.3795e-05    0.000304656   0.00354852    0.000191059   0.000457482      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000132499     2.06726       57.4275      0.369531       6.73493        100    
reco:pmtrajfittccalo:Calorimetry                      5.6496e-05     0.015133      0.419945     0.00145728     0.0529223       100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.3294e-05    0.000382241   0.00322091    0.000200257   0.000552053      100    
reco:blurredcluster:BlurredClustering                 8.6042e-05      1.04959       8.3338        0.43507       1.43838        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000129594      11.81        159.972       3.21438       25.113         100    
reco:emshower:EMShower                                0.000202671    0.0333123     0.507113     0.00442374     0.0678674       100    
reco:cvnmap:CVNMapper                                  2.109e-05     0.0113716     0.0442213     0.0076066     0.0104428       100    
reco:cvneva:CVNEvaluator                              1.6911e-05     0.603244      0.960565      0.815966       0.37094        100    
reco:energyrecnumu:EnergyReco                         0.000427565    0.0438888      1.63763     0.00423544     0.205447        100    
reco:energyrecnue:EnergyReco                          0.00013832    0.00249365     0.0606486    0.000637757   0.00731695       100    
reco:energyrecnc:EnergyReco                           0.00157566    0.00484008     0.0639379    0.00309741     0.0075123       100    
reco:energyrecnumurange:EnergyReco                    0.000175341    0.0421318      1.64654     0.00285713     0.206925        100    
reco:energyrecnumumcs:EnergyReco                      0.00136174     0.0435117      1.61734     0.00343717     0.203953        100    
reco:opdec:Deconvolution                               0.0846877     0.298897      0.506965      0.302871      0.0985912       100    
reco:ophitspe:OpHitFinderDeco                          0.318785       2.03474       4.38424       2.00451      0.949287        100    
reco:opflash:OpFlashFinder                            0.000295266   0.00162471     0.0084666    0.00142429    0.00111152       100    
reco:rns:RandomNumberSaver                            1.7764e-05    3.94907e-05   0.00048776    3.37835e-05    4.65e-05        100    
reco:anglereconue:NuAngularReco                       0.000143681   0.00207284     0.0564284    0.000416565   0.00697571       100    
reco:anglereconumu:NuAngularReco                       6.349e-05    0.00181291     0.0538643    0.000272598   0.00665755       100    
reco:anglereconuepfps:NuAngularReco                   0.00010636    0.00199985     0.0536282    0.000389273   0.00652492       100    
reco:anglereconumupfps:NuAngularReco                  8.8536e-05    0.00187914     0.0555184    0.000330182   0.00665188       100    
[art]:TriggerResults:TriggerResultInserter             8.616e-06    1.43118e-05   6.4762e-05    1.38265e-05   5.86339e-06      100    
end_path:out1:RootOutput                               2.575e-06    6.27701e-06   2.7252e-05    4.9095e-06    3.94828e-06      100    
end_path:out1:RootOutput(write)                        0.0179156     0.247865       1.16542      0.255183      0.181263        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 38597.2 MB
  Peak resident set size usage (VmHWM): 2535.68 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50538503_834_20231201T175945Z_gen_g4_detsim_hitreco__20240507T191849Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50538503_834_20231201T175945Z_gen_g4_detsim_hitreco__20240507T191849Z_reco2_reco_2024-11-23T_031331Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50538503_834_20231201T175945Z_gen_g4_detsim_hitreco__20240507T191849Z_reco2_reco_data_2024-11-23T_031331Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work