Jobsub ID 301671.129@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301671.129@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 03:19:43
Site	UK_Durham
Entry	DUNE_UK_SGridDurham_ce3
Last heartbeat	2024-11-23 04:10:17
From worker node	Hostname	n239.dur.scotgrid.ac.uk
	cpuinfo	AMD EPYC 7352 24-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 03:21:58
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66027687_441_20231207T003121Z_gen_g4_detsim_hitreco__20240510T061336Z_reco2.root
Jobscript	Exit code	0
	Real time	48m (2882s)
	CPU time	42m (2543s = 88%)
Outputting started	2024-11-23 04:10:00
Output files
Finished	2024-11-23 04:10:17
List job events Wrapper job log
Jobscript log (last 10,000 characters)

4:09:54 GMT
> Running Algorithm: Alg0001, LArPreProcessing
> Running Algorithm: Alg0002, LArDLMaster
> Running Algorithm: Alg0003, LArNeutrinoEventValidation
---RAW-MATCHING-OUTPUT--------------------------------------------------------------------------
Failure in algorithm Alg0003, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
76
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 96!
Begin processing the 100th record. run: 66027687 subRun: 1 event: 44200 at 23-Nov-2024 04:09:56 GMT
> Running Algorithm: Alg0001, LArPreProcessing
> Running Algorithm: Alg0002, LArDLMaster
> Running Algorithm: Alg0003, LArNeutrinoEventValidation
---RAW-MATCHING-OUTPUT--------------------------------------------------------------------------
Failure in algorithm Alg0003, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  23-Nov-2024 04:09:57 GMT run: 66027687 subRun: 1 event: 44200
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  23-Nov-2024 04:09:57 GMT run: 66027687 subRun: 1 event: 44200
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 04:09:57 GMT run: 66027687 subRun: 1 event: 44200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 04:09:57 GMT run: 66027687 subRun: 1 event: 44200
Comparing two wires in the same plane: return failure
%MSG
197
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 363!
23-Nov-2024 04:09:59 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66027687_441_20231207T003121Z_gen_g4_detsim_hitreco__20240510T061336Z_reco2_reco_data_2024-11-23T_032202Z.root"
23-Nov-2024 04:09:59 GMT  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/0b/e0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66027687_441_20231207T003121Z_gen_g4_detsim_hitreco__20240510T061336Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.555031       25.2392       280.923       8.53348       48.583         100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0173185     0.0267055     0.0374312     0.0229885    0.00838574       100    
reco:linecluster:LineCluster                           0.0991459     0.276682       3.0031       0.182444      0.348707        100    
reco:trajcluster:TrajCluster                            0.0498       0.605831       12.7518      0.058282       2.05662        100    
reco:pandora:StandardPandora                           0.0560084      1.84244       16.8859       1.36448       2.31125        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000118853   0.00508809     0.078694     0.00114426     0.011296        100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000173186    0.0143767     0.417745     0.00237958     0.0455376       100    
reco:pandoracalo:Calorimetry                          9.5489e-05     0.0156327     0.424234     0.000834423    0.0543559       100    
reco:pandorapid:Chi2ParticleID                        3.5757e-05    0.000142163   0.00062812    0.000115207   0.000112708      100    
reco:trkshowersplit:TrackShowerHits                   4.8351e-05     0.0139268      0.24363     0.000579243    0.0432668       100    
reco:pmtrack:PMAlgTrackMaker                          0.00121141      2.11734       31.8502      0.187625       5.86509        100    
reco:pmtrackcalo:Calorimetry                          6.2788e-05     0.0116001     0.228865     0.000635258    0.037784        100    
reco:pmtrackpid:Chi2ParticleID                        2.5438e-05    0.00028142    0.00270235    0.000118157   0.000487107      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000166022     1.86471       30.9687      0.221506       4.53569        100    
reco:pmtrajfitcalo:Calorimetry                        5.9983e-05      0.01547      0.296919     0.00143938     0.0477381       100    
reco:pmtrajfitpid:Chi2ParticleID                      2.4466e-05    0.000172576   0.00115075    0.000132169   0.000175786      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000998496     2.58628       47.3293       0.19084       7.4488         100    
reco:pmtracktccalo:Calorimetry                        6.1536e-05    0.00760779     0.137553     0.000729877    0.0215434       100    
reco:pmtracktcpid:Chi2ParticleID                      2.5047e-05    0.000244555   0.00232292    0.000132699   0.000351044      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000145554     1.4451        21.5415      0.233421       3.37564        100    
reco:pmtrajfittccalo:Calorimetry                      5.7148e-05     0.0130624     0.220586      0.0010018     0.0385531       100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.4155e-05    0.000331407   0.00298551    0.000131041   0.000530154      100    
reco:blurredcluster:BlurredClustering                 5.8219e-05     0.927986       6.33549      0.355919       1.25388        100    
reco:emtrkmichelid:EmTrackMichelId                    9.1612e-05      10.7154       108.334       2.30626       21.3896        100    
reco:emshower:EMShower                                0.000138991    0.0242119     0.182082     0.00397696     0.0378119       100    
reco:cvnmap:CVNMapper                                 1.6642e-05     0.0141078      0.11609      0.0136575     0.0157265       100    
reco:cvneva:CVNEvaluator                              1.4186e-05     0.564209       1.14232      0.749504      0.434779        100    
reco:energyrecnumu:EnergyReco                         0.000265048    0.030319      0.781071     0.00361457     0.102239        100    
reco:energyrecnue:EnergyReco                          0.000124404   0.00177588     0.0243381    0.000641636   0.00334818       100    
reco:energyrecnc:EnergyReco                           0.00187198    0.00400091     0.0268371    0.00305798    0.00378347       100    
reco:energyrecnumurange:EnergyReco                    0.000122039    0.0289657     0.772846     0.00222632     0.102091        100    
reco:energyrecnumumcs:EnergyReco                      0.00194882     0.0301122     0.779776     0.00316735     0.102206        100    
reco:opdec:Deconvolution                               0.0608559     0.277373      0.564421      0.269026      0.109707        100    
reco:ophitspe:OpHitFinderDeco                          0.149055       1.51685       3.92104       1.42956      0.826393        100    
reco:opflash:OpFlashFinder                            0.000108033   0.00137691    0.00448187    0.00133443    0.000789076      100    
reco:rns:RandomNumberSaver                            1.8736e-05    5.57847e-05    0.0005056    3.6839e-05    8.81337e-05      100    
reco:anglereconue:NuAngularReco                       0.000123031   0.00123066     0.0215078    0.000388781   0.00277143       100    
reco:anglereconumu:NuAngularReco                      6.4602e-05    0.00100809     0.0205407    0.000169213   0.00266453       100    
reco:anglereconuepfps:NuAngularReco                   0.000110778   0.00118624     0.021633     0.000357887   0.00276375       100    
reco:anglereconumupfps:NuAngularReco                  9.4668e-05    0.00107757     0.0217602    0.000231651   0.00275534       100    
[art]:TriggerResults:TriggerResultInserter             8.636e-06    1.25959e-05   4.5716e-05    1.1206e-05    5.14263e-06      100    
end_path:out1:RootOutput                               2.836e-06    6.1444e-06    2.3364e-05     6.111e-06    3.2148e-06       100    
end_path:out1:RootOutput(write)                       0.00794493     0.232882       0.88496       0.23154      0.185844        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 16875.9 MB
  Peak resident set size usage (VmHWM): 2283.38 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66027687_441_20231207T003121Z_gen_g4_detsim_hitreco__20240510T061336Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66027687_441_20231207T003121Z_gen_g4_detsim_hitreco__20240510T061336Z_reco2_reco_2024-11-23T_032202Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66027687_441_20231207T003121Z_gen_g4_detsim_hitreco__20240510T061336Z_reco2_reco_data_2024-11-23T_032202Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work