Jobsub ID 108738.91@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108738.91@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:22:32
Site	UK_Durham
Entry	DUNE_UK_SGridDurham_ce3
Last heartbeat	2024-11-23 05:51:14
From worker node	Hostname	n247.dur.scotgrid.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:34:49
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74553259_502_20231207T141133Z_gen_g4_detsim_hitreco__20240510T063325Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (4568s)
	CPU time	1h (3778s = 82%)
Outputting started	2024-11-23 05:50:58
Output files
Finished	2024-11-23 05:51:14
List job events Wrapper job log
Jobscript log (last 10,000 characters)

ModularShowerCreation:pandoraShower@BeginModule  23-Nov-2024 05:50:54 GMT run: 74553259 subRun: 1 event: 50300
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtrack@BeginModule  23-Nov-2024 05:50:54 GMT run: 74553259 subRun: 1 event: 50300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 05:50:54 GMT run: 74553259 subRun: 1 event: 50300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 05:50:54 GMT run: 74553259 subRun: 1 event: 50300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 05:50:54 GMT run: 74553259 subRun: 1 event: 50300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 05:50:54 GMT run: 74553259 subRun: 1 event: 50300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 05:50:54 GMT run: 74553259 subRun: 1 event: 50300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 05:50:54 GMT run: 74553259 subRun: 1 event: 50300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:50:54 GMT run: 74553259 subRun: 1 event: 50300
Comparing two wires in the same plane: return failure
%MSG
145
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 296!
23-Nov-2024 05:50:56 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74553259_502_20231207T141133Z_gen_g4_detsim_hitreco__20240510T063325Z_reco2_reco_data_2024-11-23T_043501Z.root"
23-Nov-2024 05:50:56 GMT  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/b9/95/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74553259_502_20231207T141133Z_gen_g4_detsim_hitreco__20240510T063325Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.21726       38.3481       471.064       10.0878       80.4709        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0342548     0.0721701     0.394404      0.0735756     0.0433982       100    
reco:linecluster:LineCluster                            0.20092      0.715804       11.2987      0.376064       1.44649        100    
reco:trajcluster:TrajCluster                           0.101114       1.28847       28.1825      0.110247       4.63568        100    
reco:pandora:StandardPandora                           0.109169       2.36736       28.2094       1.49699       3.78534        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000125347   0.00826718     0.125858      0.0013126     0.0207386       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000166666    0.0317312      1.0942      0.00320194     0.129396        100    
reco:pandoracalo:Calorimetry                          0.00010032     0.0285538      0.58885     0.000946666    0.0901515       100    
reco:pandorapid:Chi2ParticleID                        3.7691e-05    0.00019471    0.00137003    0.000137817   0.000193228      100    
reco:trkshowersplit:TrackShowerHits                    7.926e-05     0.0354753     0.625921     0.000741626     0.11641        100    
reco:pmtrack:PMAlgTrackMaker                          0.00111415      3.98965       61.2007      0.185931       11.6019        100    
reco:pmtrackcalo:Calorimetry                          7.3259e-05     0.0176831     0.496111     0.000953493    0.0652704       100    
reco:pmtrackpid:Chi2ParticleID                        2.8584e-05    0.000373394   0.00337419    0.000154994   0.000610853      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000192315     2.59879       43.4938      0.223683       6.76883        100    
reco:pmtrajfitcalo:Calorimetry                        6.4382e-05     0.0303153     0.613564     0.00134703     0.0955843       100    
reco:pmtrajfitpid:Chi2ParticleID                      2.5799e-05    0.000236867   0.00169153    0.000160234   0.000266172      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000934381     5.37127       108.192      0.252394       16.8589        100    
reco:pmtracktccalo:Calorimetry                        7.1967e-05     0.011811      0.238998     0.00100925     0.038564        100    
reco:pmtracktcpid:Chi2ParticleID                      2.7903e-05    0.000341936    0.0027194    0.000161988   0.00048943       100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000155685     2.29974       34.3286      0.286664       5.71163        100    
reco:pmtrajfittccalo:Calorimetry                      6.6766e-05     0.0213677     0.411981     0.00103422     0.0670242       100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.7102e-05    0.000472558    0.0044525    0.000162568   0.000801271      100    
reco:blurredcluster:BlurredClustering                 6.2779e-05      1.23296       10.8301       0.38895       1.83431        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000116581     14.089        159.542       2.45238       28.4923        100    
reco:emshower:EMShower                                0.000146739    0.0326583     0.262426     0.00423739     0.0544826       100    
reco:cvnmap:CVNMapper                                 2.3063e-05     0.0105893     0.0550254    0.00675283     0.0119244       100    
reco:cvneva:CVNEvaluator                              1.6041e-05     0.630931       1.29294      0.847776      0.447515        100    
reco:energyrecnumu:EnergyReco                         0.000286263    0.0593294      1.20254     0.00517327     0.176245        100    
reco:energyrecnue:EnergyReco                          0.000144093   0.00282526     0.0435677    0.000653224   0.00664731       100    
reco:energyrecnc:EnergyReco                           0.00179881    0.00558834     0.0481332    0.00369778    0.00704297       100    
reco:energyrecnumurange:EnergyReco                    0.000146098    0.0571637      1.20771     0.00361314     0.175446        100    
reco:energyrecnumumcs:EnergyReco                      0.000919804    0.0589902      1.20095     0.00395208     0.175083        100    
reco:opdec:Deconvolution                               0.112071      0.453955       1.08599      0.444449      0.154408        100    
reco:ophitspe:OpHitFinderDeco                          0.302077       2.52513       5.20579       2.37676       1.16707        100    
reco:opflash:OpFlashFinder                            0.000123153   0.00156201     0.0038423    0.00143192    0.000796004      100    
reco:rns:RandomNumberSaver                            1.7844e-05    3.87021e-05   0.00026861    3.4095e-05    2.55784e-05      100    
reco:anglereconue:NuAngularReco                       0.000143752   0.00206312     0.0382284    0.000389809   0.00569287       100    
reco:anglereconumu:NuAngularReco                      7.0153e-05    0.00182715     0.0370971    0.000251416   0.00552181       100    
reco:anglereconuepfps:NuAngularReco                   0.000112694   0.00194748     0.036866     0.000381874   0.00549354       100    
reco:anglereconumupfps:NuAngularReco                  9.7945e-05     0.0018912     0.037101     0.000325677   0.00553415       100    
[art]:TriggerResults:TriggerResultInserter             9.107e-06    1.51541e-05   4.3623e-05    1.45225e-05   4.60665e-06      100    
end_path:out1:RootOutput                               2.685e-06    6.1425e-06    1.8305e-05    5.0145e-06    2.98035e-06      100    
end_path:out1:RootOutput(write)                       0.00809519      0.2874        1.35007      0.280887      0.238873        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 38416.7 MB
  Peak resident set size usage (VmHWM): 2350.38 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74553259_502_20231207T141133Z_gen_g4_detsim_hitreco__20240510T063325Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74553259_502_20231207T141133Z_gen_g4_detsim_hitreco__20240510T063325Z_reco2_reco_2024-11-23T_043501Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74553259_502_20231207T141133Z_gen_g4_detsim_hitreco__20240510T063325Z_reco2_reco_data_2024-11-23T_043501Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work