Jobsub ID 108807.7@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108807.7@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:23:22
Site	ES_PIC
Entry	DUNE_T1_ES_PIC_ce15-multicore
Last heartbeat	2024-11-23 06:36:27
From worker node	Hostname	tds431.pic.es
	cpuinfo	AMD EPYC 7502 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	216000 (60 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:24:22
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511854_1147_20231203T232126Z_gen_g4_detsim_hitreco__20240509T193340Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (4305s)
	CPU time	1h (4223s = 98%)
Outputting started	2024-11-23 06:36:08
Output files
Finished	2024-11-23 06:36:27
List job events Wrapper job log
Jobscript log (last 10,000 characters)

ov-2024 07:36:02 CET run: 74511854 subRun: 1 event: 114800
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  23-Nov-2024 07:36:02 CET run: 74511854 subRun: 1 event: 114800
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtrack@BeginModule  23-Nov-2024 07:36:02 CET run: 74511854 subRun: 1 event: 114800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 07:36:02 CET run: 74511854 subRun: 1 event: 114800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 07:36:02 CET run: 74511854 subRun: 1 event: 114800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 07:36:02 CET run: 74511854 subRun: 1 event: 114800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:36:02 CET run: 74511854 subRun: 1 event: 114800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:36:02 CET run: 74511854 subRun: 1 event: 114800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:36:02 CET run: 74511854 subRun: 1 event: 114800
Comparing two wires in the same plane: return failure
%MSG
261
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 687!
23-Nov-2024 07:36:06 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511854_1147_20231203T232126Z_gen_g4_detsim_hitreco__20240509T193340Z_reco2_reco_data_2024-11-23T_052428Z.root"
23-Nov-2024 07:36:06 CET  Closed input file "root://xrootd.pic.es:1094/pnfs/pic.es/data/dune/RSE/fardet-hd/10/d0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511854_1147_20231203T232126Z_gen_g4_detsim_hitreco__20240509T193340Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.31073       42.0002       714.592       11.6008       95.6553        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00126301    0.00319539    0.00776899    0.00321527    0.00111435       100    
reco:linecluster:LineCluster                           0.0112487     0.196141        5.161       0.0231673     0.668736        100    
reco:trajcluster:TrajCluster                          0.00173306      1.41053       39.4759      0.013853       5.57506        100    
reco:pandora:StandardPandora                            1.11574       4.37273       132.24        1.6125        15.3591        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000165863    0.0100264     0.205884     0.00190828     0.0255885       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000218964    0.0155776     0.307222     0.00378121     0.0374552       100    
reco:pandoracalo:Calorimetry                          0.000129623    0.029733      0.715668     0.00132682     0.100904        100    
reco:pandorapid:Chi2ParticleID                         4.215e-05    0.000243853   0.00389894    0.000157619   0.000411316      100    
reco:trkshowersplit:TrackShowerHits                   0.000134142    0.0374089     0.901756     0.000898568    0.126039        100    
reco:pmtrack:PMAlgTrackMaker                           0.0016501      3.69189       72.4048      0.320667        10.3          100    
reco:pmtrackcalo:Calorimetry                          9.4222e-05     0.0179867     0.495478     0.00118601     0.0613387       100    
reco:pmtrackpid:Chi2ParticleID                        3.9791e-05    0.000459153   0.00506397    0.000188783   0.000779958      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000276675     2.81047       54.1378      0.362851       7.28234        100    
reco:pmtrajfitcalo:Calorimetry                        8.1481e-05     0.0329265     0.800845     0.00180351     0.108943        100    
reco:pmtrajfitpid:Chi2ParticleID                      3.6161e-05    0.000278287   0.00185778    0.000184034   0.000299436      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00151486      5.10959       106.204      0.334886       15.1302        100    
reco:pmtracktccalo:Calorimetry                        9.1382e-05     0.0123472     0.385778      0.0011513     0.0423168       100    
reco:pmtracktcpid:Chi2ParticleID                       3.953e-05    0.000413269   0.00469408    0.000188528   0.000643706      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000253385     2.91412       81.3355      0.387285       9.04644        100    
reco:pmtrajfittccalo:Calorimetry                      9.1182e-05     0.0285426      1.09904      0.0015694     0.117859        100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.8291e-05    0.000568253   0.00752154    0.000214499   0.00102389       100    
reco:blurredcluster:BlurredClustering                  0.0102712      1.43709       10.6879       0.65707       1.82345        100    
reco:emtrkmichelid:EmTrackMichelId                     0.322859       16.0534       238.978       3.94046       32.8162        100    
reco:emshower:EMShower                                0.000320006    0.0524877     0.407902      0.0174233     0.0830438       100    
reco:cvnmap:CVNMapper                                 2.8321e-05     0.011761      0.0504031    0.00814047     0.010996        100    
reco:cvneva:CVNEvaluator                               1.97e-05      0.873928       1.54624       1.12581      0.500848        100    
reco:energyrecnumu:EnergyReco                         0.000388658    0.0635841      2.98733     0.00628862     0.310534        100    
reco:energyrecnue:EnergyReco                          0.000163953   0.00361626     0.0951517    0.000757475    0.0104973       100    
reco:energyrecnc:EnergyReco                           0.00236177    0.00670191     0.102763     0.00385175     0.0111156       100    
reco:energyrecnumurange:EnergyReco                    0.000278495    0.0614625      3.08073     0.00366116     0.319214        100    
reco:energyrecnumumcs:EnergyReco                      0.00230861     0.0645404      3.15748     0.00491373     0.326308        100    
reco:opdec:Deconvolution                               0.0310916     0.209709      0.432394      0.198441      0.0837429       100    
reco:ophitspe:OpHitFinderDeco                          0.210248       2.11622       4.71367       1.99436      0.896583        100    
reco:opflash:OpFlashFinder                            0.000319826   0.00172823    0.00688116    0.00157385    0.000873959      100    
reco:rns:RandomNumberSaver                             1.968e-05    4.23259e-05   0.000397888   3.1526e-05    4.35797e-05      100    
reco:anglereconue:NuAngularReco                       0.000180834   0.00273692     0.0846324    0.000459584   0.00918312       100    
reco:anglereconumu:NuAngularReco                      9.1102e-05    0.00250473     0.0840147    0.000307906   0.00912029       100    
reco:anglereconuepfps:NuAngularReco                   0.000155203   0.00268398     0.0840929    0.000423903   0.00917099       100    
reco:anglereconumupfps:NuAngularReco                  0.000140323   0.00259034     0.0819754    0.000387582    0.0089504       100    
[art]:TriggerResults:TriggerResultInserter             1.003e-05    1.7016e-05    9.3482e-05     1.431e-05    9.95895e-06      100    
end_path:out1:RootOutput                               2.94e-06     5.3441e-06    3.1931e-05     3.785e-06    3.44024e-06      100    
end_path:out1:RootOutput(write)                        0.0272401     0.335806       1.6951       0.307054      0.254228        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21528.4 MB
  Peak resident set size usage (VmHWM): 2389.95 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511854_1147_20231203T232126Z_gen_g4_detsim_hitreco__20240509T193340Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511854_1147_20231203T232126Z_gen_g4_detsim_hitreco__20240509T193340Z_reco2_reco_2024-11-23T_052428Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511854_1147_20231203T232126Z_gen_g4_detsim_hitreco__20240509T193340Z_reco2_reco_data_2024-11-23T_052428Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work