Jobsub ID 108781.23@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108781.23@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:08:23
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce505
Last heartbeat	2024-11-23 05:58:28
From worker node	Hostname	b9p08p0363.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:10:21
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_961_20231205T063715Z_gen_g4_detsim_hitreco__20240510T024931Z_reco2.root
Jobscript	Exit code	0
	Real time	47m (2868s)
	CPU time	49m (2940s = 102%)
Outputting started	2024-11-23 05:58:10
Output files
Finished	2024-11-23 05:58:28
List job events Wrapper job log
Jobscript log (last 10,000 characters)

23-Nov-2024 06:58:05 CET run: 66020489 subRun: 1 event: 96200
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  23-Nov-2024 06:58:05 CET run: 66020489 subRun: 1 event: 96200
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtrack@BeginModule  23-Nov-2024 06:58:05 CET run: 66020489 subRun: 1 event: 96200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 06:58:05 CET run: 66020489 subRun: 1 event: 96200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:58:05 CET run: 66020489 subRun: 1 event: 96200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:58:05 CET run: 66020489 subRun: 1 event: 96200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:58:05 CET run: 66020489 subRun: 1 event: 96200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:58:05 CET run: 66020489 subRun: 1 event: 96200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:58:06 CET run: 66020489 subRun: 1 event: 96200
Comparing two wires in the same plane: return failure
%MSG
247
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 684!
23-Nov-2024 06:58:09 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_961_20231205T063715Z_gen_g4_detsim_hitreco__20240510T024931Z_reco2_reco_data_2024-11-23T_051026Z.root"
23-Nov-2024 06:58:09 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/0d/79/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_961_20231205T063715Z_gen_g4_detsim_hitreco__20240510T024931Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.48003       26.1743       519.477       9.28671       64.2674        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0169178     0.0462876     0.236006      0.0465086     0.0231569       100    
reco:linecluster:LineCluster                           0.150886      0.420545       17.2735      0.179078       1.74115        100    
reco:trajcluster:TrajCluster                           0.047881      0.703064       25.8565      0.0610811      3.08428        100    
reco:pandora:StandardPandora                           0.902372       3.53865       185.604       1.26551       18.4761        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000162572   0.00584826     0.0930829    0.00144855     0.0150918       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000252152    0.0152951     0.446725     0.00309365     0.0551386       100    
reco:pandoracalo:Calorimetry                          9.8721e-05     0.0157017     0.443993     0.00101201     0.0627786       100    
reco:pandorapid:Chi2ParticleID                        4.3091e-05    0.000183124   0.00264326    0.000133706   0.000266852      100    
reco:trkshowersplit:TrackShowerHits                   0.000126671    0.021249      0.698021      0.0008422     0.0868876       100    
reco:pmtrack:PMAlgTrackMaker                          0.00126702      2.18999       54.2308      0.216281       7.35791        100    
reco:pmtrackcalo:Calorimetry                          6.7451e-05    0.00906703     0.179835     0.000937011    0.0306989       100    
reco:pmtrackpid:Chi2ParticleID                        3.2211e-05    0.000249723    0.0019562    0.000162847   0.000313051      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000257312     1.65135       47.8958      0.248103       5.32592        100    
reco:pmtrajfitcalo:Calorimetry                        8.0701e-05     0.0171938     0.667626     0.00131191     0.0756786       100    
reco:pmtrajfitpid:Chi2ParticleID                       3.203e-05    0.000192585   0.000992571   0.000153657   0.00016012       100    
reco:pmtracktc:PMAlgTrackMaker                        0.00108178      2.80181       75.9809      0.175827       9.9871         100    
reco:pmtracktccalo:Calorimetry                        7.9511e-05    0.00736879      0.16924     0.000989761    0.0231133       100    
reco:pmtracktcpid:Chi2ParticleID                      3.0711e-05    0.00026217    0.00222535    0.000157502   0.000344881      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000225542     1.59377       32.6366      0.295865       4.78012        100    
reco:pmtrajfittccalo:Calorimetry                      9.8251e-05     0.0120007     0.274547     0.00118118     0.0412124       100    
reco:pmtrajfittcpid:Chi2ParticleID                     4.437e-05    0.000315866   0.00278507    0.000178192   0.000457546      100    
reco:blurredcluster:BlurredClustering                  0.0284703      0.9394        4.95576      0.523938       1.08938        100    
reco:emtrkmichelid:EmTrackMichelId                     0.251534       8.67145       96.835        2.4578        17.5188        100    
reco:emshower:EMShower                                0.000236103    0.030692      0.384241       0.01108      0.0547212       100    
reco:cvnmap:CVNMapper                                  2.347e-05    0.00994935     0.0404214     0.0106847    0.00822063       100    
reco:cvneva:CVNEvaluator                               1.639e-05     0.747412       5.08502       0.82442      0.595081        100    
reco:energyrecnumu:EnergyReco                         0.000411894    0.0663703      3.96635     0.00395162     0.400988        100    
reco:energyrecnue:EnergyReco                          0.000132252   0.00205577     0.0448101    0.000450224   0.00589306       100    
reco:energyrecnc:EnergyReco                           0.00139199    0.00445618     0.0498427    0.00309053    0.00633061       100    
reco:energyrecnumurange:EnergyReco                    0.000177832    0.0561071      3.19317     0.00200792     0.325929        100    
reco:energyrecnumumcs:EnergyReco                      0.00168009     0.0526331      2.65726      0.0035201     0.274836        100    
reco:opdec:Deconvolution                               0.133592      0.281016      0.573724      0.273057      0.0821202       100    
reco:ophitspe:OpHitFinderDeco                          0.512049       2.00787       4.1732        1.88043      0.906854        100    
reco:opflash:OpFlashFinder                            0.000506786   0.00149447    0.00681805     0.0013551    0.000779468      100    
reco:rns:RandomNumberSaver                             1.953e-05    4.49364e-05   0.000429955   3.77305e-05   4.60048e-05      100    
reco:anglereconue:NuAngularReco                       0.000165112   0.00176294     0.0257189    0.000466415   0.00445784       100    
reco:anglereconumu:NuAngularReco                      7.4761e-05    0.00152435     0.0251467    0.000234928   0.00435661       100    
reco:anglereconuepfps:NuAngularReco                   0.000113622   0.00169504     0.0262962    0.000389139   0.00448913       100    
reco:anglereconumupfps:NuAngularReco                  9.4351e-05    0.00159852     0.0262023    0.000294683   0.00442121       100    
[art]:TriggerResults:TriggerResultInserter             8.05e-06     1.56228e-05   9.0561e-05    1.3385e-05    9.69911e-06      100    
end_path:out1:RootOutput                               2.99e-06     7.50427e-06    3.675e-05    7.4455e-06    4.20209e-06      100    
end_path:out1:RootOutput(write)                        0.0314691     0.244379      0.955118      0.227851      0.162296        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21570.3 MB
  Peak resident set size usage (VmHWM): 2149.73 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_961_20231205T063715Z_gen_g4_detsim_hitreco__20240510T024931Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_961_20231205T063715Z_gen_g4_detsim_hitreco__20240510T024931Z_reco2_reco_2024-11-23T_051026Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66020489_961_20231205T063715Z_gen_g4_detsim_hitreco__20240510T024931Z_reco2_reco_data_2024-11-23T_051026Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work