Jobsub ID 301748.71@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301748.71@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:09:29
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce505
Last heartbeat	2024-11-23 06:06:38
From worker node	Hostname	b9p11p5970.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:12:08
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419482_880_20231203T234638Z_gen_g4_detsim_hitreco__20240509T201858Z_reco2.root
Jobscript	Exit code	0
	Real time	54m (3249s)
	CPU time	53m (3237s = 99%)
Outputting started	2024-11-23 06:06:17
Output files
Finished	2024-11-23 06:06:38
List job events Wrapper job log
Jobscript log (last 10,000 characters)

--------------------------------------
Failure in algorithm Alg0003, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  23-Nov-2024 07:06:12 CET run: 6419482 subRun: 1 event: 88100
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  23-Nov-2024 07:06:12 CET run: 6419482 subRun: 1 event: 88100
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtrack@BeginModule  23-Nov-2024 07:06:13 CET run: 6419482 subRun: 1 event: 88100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 07:06:13 CET run: 6419482 subRun: 1 event: 88100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:06:13 CET run: 6419482 subRun: 1 event: 88100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:06:13 CET run: 6419482 subRun: 1 event: 88100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:06:13 CET run: 6419482 subRun: 1 event: 88100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:06:13 CET run: 6419482 subRun: 1 event: 88100
Comparing two wires in the same plane: return failure
%MSG
298
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 756!
23-Nov-2024 07:06:16 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419482_880_20231203T234638Z_gen_g4_detsim_hitreco__20240509T201858Z_reco2_reco_data_2024-11-23T_051214Z.root"
23-Nov-2024 07:06:16 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/f9/cf/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419482_880_20231203T234638Z_gen_g4_detsim_hitreco__20240509T201858Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.28397       30.3112       316.704       10.4833       54.9263        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0189617     0.0407991     0.0728823     0.0417822     0.0108621       100    
reco:linecluster:LineCluster                           0.152816      0.421872       8.04222      0.188409       1.05049        100    
reco:trajcluster:TrajCluster                           0.0487478      1.07755       38.2209      0.0692129      4.44248        100    
reco:pandora:StandardPandora                            1.0127        2.01011       15.5245       1.35337       2.21171        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000164682   0.00668149     0.112071     0.00174446     0.0153978       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000247073    0.0160276     0.335985     0.00395171     0.0441426       100    
reco:pandoracalo:Calorimetry                          0.000138722    0.0194088     0.340811     0.00142619     0.0553102       100    
reco:pandorapid:Chi2ParticleID                        5.4491e-05    0.000200862   0.00141675    0.000159787   0.000174588      100    
reco:trkshowersplit:TrackShowerHits                   0.000140212    0.0312188     0.754807     0.000917122    0.108228        100    
reco:pmtrack:PMAlgTrackMaker                          0.00139002      2.95935       52.3599      0.253728       7.97085        100    
reco:pmtrackcalo:Calorimetry                          7.6761e-05    0.00962383     0.248566      0.0010932     0.0292207       100    
reco:pmtrackpid:Chi2ParticleID                        2.8191e-05    0.000312019   0.00165331    0.000180697   0.00036209       100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000242853     2.32494       30.1204      0.362251       5.1301         100    
reco:pmtrajfitcalo:Calorimetry                        6.5821e-05     0.0192295     0.384405     0.00186327     0.0552769       100    
reco:pmtrajfitpid:Chi2ParticleID                      3.0541e-05    0.000218588   0.000896741   0.000169632   0.000167378      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00129992      3.8218        65.4554      0.273981       10.9358        100    
reco:pmtracktccalo:Calorimetry                        8.8001e-05    0.00806599     0.280236     0.00112264     0.0296765       100    
reco:pmtracktcpid:Chi2ParticleID                       3.47e-05     0.000289289   0.00208099    0.000169902   0.000346404      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000230053     1.82425       16.6908      0.337724       3.62078        100    
reco:pmtrajfittccalo:Calorimetry                      9.3471e-05     0.0129044     0.192272     0.00119407     0.0331377       100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.6211e-05    0.000372727   0.00233799    0.000182913   0.00047301       100    
reco:blurredcluster:BlurredClustering                 0.00587742      1.15088       6.61873      0.654768       1.36297        100    
reco:emtrkmichelid:EmTrackMichelId                     0.299665       10.8639       108.372       3.05879       19.3994        100    
reco:emshower:EMShower                                0.000316404    0.0409295     0.502909     0.00628256     0.0713135       100    
reco:cvnmap:CVNMapper                                  2.114e-05     0.0125382     0.0378025     0.0138639    0.00997284       100    
reco:cvneva:CVNEvaluator                              1.6361e-05     0.712557       1.26577      0.932077      0.454812        100    
reco:energyrecnumu:EnergyReco                         0.000416765    0.0449195      2.20024     0.00419961     0.225843        100    
reco:energyrecnue:EnergyReco                          0.000143882   0.00248428     0.0350891    0.000626538   0.00549858       100    
reco:energyrecnc:EnergyReco                           0.00170232    0.00522446     0.0374338    0.00362696    0.00567573       100    
reco:energyrecnumurange:EnergyReco                    0.000180353     0.04261       2.15575     0.00308736     0.221348        100    
reco:energyrecnumumcs:EnergyReco                       0.0017606     0.0452897      2.26611     0.00456536      0.2316         100    
reco:opdec:Deconvolution                               0.115663      0.297267      0.608653      0.296984      0.0902051       100    
reco:ophitspe:OpHitFinderDeco                          0.328603       2.2077        4.19356       2.11037      0.923474        100    
reco:opflash:OpFlashFinder                            0.000361634    0.0017718     0.0062572    0.00166453    0.000779543      100    
reco:rns:RandomNumberSaver                             2.338e-05    5.06708e-05   0.000402495    4.618e-05    3.8363e-05       100    
reco:anglereconue:NuAngularReco                       0.000210713   0.00191722     0.0296496    0.000554032   0.00427466       100    
reco:anglereconumu:NuAngularReco                      8.4851e-05    0.00158543     0.0295877    0.000284289   0.00435223       100    
reco:anglereconuepfps:NuAngularReco                   0.000139192    0.0017821     0.0295513    0.00044878    0.00440899       100    
reco:anglereconumupfps:NuAngularReco                  0.000104631   0.00167255     0.0295463    0.00035674    0.00442971       100    
[art]:TriggerResults:TriggerResultInserter             8.57e-06     1.60781e-05    4.908e-05    1.5075e-05    5.64852e-06      100    
end_path:out1:RootOutput                               3.93e-06     7.30706e-06    1.944e-05     7.21e-06     1.94011e-06      100    
end_path:out1:RootOutput(write)                        0.0200791     0.268628       1.08699      0.277925      0.186268        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21627.1 MB
  Peak resident set size usage (VmHWM): 2197.31 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419482_880_20231203T234638Z_gen_g4_detsim_hitreco__20240509T201858Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419482_880_20231203T234638Z_gen_g4_detsim_hitreco__20240509T201858Z_reco2_reco_2024-11-23T_051214Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419482_880_20231203T234638Z_gen_g4_detsim_hitreco__20240509T201858Z_reco2_reco_data_2024-11-23T_051214Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work