Jobsub ID 301818.0@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301818.0@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:36:33
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce508
Last heartbeat	2024-11-23 06:25:47
From worker node	Hostname	b9p11p3251.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:37:41
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74574126_737_20231208T031011Z_gen_g4_detsim_hitreco__20240510T042117Z_reco2.root
Jobscript	Exit code	0
	Real time	47m (2868s)
	CPU time	50m (3032s = 105%)
Outputting started	2024-11-23 06:25:29
Output files
Finished	2024-11-23 06:25:47
List job events Wrapper job log
Jobscript log (last 10,000 characters)

determination failure: 0, 199
Classifier summary: 
Output 0: 0.00280872, 0.991267, 0.0059245, 
Output 1: 0.00514001, 0.773209, 0.220191, 0.00145977, 
Output 2: 0.974913, 0.0250872, 

395
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1223!
Begin processing the 100th record. run: 74574126 subRun: 1 event: 73800 at 23-Nov-2024 07:25:22 CET
> Running Algorithm: Alg0001, LArPreProcessing
> Running Algorithm: Alg0002, LArDLMaster
> Running Algorithm: Alg0003, LArNeutrinoEventValidation
---RAW-MATCHING-OUTPUT--------------------------------------------------------------------------
Failure in algorithm Alg0003, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  23-Nov-2024 07:25:23 CET run: 74574126 subRun: 1 event: 73800
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtrack@BeginModule  23-Nov-2024 07:25:23 CET run: 74574126 subRun: 1 event: 73800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 07:25:23 CET run: 74574126 subRun: 1 event: 73800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 07:25:23 CET run: 74574126 subRun: 1 event: 73800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:25:23 CET run: 74574126 subRun: 1 event: 73800
Comparing two wires in the same plane: return failure
%MSG
448
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1397!
23-Nov-2024 07:25:28 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74574126_737_20231208T031011Z_gen_g4_detsim_hitreco__20240510T042117Z_reco2_reco_data_2024-11-23T_053747Z.root"
23-Nov-2024 07:25:28 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/a9/ad/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74574126_737_20231208T031011Z_gen_g4_detsim_hitreco__20240510T042117Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.571992       26.623        511.281       10.8265       59.8039        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0171195     0.040144      0.0782696     0.0393712     0.0128189       100    
reco:linecluster:LineCluster                           0.0945472     0.349318       11.0787      0.176109       1.12451        100    
reco:trajcluster:TrajCluster                           0.0474354      1.04554       49.6417      0.0675482      5.38896        100    
reco:pandora:StandardPandora                           0.0569367      2.14056       50.9172       1.28883       5.10887        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000112681   0.00626657     0.106722     0.00199655     0.0136543       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000157722    0.0106416      0.13152     0.00387285     0.0193357       100    
reco:pandoracalo:Calorimetry                          8.6501e-05     0.0203374     0.836945     0.00176109     0.0886549       100    
reco:pandorapid:Chi2ParticleID                         3.464e-05    0.000167258   0.00116493    0.000137351   0.000137319      100    
reco:trkshowersplit:TrackShowerHits                   7.1881e-05     0.0230248     0.812278      0.0013288     0.0939719       100    
reco:pmtrack:PMAlgTrackMaker                          0.00097253      2.34814       67.0355      0.336221       7.78764        100    
reco:pmtrackcalo:Calorimetry                          6.0471e-05     0.0126013     0.560177     0.00110009     0.0580169       100    
reco:pmtrackpid:Chi2ParticleID                         2.393e-05    0.000285539   0.00336567    0.000170777   0.000427261      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000162602     1.76571       33.8359       0.39007       4.48712        100    
reco:pmtrajfitcalo:Calorimetry                        5.7671e-05     0.0262236      1.39932     0.00224484     0.142295        100    
reco:pmtrajfitpid:Chi2ParticleID                       2.363e-05    0.000201008   0.00118384    0.000166517   0.000177433      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000785188     3.24251       105.926      0.346331       12.0582        100    
reco:pmtracktccalo:Calorimetry                        5.7781e-05    0.00724495     0.176775     0.00106744     0.0218269       100    
reco:pmtracktcpid:Chi2ParticleID                       2.376e-05    0.000252526   0.00186236    0.000153562   0.000306147      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000160262     1.44257       30.2455      0.345049       3.57993        100    
reco:pmtrajfittccalo:Calorimetry                       5.198e-05     0.0127738     0.452474     0.00150631     0.0494092       100    
reco:pmtrajfittcpid:Chi2ParticleID                     2.234e-05    0.000316968   0.00307117    0.000172497   0.000448585      100    
reco:blurredcluster:BlurredClustering                 0.000133492    0.930119       5.16928      0.587267       1.14101        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000146802     9.81761       144.801       3.36758       19.6266        100    
reco:emshower:EMShower                                0.000214982    0.0320166     0.254928     0.00685767     0.0530137       100    
reco:cvnmap:CVNMapper                                  2.283e-05    0.00908621     0.0318811    0.00691764    0.00757182       100    
reco:cvneva:CVNEvaluator                               1.521e-05     0.719817       5.62451      0.755018      0.645253        100    
reco:energyrecnumu:EnergyReco                         0.000307523    0.043168       1.01473     0.00392017     0.144071        100    
reco:energyrecnue:EnergyReco                          0.000117841   0.00193406     0.0322056    0.000633682   0.00409887       100    
reco:energyrecnc:EnergyReco                           0.00144621     0.0041049     0.0357004    0.00315059    0.00431169       100    
reco:energyrecnumurange:EnergyReco                    0.000165152    0.0399522      1.05022     0.00248904      0.13775        100    
reco:energyrecnumumcs:EnergyReco                      0.000524916    0.0405871      1.01823     0.00362041     0.134276        100    
reco:opdec:Deconvolution                               0.0679953     0.280744      0.478438      0.280756      0.0876783       100    
reco:ophitspe:OpHitFinderDeco                          0.141448       1.96958       4.13832       1.89492      0.873001        100    
reco:opflash:OpFlashFinder                            0.000140101   0.00140847     0.0059915    0.00131924    0.00078276       100    
reco:rns:RandomNumberSaver                             1.717e-05    3.66448e-05   0.000395104    3.331e-05    3.74617e-05      100    
reco:anglereconue:NuAngularReco                       0.000148881   0.00158331     0.0292146    0.000468055   0.00399639       100    
reco:anglereconumu:NuAngularReco                      6.4821e-05    0.00139022     0.0284486    0.000268382    0.0039208       100    
reco:anglereconuepfps:NuAngularReco                   0.000102831   0.00152175     0.0283786    0.000416719   0.00391077       100    
reco:anglereconumupfps:NuAngularReco                   8.61e-05     0.00144206     0.0281271    0.000340603   0.00389273       100    
[art]:TriggerResults:TriggerResultInserter             8.83e-06     1.31552e-05   5.4131e-05    1.23855e-05   5.09492e-06      100    
end_path:out1:RootOutput                                2.9e-06     5.80587e-06    2.299e-05    5.8655e-06    2.66196e-06      100    
end_path:out1:RootOutput(write)                        0.0157022     0.230179      0.969583      0.214754      0.169898        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21821.4 MB
  Peak resident set size usage (VmHWM): 2400.81 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74574126_737_20231208T031011Z_gen_g4_detsim_hitreco__20240510T042117Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74574126_737_20231208T031011Z_gen_g4_detsim_hitreco__20240510T042117Z_reco2_reco_2024-11-23T_053747Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74574126_737_20231208T031011Z_gen_g4_detsim_hitreco__20240510T042117Z_reco2_reco_data_2024-11-23T_053747Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work