Jobsub ID 301702.7@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301702.7@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:26:48
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce506
Last heartbeat	2024-11-23 05:04:01
From worker node	Hostname	b9p05p4086.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:27:32
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74581430_697_20231208T082350Z_gen_g4_detsim_hitreco__20240510T062742Z_reco2.root
Jobscript	Exit code	0
	Real time	36m (2171s)
	CPU time	41m (2489s = 114%)
Outputting started	2024-11-23 05:03:44
Output files
Finished	2024-11-23 05:04:01
List job events Wrapper job log
Jobscript log (last 10,000 characters)

summary: 
Output 0: 0.0824814, 0.00262626, 0.914892, 
Output 1: 0.00174468, 0.0593355, 0.671472, 0.267448, 
Output 2: 0.971441, 0.028559, 

481
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1492!
Begin processing the 100th record. run: 74581430 subRun: 1 event: 69800 at 23-Nov-2024 06:03:39 CET
> Running Algorithm: Alg0001, LArPreProcessing
> Running Algorithm: Alg0002, LArDLMaster
> Running Algorithm: Alg0003, LArNeutrinoEventValidation
---RAW-MATCHING-OUTPUT--------------------------------------------------------------------------
Failure in algorithm Alg0003, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  23-Nov-2024 06:03:40 CET run: 74581430 subRun: 1 event: 69800
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  23-Nov-2024 06:03:40 CET run: 74581430 subRun: 1 event: 69800
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtrack@BeginModule  23-Nov-2024 06:03:40 CET run: 74581430 subRun: 1 event: 69800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:03:40 CET run: 74581430 subRun: 1 event: 69800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:03:40 CET run: 74581430 subRun: 1 event: 69800
Comparing two wires in the same plane: return failure
%MSG
250
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 689!
23-Nov-2024 06:03:43 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74581430_697_20231208T082350Z_gen_g4_detsim_hitreco__20240510T062742Z_reco2_reco_data_2024-11-23T_042736Z.root"
23-Nov-2024 06:03:43 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/6e/d0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74581430_697_20231208T082350Z_gen_g4_detsim_hitreco__20240510T062742Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.152859       20.912        282.886       7.60326       40.9313        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00357288     0.0057561     0.0124013    0.00526523    0.00169289       100    
reco:linecluster:LineCluster                           0.0177946     0.160323       6.29712      0.0415681     0.667812        100    
reco:trajcluster:TrajCluster                          0.00891099     0.425019       10.7515      0.0184322      1.50971        100    
reco:pandora:StandardPandora                           0.0225502      1.66505       23.1817       1.09711       2.93993        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000113431   0.00434939     0.0548691    0.00131663     0.008726        100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000152012    0.0116239      0.19404     0.00296884     0.0283227       100    
reco:pandoracalo:Calorimetry                          8.3341e-05     0.0162296     0.512362     0.000938603    0.0642368       100    
reco:pandorapid:Chi2ParticleID                        3.3501e-05    0.000142071   0.00183793    0.000117926   0.000185678      100    
reco:trkshowersplit:TrackShowerHits                   6.5291e-05     0.0204318     0.531467     0.000622858    0.0695752       100    
reco:pmtrack:PMAlgTrackMaker                          0.000874671     1.92198       55.6806      0.146425       6.46417        100    
reco:pmtrackcalo:Calorimetry                          5.9621e-05    0.00737488     0.230107     0.000734405    0.0267626       100    
reco:pmtrackpid:Chi2ParticleID                         2.316e-05    0.000225335   0.00231657    0.000119806   0.000331874      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000158792     1.55751       31.0581      0.181472       4.0323         100    
reco:pmtrajfitcalo:Calorimetry                        5.3661e-05     0.0177249     0.481451     0.00114026     0.0676177       100    
reco:pmtrajfitpid:Chi2ParticleID                       2.372e-05    0.000162591   0.000510036   0.000127782   0.000122306      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00079177      2.25631       45.3361      0.169361       6.71776        100    
reco:pmtracktccalo:Calorimetry                        5.6371e-05    0.00446771     0.110869     0.000704745    0.0125654       100    
reco:pmtracktcpid:Chi2ParticleID                      2.3331e-05     0.000204     0.00167218    0.000118797   0.000239896      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000153282     1.23682       20.2905      0.228962       3.05904        100    
reco:pmtrajfittccalo:Calorimetry                      5.4201e-05     0.0113674     0.288981     0.000921832    0.0389829       100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.3411e-05    0.000289655   0.00272502    0.000136762   0.000425699      100    
reco:blurredcluster:BlurredClustering                 9.3061e-05     0.917885       5.62638      0.462001       1.09342        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000140122     8.00521       83.5277       2.12258       14.8133        100    
reco:emshower:EMShower                                0.000178773    0.0237629     0.205698     0.00225324     0.0428496       100    
reco:cvnmap:CVNMapper                                  1.985e-05    0.00787875     0.0320259    0.00549506    0.00745164       100    
reco:cvneva:CVNEvaluator                              1.4491e-05     0.594596       1.30358      0.739778      0.375404        100    
reco:energyrecnumu:EnergyReco                         0.000328095    0.0386743     0.603817     0.00375568     0.115586        100    
reco:energyrecnue:EnergyReco                          0.000119662   0.00165591     0.0176163    0.000479291   0.00284577       100    
reco:energyrecnc:EnergyReco                           0.00157399    0.00354518     0.0190832    0.00248908    0.00311672       100    
reco:energyrecnumurange:EnergyReco                    0.000141331    0.0370133      0.59187      0.0027596     0.114951        100    
reco:energyrecnumumcs:EnergyReco                      0.00075046     0.0375781     0.587853     0.00336721     0.113042        100    
reco:opdec:Deconvolution                               0.0141993     0.162624      0.327627      0.161033      0.0647506       100    
reco:ophitspe:OpHitFinderDeco                          0.0332351      1.54929       3.73881       1.49528      0.759792        100    
reco:opflash:OpFlashFinder                            0.000112012   0.00121121     0.0049354    0.00112633    0.000605401      100    
reco:rns:RandomNumberSaver                             1.724e-05    3.25481e-05   0.000258304    2.772e-05    2.47424e-05      100    
reco:anglereconue:NuAngularReco                       0.000122322    0.0010996     0.0147775    0.000320344   0.00216901       100    
reco:anglereconumu:NuAngularReco                       5.688e-05    0.000931374    0.013553     0.000177137   0.00203596       100    
reco:anglereconuepfps:NuAngularReco                   9.6991e-05    0.00105057     0.0157457    0.000293489   0.00218872       100    
reco:anglereconumupfps:NuAngularReco                  8.9331e-05    0.00100287     0.0137497    0.000249144   0.00206064       100    
[art]:TriggerResults:TriggerResultInserter             8.17e-06     1.2579e-05    5.5241e-05    1.1461e-05    5.65147e-06      100    
end_path:out1:RootOutput                               2.52e-06     5.01457e-06    2.308e-05     3.975e-06    2.95098e-06      100    
end_path:out1:RootOutput(write)                        0.0122646     0.201868      0.702882      0.181964       0.13577        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21594 MB
  Peak resident set size usage (VmHWM): 2179.22 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74581430_697_20231208T082350Z_gen_g4_detsim_hitreco__20240510T062742Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74581430_697_20231208T082350Z_gen_g4_detsim_hitreco__20240510T062742Z_reco2_reco_2024-11-23T_042736Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74581430_697_20231208T082350Z_gen_g4_detsim_hitreco__20240510T062742Z_reco2_reco_data_2024-11-23T_042736Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work