Jobsub ID 108787.55@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108787.55@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:19:04
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce509
Last heartbeat	2024-11-23 05:51:11
From worker node	Hostname	b9p08p1455.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:20:19
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50649927_181_20231208T045000Z_gen_g4_detsim_hitreco__20240510T065237Z_reco2.root
Jobscript	Exit code	0
	Real time	30m (1834s)
	CPU time	33m (1996s = 108%)
Outputting started	2024-11-23 05:50:54
Output files
Finished	2024-11-23 05:51:11
List job events Wrapper job log
Jobscript log (last 10,000 characters)

ter
> Running Algorithm: Alg0003, LArNeutrinoEventValidation
---RAW-MATCHING-OUTPUT--------------------------------------------------------------------------
Failure in algorithm Alg0003, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtrack@BeginModule  23-Nov-2024 06:50:48 CET run: 50649927 subRun: 1 event: 18199
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:50:48 CET run: 50649927 subRun: 1 event: 18199
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:50:48 CET run: 50649927 subRun: 1 event: 18199
Comparing two wires in the same plane: return failure
%MSG
272
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 657!
Begin processing the 100th record. run: 50649927 subRun: 1 event: 18200 at 23-Nov-2024 06:50:50 CET
> Running Algorithm: Alg0001, LArPreProcessing
> Running Algorithm: Alg0002, LArDLMaster
> Running Algorithm: Alg0003, LArNeutrinoEventValidation
---RAW-MATCHING-OUTPUT--------------------------------------------------------------------------
Failure in algorithm Alg0003, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:50:51 CET run: 50649927 subRun: 1 event: 18200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:50:51 CET run: 50649927 subRun: 1 event: 18200
Comparing two wires in the same plane: return failure
%MSG
160
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 364!
23-Nov-2024 06:50:53 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50649927_181_20231208T045000Z_gen_g4_detsim_hitreco__20240510T065237Z_reco2_reco_data_2024-11-23T_052026Z.root"
23-Nov-2024 06:50:53 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/cc/1a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50649927_181_20231208T045000Z_gen_g4_detsim_hitreco__20240510T065237Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.532962       16.3489       283.931       7.12294       33.2535        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0163149     0.0414406     0.247143      0.0407174     0.0241007       100    
reco:linecluster:LineCluster                           0.094817      0.209868       1.84085      0.165455      0.224392        100    
reco:trajcluster:TrajCluster                           0.0473862     0.409377       20.5924      0.0535342      2.14354        100    
reco:pandora:StandardPandora                           0.0560673      1.50875       14.9173       1.21748       1.59063        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000112271   0.00411122     0.118954     0.00096564     0.0130194       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000150492   0.00773721     0.194881     0.00196462     0.0212699       100    
reco:pandoracalo:Calorimetry                          9.6051e-05    0.00931291      0.45778     0.000665057    0.0468696       100    
reco:pandorapid:Chi2ParticleID                        3.5811e-05    0.000150755   0.00154783    0.000108616   0.000168918      100    
reco:trkshowersplit:TrackShowerHits                   0.000103911    0.0143062     0.705742     0.000420765    0.0745069       100    
reco:pmtrack:PMAlgTrackMaker                          0.00095428     0.981214       30.7326      0.0895309      3.43946        100    
reco:pmtrackcalo:Calorimetry                          5.8831e-05     0.0054727     0.239965     0.000575152    0.0251043       100    
reco:pmtrackpid:Chi2ParticleID                        2.2861e-05    0.000202762   0.00251462    0.000103986   0.000319183      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000154122     1.22553       40.9805      0.114462       4.40283        100    
reco:pmtrajfitcalo:Calorimetry                        5.3031e-05    0.00981432      0.47001     0.000832184    0.0482762       100    
reco:pmtrajfitpid:Chi2ParticleID                       2.249e-05    0.000155014   0.00096258    0.000122082   0.000153805      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000765939     1.40843       50.5892      0.107642       5.55755        100    
reco:pmtracktccalo:Calorimetry                        5.6801e-05     0.0029371     0.0636437    0.000542326   0.00840275       100    
reco:pmtracktcpid:Chi2ParticleID                       2.236e-05    0.000171441   0.00113209    0.000111081   0.000185806      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000150102    0.802476       15.6629      0.195667       2.11637        100    
reco:pmtrajfittccalo:Calorimetry                      5.2141e-05    0.00679602     0.184577     0.000695698    0.0239728       100    
reco:pmtrajfittcpid:Chi2ParticleID                     2.129e-05     0.0002396    0.00179378    0.000134502   0.000336982      100    
reco:blurredcluster:BlurredClustering                 0.000110001    0.761789       6.18432      0.341783       1.0136         100    
reco:emtrkmichelid:EmTrackMichelId                    0.000137722     6.01267       94.2107       1.62366       12.8743        100    
reco:emshower:EMShower                                0.000169762    0.0214865     0.295238     0.00553002     0.0405738       100    
reco:cvnmap:CVNMapper                                 2.0591e-05     0.0107692     0.0396122     0.0136081    0.00898297       100    
reco:cvneva:CVNEvaluator                               1.62e-05      0.574814       1.15775       0.7544       0.414385        100    
reco:energyrecnumu:EnergyReco                         0.000302403    0.0206415     0.459621      0.0033552     0.0633711       100    
reco:energyrecnue:EnergyReco                          0.000119551   0.00162642     0.0383105    0.000438875   0.00462694       100    
reco:energyrecnc:EnergyReco                           0.00161497    0.00381844     0.0424631     0.0029891    0.00498011       100    
reco:energyrecnumurange:EnergyReco                    0.000116402    0.0189358     0.467047      0.0024773     0.0632523       100    
reco:energyrecnumumcs:EnergyReco                      0.000768049    0.0200823     0.449186     0.00320446     0.0610463       100    
reco:opdec:Deconvolution                               0.0496484     0.254131      0.683168      0.265593      0.0863191       100    
reco:ophitspe:OpHitFinderDeco                          0.142006       1.78849       4.14962       1.7791       0.795319        100    
reco:opflash:OpFlashFinder                            0.000144472   0.00136904     0.0045915     0.0013805    0.000615896      100    
reco:rns:RandomNumberSaver                            1.7811e-05    4.44605e-05   0.000454295   4.03555e-05   4.20257e-05      100    
reco:anglereconue:NuAngularReco                       0.000135711   0.00110137      0.02078     0.00040323    0.00293586       100    
reco:anglereconumu:NuAngularReco                      6.0831e-05    0.000848197    0.0191303    0.000188937   0.00277384       100    
reco:anglereconuepfps:NuAngularReco                   0.000101962   0.00100638     0.0191472    0.000342624   0.00280033       100    
reco:anglereconumupfps:NuAngularReco                  8.8961e-05    0.000925191    0.0193666    0.000257683   0.00283926       100    
[art]:TriggerResults:TriggerResultInserter             8.81e-06     1.37511e-05   7.0201e-05    1.2695e-05    6.42072e-06      100    
end_path:out1:RootOutput                               2.58e-06     6.54864e-06    2.817e-05     6.515e-06    2.61512e-06      100    
end_path:out1:RootOutput(write)                        0.011423      0.203948      0.692533      0.233763      0.135438        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21767.6 MB
  Peak resident set size usage (VmHWM): 2330.44 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50649927_181_20231208T045000Z_gen_g4_detsim_hitreco__20240510T065237Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50649927_181_20231208T045000Z_gen_g4_detsim_hitreco__20240510T065237Z_reco2_reco_2024-11-23T_052026Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50649927_181_20231208T045000Z_gen_g4_detsim_hitreco__20240510T065237Z_reco2_reco_data_2024-11-23T_052026Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work