Jobsub ID 301783.6@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301783.6@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:24:30
Site	UK_Imperial
Entry	DUNE_T2_UK_London_IC_ceprod03
Last heartbeat	2024-11-23 05:34:26
From worker node	Hostname	wj51.grid.hep.ph.ic.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:26:11
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50258586_176_20231122T061146Z_gen_g4_detsim_hitreco__20240503T070835Z_reco2.root
Jobscript	Exit code	0
	Real time	7m (479s)
	CPU time	1m (90s = 18%)
Outputting started	2024-11-23 05:34:11
Output files
Finished	2024-11-23 05:34:26
List job events Wrapper job log
Jobscript log (last 10,000 characters)

SG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 05:33:42 GMT run: 50258586 subRun: 1 event: 35203
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:33:42 GMT run: 50258586 subRun: 1 event: 35203
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:33:42 GMT run: 50258586 subRun: 1 event: 35203
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:33:42 GMT run: 50258586 subRun: 1 event: 35203
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 46, 36, 45
minwire 0: 568
minwire 1: 2760
minwire 2: 29
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Could not find serving_default in model signatures.
[libprotobuf FATAL /cvmfs/larsoft.opensciencegrid.org/products/protobuf/v3_21_12a/Linux64bit+3.10-2.17-e26/include/google/protobuf/map.h:1300] CHECK failed: it != end(): key not found: serving_default
23-Nov-2024 05:34:10 GMT  Closed input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/9b/65/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50258586_176_20231122T061146Z_gen_g4_detsim_hitreco__20240503T070835Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                            0.00498336      81.1986       241.907       1.68361       113.64          3     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00305576    0.00438868    0.00512693    0.00498336    0.000944341       3     
reco:linecluster:LineCluster                           0.038675      0.090397      0.128059      0.104457      0.037821         3     
reco:trajcluster:TrajCluster                          0.00845792     0.0157192     0.0266175     0.0120822    0.00784701        3     
reco:pandora:StandardPandora                           0.0329541      79.6545       238.893      0.0370369      112.599         3     
reco:pandoraTrack:LArPandoraTrackCreation             0.000154481   0.000789954   0.00205462    0.000160764   0.000894255       3     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000200214   0.00109033    0.00285116    0.00021962    0.00124512        3     
reco:pandoracalo:Calorimetry                          0.000118162   0.000477763   0.00118589    0.000129237   0.000500741       3     
reco:pandorapid:Chi2ParticleID                        4.9547e-05    0.000248871   0.000642191   5.4876e-05    0.000278128       3     
reco:trkshowersplit:TrackShowerHits                   0.000137366   0.000770275   0.00186888    0.000304578   0.000779826       3     
reco:pmtrack:PMAlgTrackMaker                          0.00149832     0.0245969     0.0375674     0.0347249     0.0163743        3     
reco:pmtrackcalo:Calorimetry                          9.9905e-05    0.00102092    0.00237913    0.000583729   0.000980498       3     
reco:pmtrackpid:Chi2ParticleID                        4.1963e-05    0.000318421   0.000806102   0.000107199   0.000345869       3     
reco:pmtrajfit:PMAlgTrajFitter                        0.000259282   0.00075934    0.00175739    0.000261352   0.000705726       3     
reco:pmtrajfitcalo:Calorimetry                        9.1939e-05    0.000107072   0.000122365   0.000106912   1.24219e-05       3     
reco:pmtrajfitpid:Chi2ParticleID                      4.1418e-05    4.3444e-05    4.6963e-05    4.1951e-05    2.4978e-06        3     
reco:pmtracktc:PMAlgTrackMaker                        0.00138447     0.0356716     0.0915528     0.0140775     0.0398523        3     
reco:pmtracktccalo:Calorimetry                        9.8061e-05    0.000339136   0.000587655   0.000331692   0.000199945       3     
reco:pmtracktcpid:Chi2ParticleID                      4.1622e-05    9.06663e-05   0.000141982   8.8395e-05    4.10033e-05       3     
reco:pmtrajfittc:PMAlgTrajFitter                       0.0149765     0.0632719     0.101426      0.0734128     0.0360141        3     
reco:pmtrajfittccalo:Calorimetry                      0.000286909   0.000400274   0.000541176   0.000372737   0.000105615       3     
reco:pmtrajfittcpid:Chi2ParticleID                    7.9955e-05    9.86843e-05   0.000120831   9.5267e-05    1.68616e-05       3     
reco:blurredcluster:BlurredClustering                  0.0168316     0.145977      0.318992      0.102106      0.127197         3     
reco:emtrkmichelid:EmTrackMichelId                     0.331719       0.73172       1.20764       0.6558       0.361601         3     
reco:emshower:EMShower                                0.000294467   0.00117826    0.00292995    0.000310368   0.00123865        3     
reco:cvnmap:CVNMapper                                  3.132e-05    0.00925962     0.0275253    0.000222269    0.012916         3     
reco:cvneva:CVNEvaluator                              2.2652e-05    0.000121632   0.000220612   0.000121632    9.898e-05        2     
reco:energyrecnumu:EnergyReco                         0.00284189    0.00404057    0.00523925    0.00404057    0.00119868        2     
reco:energyrecnue:EnergyReco                          0.00277525    0.00279683    0.00281841    0.00279683    2.1582e-05        2     
reco:energyrecnc:EnergyReco                           0.00269874    0.00280127    0.00290381    0.00280127    0.000102536       2     
reco:energyrecnumurange:EnergyReco                    0.00291666    0.00293934    0.00296202    0.00293934    2.2677e-05        2     
reco:energyrecnumumcs:EnergyReco                      0.00291016    0.00293226    0.00295435    0.00293226    2.20935e-05       2     
reco:opdec:Deconvolution                               0.196507      0.210179      0.223851      0.210179      0.0136718        2     
reco:ophitspe:OpHitFinderDeco                          0.892545       1.14656       1.40058       1.14656      0.254017         2     
reco:opflash:OpFlashFinder                            0.000620219   0.00193632    0.00325242    0.00193632     0.0013161        2     
reco:rns:RandomNumberSaver                            2.3693e-05    0.000143796   0.000263899   0.000143796   0.000120103       2     
reco:anglereconue:NuAngularReco                       0.000158114   0.000469798   0.000781482   0.000469798   0.000311684       2     
reco:anglereconumu:NuAngularReco                      9.8597e-05    0.000128971   0.000159345   0.000128971   3.0374e-05        2     
reco:anglereconuepfps:NuAngularReco                   0.00016091    0.000660749   0.00116059    0.000660749   0.000499839       2     
reco:anglereconumupfps:NuAngularReco                  0.000142258   0.000172722   0.000203185   0.000172722   3.04635e-05       2     
[art]:TriggerResults:TriggerResultInserter            1.2446e-05    2.9018e-05     4.559e-05    2.9018e-05    1.6572e-05        2     
end_path:out1:RootOutput                               3.184e-06    9.0835e-06    1.4983e-05    9.0835e-06    5.8995e-06        2     
end_path:out1:RootOutput(write)                        0.0603158     0.060923      0.0615303     0.060923     0.000607283       2     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 7071.46 MB
  Peak resident set size usage (VmHWM): 1154.61 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Nov-2024 05:34:10 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- StdException BEGIN
      An exception was thrown while processing module CVNEvaluator/cvneva run: 50258586 subRun: 1 event: 35203
      CHECK failed: it != end(): key not found: serving_default
    ---- StdException END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 2 entries while sim::OpDetDivRecs_opdigi__detsim. has 200 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-a556-e4eb-a585-de02.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50258586_176_20231122T061146Z_gen_g4_detsim_hitreco__20240503T070835Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50258586_176_20231122T061146Z_gen_g4_detsim_hitreco__20240503T070835Z_reco2_reco_2024-11-23T_052614Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work