Jobsub ID 108652.97@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108652.97@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 01:01:25
Site	UK_Imperial
Entry	DUNE_T2_UK_London_IC_ceprod03
Last heartbeat	2024-11-23 01:33:05
From worker node	Hostname	wj17.grid.hep.ph.ic.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 01:22:21
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467376_206_20231207T070654Z_gen_g4_detsim_hitreco__20240510T051457Z_reco2.root
Jobscript	Exit code	0
	Real time	10m (624s)
	CPU time	1m (101s = 16%)
Outputting started	2024-11-23 01:32:45
Output files
Finished	2024-11-23 01:33:05
List job events Wrapper job log
Jobscript log (last 10,000 characters)

467376 subRun: 1 event: 20602
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 01:31:59 GMT run: 6467376 subRun: 1 event: 20602
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 01:31:59 GMT run: 6467376 subRun: 1 event: 20602
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 64, 57, 57
minwire 0: 1758
minwire 1: 1040
minwire 2: 1538
Used alternate method to get min and max wires due to vertex determination failure: 166, 365
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max wires due to vertex determination failure: 185, 384
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max wires due to vertex determination failure: 192, 391
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Could not find serving_default in model signatures.
[libprotobuf FATAL /cvmfs/larsoft.opensciencegrid.org/products/protobuf/v3_21_12a/Linux64bit+3.10-2.17-e26/include/google/protobuf/map.h:1300] CHECK failed: it != end(): key not found: serving_default
23-Nov-2024 01:32:44 GMT  Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/f7/66/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467376_206_20231207T070654Z_gen_g4_detsim_hitreco__20240510T051457Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                            0.00747733      137.631       275.255       137.631       137.624         2     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00747733    0.00769028    0.00790323    0.00769028    0.000212952       2     
reco:linecluster:LineCluster                           0.0987465     0.205406      0.312065      0.205406      0.106659         2     
reco:trajcluster:TrajCluster                           0.026299      0.0335964     0.0408939     0.0335964    0.00729745        2     
reco:pandora:StandardPandora                           0.0748752      135.811       271.548       135.811       135.736         2     
reco:pandoraTrack:LArPandoraTrackCreation             0.000192047   0.00106197    0.00193189    0.00106197    0.000869923       2     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000439327   0.00157991    0.00272049    0.00157991    0.00114058        2     
reco:pandoracalo:Calorimetry                          0.000160054   0.000685411   0.00121077    0.000685411   0.000525357       2     
reco:pandorapid:Chi2ParticleID                        7.6891e-05    0.000382521   0.00068815    0.000382521   0.00030563        2     
reco:trkshowersplit:TrackShowerHits                   0.000425618   0.00118575    0.00194589    0.00118575    0.000760137       2     
reco:pmtrack:PMAlgTrackMaker                          0.00467434     0.0169196     0.0291648     0.0169196     0.0122452        2     
reco:pmtrackcalo:Calorimetry                          0.000562332    0.0015135    0.00246467     0.0015135    0.00095117        2     
reco:pmtrackpid:Chi2ParticleID                        0.000120074   0.000516059   0.000912045   0.000516059   0.000395985       2     
reco:pmtrajfit:PMAlgTrajFitter                        0.000479057   0.00110381    0.00172855    0.00110381    0.000624748       2     
reco:pmtrajfitcalo:Calorimetry                        0.000101259   0.000109101   0.000116942   0.000109101   7.8415e-06        2     
reco:pmtrajfitpid:Chi2ParticleID                      4.7729e-05    4.80935e-05   4.8458e-05    4.80935e-05    3.645e-07        2     
reco:pmtracktc:PMAlgTrackMaker                        0.00983394     0.0272109     0.0445879     0.0272109     0.017377         2     
reco:pmtracktccalo:Calorimetry                        0.000336955   0.000429424   0.000521894   0.000429424   9.24695e-05       2     
reco:pmtracktcpid:Chi2ParticleID                      9.3274e-05    0.000137645   0.000182016   0.000137645   4.4371e-05        2     
reco:pmtrajfittc:PMAlgTrajFitter                       0.0206236     0.046634      0.0726443     0.046634      0.0260103        2     
reco:pmtrajfittccalo:Calorimetry                      0.000354022   0.000450616   0.00054721    0.000450616   9.6594e-05        2     
reco:pmtrajfittcpid:Chi2ParticleID                    0.000107536   0.000123585   0.000139634   0.000123585   1.6049e-05        2     
reco:blurredcluster:BlurredClustering                  0.0414332      1.41374       2.78605       1.41374       1.37231         2     
reco:emtrkmichelid:EmTrackMichelId                     0.577107       1.43284       2.28857       1.43284       0.85573         2     
reco:emshower:EMShower                                0.000585379    0.0018688    0.00315222     0.0018688    0.00128342        2     
reco:cvnmap:CVNMapper                                 0.000257907    0.0219449     0.0436319     0.0219449     0.021687         2     
reco:cvneva:CVNEvaluator                              0.000230302   0.000230302   0.000230302   0.000230302        0            1     
reco:energyrecnumu:EnergyReco                         0.00546671    0.00546671    0.00546671    0.00546671         0            1     
reco:energyrecnue:EnergyReco                          0.00296949    0.00296949    0.00296949    0.00296949         0            1     
reco:energyrecnc:EnergyReco                           0.00303173    0.00303173    0.00303173    0.00303173         0            1     
reco:energyrecnumurange:EnergyReco                    0.00294775    0.00294775    0.00294775    0.00294775         0            1     
reco:energyrecnumumcs:EnergyReco                      0.00282622    0.00282622    0.00282622    0.00282622         0            1     
reco:opdec:Deconvolution                               0.274046      0.274046      0.274046      0.274046          0            1     
reco:ophitspe:OpHitFinderDeco                           2.15487       2.15487       2.15487       2.15487          0            1     
reco:opflash:OpFlashFinder                             0.0051993     0.0051993     0.0051993     0.0051993         0            1     
reco:rns:RandomNumberSaver                            0.000394961   0.000394961   0.000394961   0.000394961        0            1     
reco:anglereconue:NuAngularReco                       0.00123163    0.00123163    0.00123163    0.00123163         0            1     
reco:anglereconumu:NuAngularReco                      0.000245515   0.000245515   0.000245515   0.000245515        0            1     
reco:anglereconuepfps:NuAngularReco                   0.00172578    0.00172578    0.00172578    0.00172578         0            1     
reco:anglereconumupfps:NuAngularReco                  0.00031906    0.00031906    0.00031906    0.00031906         0            1     
[art]:TriggerResults:TriggerResultInserter            6.6834e-05    6.6834e-05    6.6834e-05    6.6834e-05         0            1     
end_path:out1:RootOutput                              2.2919e-05    2.2919e-05    2.2919e-05    2.2919e-05         0            1     
end_path:out1:RootOutput(write)                        0.156995      0.156995      0.156995      0.156995          0            1     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 7315.3 MB
  Peak resident set size usage (VmHWM): 1399.3 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Nov-2024 01:32:44 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- StdException BEGIN
      An exception was thrown while processing module CVNEvaluator/cvneva run: 6467376 subRun: 1 event: 20602
      CHECK failed: it != end(): key not found: serving_default
    ---- StdException END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 1 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-1465-cd48-91f0-ea61.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467376_206_20231207T070654Z_gen_g4_detsim_hitreco__20240510T051457Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467376_206_20231207T070654Z_gen_g4_detsim_hitreco__20240510T051457Z_reco2_reco_2024-11-23T_012224Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work