Jobsub ID 301824.5@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301824.5@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:51:28
Site	UK_Imperial
Entry	DUNE_T2_UK_London_IC_ceprod03
Last heartbeat	2024-11-23 06:01:03
From worker node	Hostname	wm22.grid.hep.ph.ic.ac.uk
	cpuinfo	AMD EPYC 7742 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:52:23
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50620606_729_20231207T120858Z_gen_g4_detsim_hitreco__20240510T042030Z_reco2.root
Jobscript	Exit code	0
	Real time	8m (505s)
	CPU time	1m (61s = 12%)
Outputting started	2024-11-23 06:00:48
Output files
Finished	2024-11-23 06:01:03
List job events Wrapper job log
Jobscript log (last 10,000 characters)

G
%MSG-e pma::Track3D:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:00:04 GMT run: 50620606 subRun: 1 event: 72902
Hit sorting problem.
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:00:04 GMT run: 50620606 subRun: 1 event: 72902
1st wire C:0 T:9 P:2 W:538 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:00:04 GMT run: 50620606 subRun: 1 event: 72902
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:00:05 GMT run: 50620606 subRun: 1 event: 72902
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 516, 384, 120
minwire 0: 1183
minwire 1: 1236
minwire 2: 970
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Could not find serving_default in model signatures.
[libprotobuf FATAL /cvmfs/larsoft.opensciencegrid.org/products/protobuf/v3_21_12a/Linux64bit+3.10-2.17-e26/include/google/protobuf/map.h:1300] CHECK failed: it != end(): key not found: serving_default
23-Nov-2024 06:00:47 GMT  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/75/02/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50620606_729_20231207T120858Z_gen_g4_detsim_hitreco__20240510T042030Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.0351993      127.245       254.454       127.245       127.209         2     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0205492     0.0278742     0.0351993     0.0278742    0.00732507        2     
reco:linecluster:LineCluster                           0.193238       0.20188      0.210522       0.20188     0.00864223        2     
reco:trajcluster:TrajCluster                           0.0648861     0.0815519     0.0982177     0.0815519     0.0166658        2     
reco:pandora:StandardPandora                           0.155696       126.532       252.908       126.532       126.376         2     
reco:pandoraTrack:LArPandoraTrackCreation             0.000129163   0.00101246    0.00189576    0.00101246    0.000883296       2     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000385647   0.00144269    0.00249972    0.00144269    0.00105704        2     
reco:pandoracalo:Calorimetry                          0.000103375   0.000548168   0.000992962   0.000548168   0.000444794       2     
reco:pandorapid:Chi2ParticleID                        5.7208e-05    0.000288108   0.000519009   0.000288108   0.000230901       2     
reco:trkshowersplit:TrackShowerHits                   0.00162575     0.0019852    0.00234465     0.0019852    0.000359453       2     
reco:pmtrack:PMAlgTrackMaker                          0.00388295     0.471055      0.938228      0.471055      0.467172         2     
reco:pmtrackcalo:Calorimetry                          8.2345e-05     0.0026722    0.00526205     0.0026722    0.00258985        2     
reco:pmtrackpid:Chi2ParticleID                        3.3193e-05    0.000387706   0.00074222    0.000387706   0.000354513       2     
reco:pmtrajfit:PMAlgTrajFitter                        0.000541612   0.000904055    0.0012665    0.000904055   0.000362443       2     
reco:pmtrajfitcalo:Calorimetry                        6.8269e-05    6.90905e-05   6.9912e-05    6.90905e-05    8.215e-07        2     
reco:pmtrajfitpid:Chi2ParticleID                      2.7232e-05    2.86795e-05   3.0127e-05    2.86795e-05   1.4475e-06        2     
reco:pmtracktc:PMAlgTrackMaker                        0.00132197     0.143825      0.286328      0.143825      0.142503         2     
reco:pmtracktccalo:Calorimetry                        7.0524e-05    0.00169301     0.0033155    0.00169301    0.00162249        2     
reco:pmtracktcpid:Chi2ParticleID                      3.0217e-05    8.5837e-05    0.000141457   8.5837e-05     5.562e-05        2     
reco:pmtrajfittc:PMAlgTrajFitter                      0.000182915    0.543194       1.08621      0.543194      0.543012         2     
reco:pmtrajfittccalo:Calorimetry                      6.2718e-05    0.00143612    0.00280953    0.00143612    0.00137341        2     
reco:pmtrajfittcpid:Chi2ParticleID                    2.7281e-05    7.21655e-05   0.00011705    7.21655e-05   4.48845e-05       2     
reco:blurredcluster:BlurredClustering                  0.0973201     0.442471      0.787621      0.442471      0.345151         2     
reco:emtrkmichelid:EmTrackMichelId                     0.386197       3.71363       7.04106       3.71363       3.32743         2     
reco:emshower:EMShower                                0.000712415   0.00183631    0.00296021    0.00183631     0.0011239        2     
reco:cvnmap:CVNMapper                                 0.000184318   0.00916642     0.0181485    0.00916642     0.0089821        2     
reco:cvneva:CVNEvaluator                              0.000178146   0.000178146   0.000178146   0.000178146        0            1     
reco:energyrecnumu:EnergyReco                          0.0046958     0.0046958     0.0046958     0.0046958         0            1     
reco:energyrecnue:EnergyReco                          0.00229041    0.00229041    0.00229041    0.00229041         0            1     
reco:energyrecnc:EnergyReco                            0.0023477     0.0023477     0.0023477     0.0023477         0            1     
reco:energyrecnumurange:EnergyReco                    0.00225106    0.00225106    0.00225106    0.00225106         0            1     
reco:energyrecnumumcs:EnergyReco                      0.00227719    0.00227719    0.00227719    0.00227719         0            1     
reco:opdec:Deconvolution                               0.151819      0.151819      0.151819      0.151819          0            1     
reco:ophitspe:OpHitFinderDeco                          0.545598      0.545598      0.545598      0.545598          0            1     
reco:opflash:OpFlashFinder                            0.00300527    0.00300527    0.00300527    0.00300527         0            1     
reco:rns:RandomNumberSaver                            0.000242527   0.000242527   0.000242527   0.000242527        0            1     
reco:anglereconue:NuAngularReco                       0.000734194   0.000734194   0.000734194   0.000734194        0            1     
reco:anglereconumu:NuAngularReco                      0.000114806   0.000114806   0.000114806   0.000114806        0            1     
reco:anglereconuepfps:NuAngularReco                   0.000979046   0.000979046   0.000979046   0.000979046        0            1     
reco:anglereconumupfps:NuAngularReco                  0.000147197   0.000147197   0.000147197   0.000147197        0            1     
[art]:TriggerResults:TriggerResultInserter            3.9395e-05    3.9395e-05    3.9395e-05    3.9395e-05         0            1     
end_path:out1:RootOutput                              1.3005e-05    1.3005e-05    1.3005e-05    1.3005e-05         0            1     
end_path:out1:RootOutput(write)                        0.0481058     0.0481058     0.0481058     0.0481058         0            1     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 37136.2 MB
  Peak resident set size usage (VmHWM): 1147.18 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Nov-2024 06:00:47 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- StdException BEGIN
      An exception was thrown while processing module CVNEvaluator/cvneva run: 50620606 subRun: 1 event: 72902
      CHECK failed: it != end(): key not found: serving_default
    ---- StdException END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 1 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-6a24-7e9f-5b60-88bc.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50620606_729_20231207T120858Z_gen_g4_detsim_hitreco__20240510T042030Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50620606_729_20231207T120858Z_gen_g4_detsim_hitreco__20240510T042030Z_reco2_reco_2024-11-23T_055226Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work