Jobsub ID 301858.19@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301858.19@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 06:59:37
Site	UK_Imperial
Entry	DUNE_T2_UK_London_IC_ceprod03
Last heartbeat	2024-11-23 07:16:55
From worker node	Hostname	wj23.grid.hep.ph.ic.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 07:01:15
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6209058_245_20231122T105853Z_gen_g4_detsim_hitreco__20240507T181804Z_reco2.root
Jobscript	Exit code	0
	Real time	15m (924s)
	CPU time	2m (132s = 14%)
Outputting started	2024-11-23 07:16:40
Output files
Finished	2024-11-23 07:16:55
List job events Wrapper job log
Jobscript log (last 10,000 characters)

pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:15:30 GMT run: 6209058 subRun: 1 event: 49003
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:15:30 GMT run: 6209058 subRun: 1 event: 49003
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:15:30 GMT run: 6209058 subRun: 1 event: 49003
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:15:30 GMT run: 6209058 subRun: 1 event: 49003
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:15:30 GMT run: 6209058 subRun: 1 event: 49003
TuneFullTree failed.
%MSG
Boundary wire vector sizes: 1949, 1359, 887
minwire 0: 265
minwire 1: 2242
minwire 2: 45
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Could not find serving_default in model signatures.
[libprotobuf FATAL /cvmfs/larsoft.opensciencegrid.org/products/protobuf/v3_21_12a/Linux64bit+3.10-2.17-e26/include/google/protobuf/map.h:1300] CHECK failed: it != end(): key not found: serving_default
23-Nov-2024 07:16:38 GMT  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/7c/10/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6209058_245_20231122T105853Z_gen_g4_detsim_hitreco__20240507T181804Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              0.38363       81.0934       238.964       3.93275       111.641         3     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0260274     0.148762       0.38363      0.0366279     0.166133         3     
reco:linecluster:LineCluster                           0.238211      0.374439       0.4833       0.401806      0.101911         3     
reco:trajcluster:TrajCluster                           0.0641148     0.280464      0.682438      0.0948389     0.284515         3     
reco:pandora:StandardPandora                           0.0887071      78.5656       234.447       1.16098       110.226         3     
reco:pandoraTrack:LArPandoraTrackCreation             0.000171981   0.000772586   0.00189823    0.000247544   0.00079655        3     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000264942    0.0011184    0.00270665    0.000383605   0.00112411        3     
reco:pandoracalo:Calorimetry                          0.000138267   0.000516028    0.001209     0.000200813   0.000490672       3     
reco:pandorapid:Chi2ParticleID                        6.9777e-05    0.000274638   0.000680041   7.4097e-05    0.000286668       3     
reco:trkshowersplit:TrackShowerHits                   0.000250258    0.0115991     0.0325896    0.00195752     0.0148589        3     
reco:pmtrack:PMAlgTrackMaker                           0.0025881      1.18157       3.53702     0.00510674      1.66556         3     
reco:pmtrackcalo:Calorimetry                          0.000119804   0.00521399     0.0152656    0.000256537    0.0071078        3     
reco:pmtrackpid:Chi2ParticleID                        4.9876e-05    0.000248199   0.000604932   8.9789e-05    0.000252774       3     
reco:pmtrajfit:PMAlgTrajFitter                        0.000445517   0.000845013   0.00154108    0.000548439   0.000493986       3     
reco:pmtrajfitcalo:Calorimetry                        9.8219e-05    0.000104796   0.000110751   0.000105418   5.13504e-06       3     
reco:pmtrajfitpid:Chi2ParticleID                      4.5663e-05    4.79667e-05   5.2208e-05    4.6029e-05    3.0028e-06        3     
reco:pmtracktc:PMAlgTrackMaker                        0.00675804     0.719518       2.13513      0.016671       1.00099         3     
reco:pmtracktccalo:Calorimetry                        0.000302687    0.0049742     0.0120772    0.00254267    0.00510517        3     
reco:pmtracktcpid:Chi2ParticleID                      9.5907e-05    0.000538518   0.00087623    0.000643417   0.000327087       3     
reco:pmtrajfittc:PMAlgTrajFitter                      0.00336483     0.668234       1.93136      0.0699748      0.89358         3     
reco:pmtrajfittccalo:Calorimetry                      0.000194294    0.0049728     0.0143744    0.00034967    0.00664827        3     
reco:pmtrajfittcpid:Chi2ParticleID                    7.8028e-05    0.000274853   0.000652138   9.4392e-05    0.000266865       3     
reco:blurredcluster:BlurredClustering                  0.195739       1.33371       2.73377       1.07162       1.05259         3     
reco:emtrkmichelid:EmTrackMichelId                      0.81069       11.141        31.6964       0.91601       14.5349         3     
reco:emshower:EMShower                                0.000305047    0.0021253    0.00324421    0.00282664    0.00129835        3     
reco:cvnmap:CVNMapper                                 4.8175e-05     0.0117024     0.0348321    0.000226846    0.0163553        3     
reco:cvneva:CVNEvaluator                              2.3259e-05    0.000123307   0.000223356   0.000123307   0.000100048       2     
reco:energyrecnumu:EnergyReco                         0.00309865    0.00405267    0.00500669    0.00405267    0.000954021       2     
reco:energyrecnue:EnergyReco                          0.00285555     0.0028845    0.00291346     0.0028845    2.8954e-05        2     
reco:energyrecnc:EnergyReco                           0.00284207    0.00289217    0.00294228    0.00289217    5.0104e-05        2     
reco:energyrecnumurange:EnergyReco                    0.00276855    0.00278907     0.0028096    0.00278907    2.05265e-05       2     
reco:energyrecnumumcs:EnergyReco                      0.00283592    0.00289725    0.00295859    0.00289725    6.13355e-05       2     
reco:opdec:Deconvolution                               0.357249      0.376066      0.394883      0.376066      0.0188168        2     
reco:ophitspe:OpHitFinderDeco                           1.52111       1.68893       1.85675       1.68893      0.167821         2     
reco:opflash:OpFlashFinder                            0.00138286    0.00266228     0.0039417    0.00266228    0.00127942        2     
reco:rns:RandomNumberSaver                            4.1158e-05    0.000178748   0.000316338   0.000178748   0.00013759        2     
reco:anglereconue:NuAngularReco                       0.000320169   0.000613194   0.000906218   0.000613194   0.000293024       2     
reco:anglereconumu:NuAngularReco                      0.00014271    0.000156952   0.000171194   0.000156952   1.4242e-05        2     
reco:anglereconuepfps:NuAngularReco                   0.000178974   0.000705359   0.00123175    0.000705359   0.000526386       2     
reco:anglereconumupfps:NuAngularReco                  0.000150275   0.00017875    0.000207226   0.00017875    2.84755e-05       2     
[art]:TriggerResults:TriggerResultInserter            1.6533e-05    3.53965e-05    5.426e-05    3.53965e-05   1.88635e-05       2     
end_path:out1:RootOutput                               4.09e-06     1.7578e-05    3.1066e-05    1.7578e-05    1.3488e-05        2     
end_path:out1:RootOutput(write)                        0.0980838     0.102795      0.107506      0.102795     0.00471126        2     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 7135.49 MB
  Peak resident set size usage (VmHWM): 1255.53 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Nov-2024 07:16:39 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- StdException BEGIN
      An exception was thrown while processing module CVNEvaluator/cvneva run: 6209058 subRun: 1 event: 49003
      CHECK failed: it != end(): key not found: serving_default
    ---- StdException END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 2 entries while sim::OpDetDivRecs_opdigi__detsim. has 200 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-c913-62fc-38f5-0c0c.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6209058_245_20231122T105853Z_gen_g4_detsim_hitreco__20240507T181804Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6209058_245_20231122T105853Z_gen_g4_detsim_hitreco__20240507T181804Z_reco2_reco_2024-11-23T_070119Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work