Jobsub ID 301639.123@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301639.123@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 02:10:16
Site	UK_Imperial
Entry	DUNE_T2_UK_London_IC_ceprod00
Last heartbeat	2024-11-23 02:25:26
From worker node	Hostname	wk05.grid.hep.ph.ic.ac.uk
	cpuinfo	Intel(R) Xeon(R) Gold 6130 CPU @ 2.10GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 02:17:17
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6398928_111_20231202T043702Z_gen_g4_detsim_hitreco__20240507T202955Z_reco2.root
Jobscript	Exit code	0
	Real time	7m (470s)
	CPU time	2m (122s = 25%)
Outputting started	2024-11-23 02:25:07
Output files
Finished	2024-11-23 02:25:26
List job events Wrapper job log
Jobscript log (last 10,000 characters)

rt.
%MSG
%MSG-e pma::Track3D:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 02:24:04 GMT run: 6398928 subRun: 1 event: 11103
Flip, endpoint closer to vStart.
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 02:24:05 GMT run: 6398928 subRun: 1 event: 11103
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 02:24:05 GMT run: 6398928 subRun: 1 event: 11103
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 02:24:06 GMT run: 6398928 subRun: 1 event: 11103
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 600, 652, 537
minwire 0: 1960
minwire 1: 1364
minwire 2: 1486
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Could not find serving_default in model signatures.
[libprotobuf FATAL /cvmfs/larsoft.opensciencegrid.org/products/protobuf/v3_21_12a/Linux64bit+3.10-2.17-e26/include/google/protobuf/map.h:1300] CHECK failed: it != end(): key not found: serving_default
23-Nov-2024 02:25:06 GMT  Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/2a/bd/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6398928_111_20231202T043702Z_gen_g4_detsim_hitreco__20240507T202955Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.0136791      82.4128       245.314       1.91107       115.191         3     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00750853    0.00979244     0.0136791    0.00818969    0.00276232        3     
reco:linecluster:LineCluster                           0.0714151     0.0843052     0.0951974     0.0863031     0.0098113        3     
reco:trajcluster:TrajCluster                           0.0210448     0.0542746     0.102943      0.0388363     0.0351717        3     
reco:pandora:StandardPandora                           0.0339945      80.3938       240.893       0.25463       113.49          3     
reco:pandoraTrack:LArPandoraTrackCreation             0.000194677   0.00105558    0.00274062    0.000231442    0.0011916        3     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000274512   0.00162783    0.00432666    0.000282324   0.00190836        3     
reco:pandoracalo:Calorimetry                          0.000155635   0.000738413   0.00188578    0.000173825   0.000811344       3     
reco:pandorapid:Chi2ParticleID                        5.5629e-05    0.000366771   0.000983926   6.0757e-05     0.0004364        3     
reco:trkshowersplit:TrackShowerHits                   0.000446454   0.00296907    0.00559326     0.0028675     0.0021024        3     
reco:pmtrack:PMAlgTrackMaker                          0.00207214     0.465242       1.37423      0.0194296     0.642787         3     
reco:pmtrackcalo:Calorimetry                          0.000128949   0.00467728     0.0108093    0.00309355    0.00450176        3     
reco:pmtrackpid:Chi2ParticleID                         5.301e-05    0.000478737    0.001149      0.0002342    0.000479685       3     
reco:pmtrajfit:PMAlgTrajFitter                        0.000313521   0.00107616    0.00230885    0.000606104   0.000879792       3     
reco:pmtrajfitcalo:Calorimetry                        0.000111636   0.000136872   0.000162214   0.000136765   2.06485e-05       3     
reco:pmtrajfitpid:Chi2ParticleID                      4.3683e-05    4.8231e-05    5.2386e-05    4.8624e-05    3.56384e-06       3     
reco:pmtracktc:PMAlgTrackMaker                        0.00221446     0.460246       1.37395     0.00457214     0.646088         3     
reco:pmtracktccalo:Calorimetry                        0.000120849    0.0039444     0.0114519    0.000260474   0.00530889        3     
reco:pmtracktcpid:Chi2ParticleID                      4.7257e-05    0.000136354   0.000279701   8.2105e-05    0.000102355       3     
reco:pmtrajfittc:PMAlgTrajFitter                      0.00746601     0.366298       1.02508      0.0663456     0.466451         3     
reco:pmtrajfittccalo:Calorimetry                      0.000284968   0.00269651    0.00744299    0.000361584   0.00335641        3     
reco:pmtrajfittcpid:Chi2ParticleID                    9.1742e-05    0.00015641    0.000280504   9.6984e-05    8.77738e-05       3     
reco:blurredcluster:BlurredClustering                  0.0449093      1.19654       3.46065      0.0840536      1.60105         3     
reco:emtrkmichelid:EmTrackMichelId                      0.44681       7.95544       22.7857      0.633804       10.4869         3     
reco:emshower:EMShower                                0.000473601   0.00212773    0.00442864    0.00148094    0.00167816        3     
reco:cvnmap:CVNMapper                                 3.5293e-05     0.0163519     0.0487265    0.000293947    0.0228926        3     
reco:cvneva:CVNEvaluator                              3.2285e-05    0.000165629   0.000298972   0.000165629   0.000133343       2     
reco:energyrecnumu:EnergyReco                         0.00391628    0.00545539     0.0069945    0.00545539    0.00153911        2     
reco:energyrecnue:EnergyReco                          0.00391198    0.00398314    0.00405431    0.00398314    7.11655e-05       2     
reco:energyrecnc:EnergyReco                           0.00387282    0.00395257    0.00403232    0.00395257    7.9751e-05        2     
reco:energyrecnumurange:EnergyReco                     0.0038959    0.00390725    0.00391859    0.00390725    1.1342e-05        2     
reco:energyrecnumumcs:EnergyReco                      0.00388496    0.00390806    0.00393116    0.00390806    2.30965e-05       2     
reco:opdec:Deconvolution                               0.0811766     0.174768       0.26836      0.174768      0.0935915        2     
reco:ophitspe:OpHitFinderDeco                          0.903539       2.09194       3.28035       2.09194       1.1884          2     
reco:opflash:OpFlashFinder                            0.000549602   0.00323552    0.00592144    0.00323552    0.00268592        2     
reco:rns:RandomNumberSaver                            4.2517e-05    0.000276222   0.000509926   0.000276222   0.000233705       2     
reco:anglereconue:NuAngularReco                       0.000323909   0.000860194   0.00139648    0.000860194   0.000536285       2     
reco:anglereconumu:NuAngularReco                      0.000114824   0.000188709   0.000262595   0.000188709   7.38855e-05       2     
reco:anglereconuepfps:NuAngularReco                   0.000215212   0.00107136    0.00192752    0.00107136    0.000856152       2     
reco:anglereconumupfps:NuAngularReco                  0.000164537   0.000230574   0.000296611   0.000230574   6.6037e-05        2     
[art]:TriggerResults:TriggerResultInserter            1.4312e-05    3.5836e-05     5.736e-05    3.5836e-05    2.1524e-05        2     
end_path:out1:RootOutput                               6.477e-06    1.5048e-05    2.3619e-05    1.5048e-05     8.571e-06        2     
end_path:out1:RootOutput(write)                        0.0370858     0.0661722     0.0952586     0.0661722     0.0290864        2     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 11645.6 MB
  Peak resident set size usage (VmHWM): 1437.63 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Nov-2024 02:25:06 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- StdException BEGIN
      An exception was thrown while processing module CVNEvaluator/cvneva run: 6398928 subRun: 1 event: 11103
      CHECK failed: it != end(): key not found: serving_default
    ---- StdException END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 2 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-6a80-e1e6-0560-6f32.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6398928_111_20231202T043702Z_gen_g4_detsim_hitreco__20240507T202955Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6398928_111_20231202T043702Z_gen_g4_detsim_hitreco__20240507T202955Z_reco2_reco_2024-11-23T_021722Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work