Jobsub ID 108779.26@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108779.26@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:04:07
Site	UK_Imperial
Entry	DUNE_T2_UK_London_IC_ceprod00
Last heartbeat	2024-11-23 05:12:01
From worker node	Hostname	wm29.grid.hep.ph.ic.ac.uk
	cpuinfo	AMD EPYC 7742 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:04:47
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65974923_471_20231201T081723Z_gen_g4_detsim_hitreco__20240507T192643Z_reco2.root
Jobscript	Exit code	0
	Real time	6m (415s)
	CPU time	1m (91s = 21%)
Outputting started	2024-11-23 05:11:43
Output files
Finished	2024-11-23 05:12:01
List job events Wrapper job log
Jobscript log (last 10,000 characters)

24 05:11:13 GMT run: 65974923 subRun: 1 event: 47102
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:11:13 GMT run: 65974923 subRun: 1 event: 47102
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:11:13 GMT run: 65974923 subRun: 1 event: 47102
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 55, 55, 47
minwire 0: 2004
minwire 1: 303
minwire 2: 2179
Used alternate method to get min and max wires due to vertex determination failure: 331, 530
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max wires due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max wires due to vertex determination failure: 385, 584
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Could not find serving_default in model signatures.
[libprotobuf FATAL /cvmfs/larsoft.opensciencegrid.org/products/protobuf/v3_21_12a/Linux64bit+3.10-2.17-e26/include/google/protobuf/map.h:1300] CHECK failed: it != end(): key not found: serving_default
23-Nov-2024 05:11:42 GMT  Closed input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/62/43/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65974923_471_20231201T081723Z_gen_g4_detsim_hitreco__20240507T192643Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                            0.00314611      118.783       237.564       118.783       118.78          2     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00314611    0.00779491     0.0124437    0.00779491     0.0046488        2     
reco:linecluster:LineCluster                           0.0382035     0.0787507     0.119298      0.0787507     0.0405472        2     
reco:trajcluster:TrajCluster                           0.018771      0.0189132     0.0190555     0.0189132    0.000142265       2     
reco:pandora:StandardPandora                           0.0553998      117.354       234.653       117.354       117.299         2     
reco:pandoraTrack:LArPandoraTrackCreation             0.000175331   0.00152482    0.00287431    0.00152482    0.00134949        2     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000259731   0.00211443    0.00396912    0.00211443    0.00185469        2     
reco:pandoracalo:Calorimetry                          0.000147088   0.000893478   0.00163987    0.000893478   0.00074639        2     
reco:pandorapid:Chi2ParticleID                        6.1937e-05    0.00046827    0.000874603   0.00046827    0.000406333       2     
reco:trkshowersplit:TrackShowerHits                   0.000367955   0.00150351    0.00263907    0.00150351    0.00113556        2     
reco:pmtrack:PMAlgTrackMaker                           0.0066445     0.0129755     0.0193065     0.0129755    0.00633101        2     
reco:pmtrackcalo:Calorimetry                          0.000141077   0.00166662    0.00319216    0.00166662    0.00152554        2     
reco:pmtrackpid:Chi2ParticleID                        5.4774e-05    0.00054792    0.00104107    0.00054792    0.000493147       2     
reco:pmtrajfit:PMAlgTrajFitter                        0.000424662   0.00125104    0.00207741    0.00125104    0.000826376       2     
reco:pmtrajfitcalo:Calorimetry                        0.000101602   0.000103747   0.000105891   0.000103747   2.1445e-06        2     
reco:pmtrajfitpid:Chi2ParticleID                      4.3632e-05    4.4163e-05    4.4694e-05    4.4163e-05     5.31e-07         2     
reco:pmtracktc:PMAlgTrackMaker                        0.00182973    0.00755739     0.0132851    0.00755739    0.00572766        2     
reco:pmtracktccalo:Calorimetry                        9.7684e-05    0.000335293   0.000572902   0.000335293   0.000237609       2     
reco:pmtracktcpid:Chi2ParticleID                      4.2491e-05    8.6534e-05    0.000130577   8.6534e-05    4.4043e-05        2     
reco:pmtrajfittc:PMAlgTrajFitter                      0.000381812    0.0929923     0.185603      0.0929923     0.0926105        2     
reco:pmtrajfittccalo:Calorimetry                      9.6092e-05    0.00031709    0.000538087   0.00031709    0.000220997       2     
reco:pmtrajfittcpid:Chi2ParticleID                    4.3974e-05    8.9069e-05    0.000134164   8.9069e-05    4.5095e-05        2     
reco:blurredcluster:BlurredClustering                  0.0400311     0.806054       1.57208      0.806054      0.766023         2     
reco:emtrkmichelid:EmTrackMichelId                     0.404294       1.01926       1.63423       1.01926      0.614967         2     
reco:emshower:EMShower                                0.000265061   0.00220329    0.00414152    0.00220329    0.00193823        2     
reco:cvnmap:CVNMapper                                 0.000268928    0.0145172     0.0287655     0.0145172     0.0142483        2     
reco:cvneva:CVNEvaluator                              0.000250634   0.000250634   0.000250634   0.000250634        0            1     
reco:energyrecnumu:EnergyReco                         0.00652432    0.00652432    0.00652432    0.00652432         0            1     
reco:energyrecnue:EnergyReco                          0.00345641    0.00345641    0.00345641    0.00345641         0            1     
reco:energyrecnc:EnergyReco                           0.00358704    0.00358704    0.00358704    0.00358704         0            1     
reco:energyrecnumurange:EnergyReco                    0.00365983    0.00365983    0.00365983    0.00365983         0            1     
reco:energyrecnumumcs:EnergyReco                      0.00346852    0.00346852    0.00346852    0.00346852         0            1     
reco:opdec:Deconvolution                               0.324619      0.324619      0.324619      0.324619          0            1     
reco:ophitspe:OpHitFinderDeco                           1.86716       1.86716       1.86716       1.86716          0            1     
reco:opflash:OpFlashFinder                             0.0030893     0.0030893     0.0030893     0.0030893         0            1     
reco:rns:RandomNumberSaver                            0.000299977   0.000299977   0.000299977   0.000299977        0            1     
reco:anglereconue:NuAngularReco                       0.000692268   0.000692268   0.000692268   0.000692268        0            1     
reco:anglereconumu:NuAngularReco                      0.000111762   0.000111762   0.000111762   0.000111762        0            1     
reco:anglereconuepfps:NuAngularReco                   0.00100938    0.00100938    0.00100938    0.00100938         0            1     
reco:anglereconumupfps:NuAngularReco                  0.000212532   0.000212532   0.000212532   0.000212532        0            1     
[art]:TriggerResults:TriggerResultInserter            4.0176e-05    4.0176e-05    4.0176e-05    4.0176e-05         0            1     
end_path:out1:RootOutput                              1.4017e-05    1.4017e-05    1.4017e-05    1.4017e-05         0            1     
end_path:out1:RootOutput(write)                        0.0676613     0.0676613     0.0676613     0.0676613         0            1     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 37206.5 MB
  Peak resident set size usage (VmHWM): 1260.53 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Nov-2024 05:11:42 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- StdException BEGIN
      An exception was thrown while processing module CVNEvaluator/cvneva run: 65974923 subRun: 1 event: 47102
      CHECK failed: it != end(): key not found: serving_default
    ---- StdException END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 1 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-e1ae-b1bd-4270-52d5.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65974923_471_20231201T081723Z_gen_g4_detsim_hitreco__20240507T192643Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65974923_471_20231201T081723Z_gen_g4_detsim_hitreco__20240507T192643Z_reco2_reco_2024-11-23T_050452Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work