Jobsub ID 108829.39@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108829.39@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:32:18
Site	UK_Imperial
Entry	DUNE_T2_UK_London_IC_ceprod01
Last heartbeat	2024-11-23 05:42:20
From worker node	Hostname	wj58.grid.hep.ph.ic.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:33:36
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481105_8_20231201T101436Z_gen_g4_detsim_hitreco__20240507T193611Z_reco2.root
Jobscript	Exit code	0
	Real time	8m (505s)
	CPU time	2m (128s = 25%)
Outputting started	2024-11-23 05:42:01
Output files
Finished	2024-11-23 05:42:20
List job events Wrapper job log
Jobscript log (last 10,000 characters)

PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:41:31 GMT run: 74481105 subRun: 1 event: 803
1st wire C:0 T:18 P:2 W:601 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:41:31 GMT run: 74481105 subRun: 1 event: 803
1st wire C:0 T:18 P:2 W:597 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:41:31 GMT run: 74481105 subRun: 1 event: 803
1st wire C:0 T:18 P:2 W:858 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:41:31 GMT run: 74481105 subRun: 1 event: 803
1st wire C:0 T:18 P:2 W:854 does not exist (max wire number: 480)
%MSG
Boundary wire vector sizes: 55, 57, 54
minwire 0: 2436
minwire 1: 967
minwire 2: 2275
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Could not find serving_default in model signatures.
[libprotobuf FATAL /cvmfs/larsoft.opensciencegrid.org/products/protobuf/v3_21_12a/Linux64bit+3.10-2.17-e26/include/google/protobuf/map.h:1300] CHECK failed: it != end(): key not found: serving_default
23-Nov-2024 05:41:59 GMT  Closed input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/6a/23/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481105_8_20231201T101436Z_gen_g4_detsim_hitreco__20240507T193611Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.248879       83.1171       246.337       2.7651        115.419         3     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00430017     0.085835      0.248879     0.00432548      0.11529         3     
reco:linecluster:LineCluster                           0.0758293      1.00438       2.19104      0.746286      0.882605         3     
reco:trajcluster:TrajCluster                          0.00939786     0.0191681     0.0286863     0.0194201    0.00787649        3     
reco:pandora:StandardPandora                           0.084334       79.8636       239.165      0.341821       112.643         3     
reco:pandoraTrack:LArPandoraTrackCreation             0.000258069   0.00140008    0.00368117    0.000260986   0.00161298        3     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000359502   0.00191111    0.00499878    0.00037504    0.00218332        3     
reco:pandoracalo:Calorimetry                          0.00020239    0.000876099   0.00215145    0.000274462   0.000902286       3     
reco:pandorapid:Chi2ParticleID                        9.4465e-05    0.000455394   0.00117107    0.000100649   0.000506065       3     
reco:trkshowersplit:TrackShowerHits                   0.000265927    0.0014649    0.00344154    0.00068722    0.00140824        3     
reco:pmtrack:PMAlgTrackMaker                          0.00289674     0.0156665     0.0244428     0.0196599    0.00923828        3     
reco:pmtrackcalo:Calorimetry                          0.000175146   0.00156812    0.00409032    0.000438903   0.00178671        3     
reco:pmtrackpid:Chi2ParticleID                        7.4118e-05    0.000522106   0.00135237    0.000139831   0.000587697       3     
reco:pmtrajfit:PMAlgTrajFitter                        0.000449363   0.00127551    0.00291743    0.000459735   0.00116102        3     
reco:pmtrajfitcalo:Calorimetry                        0.000160434   0.000175561   0.000193001   0.000173247   1.33957e-05       3     
reco:pmtrajfitpid:Chi2ParticleID                      7.0432e-05    7.90547e-05   8.8706e-05    7.8026e-05    7.4957e-06        3     
reco:pmtracktc:PMAlgTrackMaker                        0.00235617     0.0683158     0.194836     0.00775529     0.0894905        3     
reco:pmtracktccalo:Calorimetry                        0.000158025   0.000468111   0.000748912   0.000497395   0.000242116       3     
reco:pmtracktcpid:Chi2ParticleID                      7.1039e-05    0.00013675    0.000195243   0.000143968   5.09623e-05       3     
reco:pmtrajfittc:PMAlgTrajFitter                      0.000418228    0.0944486     0.274672     0.00825598     0.127477         3     
reco:pmtrajfittccalo:Calorimetry                      0.00016304    0.000574974   0.00109214    0.00046974    0.000386535       3     
reco:pmtrajfittcpid:Chi2ParticleID                    6.9619e-05    0.000154042   0.000249201   0.000143306   7.3706e-05        3     
reco:blurredcluster:BlurredClustering                  0.120421      0.440848      0.837721      0.364401      0.297784         3     
reco:emtrkmichelid:EmTrackMichelId                     0.637612       1.4347        2.22732       1.43917      0.649004         3     
reco:emshower:EMShower                                0.000459082   0.00212251    0.00531918    0.000589264   0.00226101        3     
reco:cvnmap:CVNMapper                                 5.7032e-05     0.0105753     0.0313079    0.000360996    0.0146607        3     
reco:cvneva:CVNEvaluator                              3.7316e-05    0.000187051   0.000336786   0.000187051   0.000149735       2     
reco:energyrecnumu:EnergyReco                         0.00399309    0.00676624     0.0095394    0.00676624    0.00277316        2     
reco:energyrecnue:EnergyReco                          0.00383186    0.00418289    0.00453392    0.00418289    0.000351026       2     
reco:energyrecnc:EnergyReco                           0.00389222    0.00418063    0.00446903    0.00418063    0.000288406       2     
reco:energyrecnumurange:EnergyReco                    0.00393269    0.00406116    0.00418962    0.00406116    0.000128465       2     
reco:energyrecnumumcs:EnergyReco                      0.00400169    0.00416127    0.00432085    0.00416127    0.000159579       2     
reco:opdec:Deconvolution                               0.285419       1.08389       1.88235       1.08389      0.798467         2     
reco:ophitspe:OpHitFinderDeco                           1.21339       1.84064       2.4679        1.84064      0.627255         2     
reco:opflash:OpFlashFinder                            0.000599662   0.00503567    0.00947169    0.00503567    0.00443601        2     
reco:rns:RandomNumberSaver                            4.0003e-05    0.000342252   0.000644501   0.000342252   0.000302249       2     
reco:anglereconue:NuAngularReco                       0.000288917   0.00101364    0.00173837    0.00101364    0.000724727       2     
reco:anglereconumu:NuAngularReco                      0.000158445   0.000236136   0.000313826   0.000236136   7.76905e-05       2     
reco:anglereconuepfps:NuAngularReco                   0.000262577   0.00161976    0.00297694    0.00161976    0.00135718        2     
reco:anglereconumupfps:NuAngularReco                  0.000232051   0.000299029   0.000366007   0.000299029   6.6978e-05        2     
[art]:TriggerResults:TriggerResultInserter             2.002e-05    4.8935e-05     7.785e-05    4.8935e-05    2.8915e-05        2     
end_path:out1:RootOutput                               4.646e-06    2.2265e-05    3.9884e-05    2.2265e-05    1.7619e-05        2     
end_path:out1:RootOutput(write)                        0.0437638     0.0819741     0.120184      0.0819741     0.0382102        2     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 7071.44 MB
  Peak resident set size usage (VmHWM): 1155.48 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Nov-2024 05:42:00 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- StdException BEGIN
      An exception was thrown while processing module CVNEvaluator/cvneva run: 74481105 subRun: 1 event: 803
      CHECK failed: it != end(): key not found: serving_default
    ---- StdException END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 2 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-7480-87dc-5c0e-6449.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481105_8_20231201T101436Z_gen_g4_detsim_hitreco__20240507T193611Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74481105_8_20231201T101436Z_gen_g4_detsim_hitreco__20240507T193611Z_reco2_reco_2024-11-23T_053340Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work