Jobsub ID 108687.13@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108687.13@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 02:25:29
Site	UK_Imperial
Entry	DUNE_T2_UK_London_IC_ceprod00
Last heartbeat	2024-11-23 02:39:58
From worker node	Hostname	wm04.grid.hep.ph.ic.ac.uk
	cpuinfo	AMD EPYC 7742 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 02:30:59
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50598687_285_20231204T211904Z_gen_g4_detsim_hitreco__20240509T221106Z_reco2.root
Jobscript	Exit code	0
	Real time	8m (520s)
	CPU time	2m (124s = 23%)
Outputting started	2024-11-23 02:39:40
Output files
Finished	2024-11-23 02:39:58
List job events Wrapper job log
Jobscript log (last 10,000 characters)

ov-2024 02:38:32 GMT run: 50598687 subRun: 1 event: 28502
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 02:38:32 GMT run: 50598687 subRun: 1 event: 28502
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 02:38:32 GMT run: 50598687 subRun: 1 event: 28502
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 02:38:32 GMT run: 50598687 subRun: 1 event: 28502
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 02:38:32 GMT run: 50598687 subRun: 1 event: 28502
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 1401, 1284, 804
minwire 0: 1435
minwire 1: 1043
minwire 2: 1280
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Could not find serving_default in model signatures.
[libprotobuf FATAL /cvmfs/larsoft.opensciencegrid.org/products/protobuf/v3_21_12a/Linux64bit+3.10-2.17-e26/include/google/protobuf/map.h:1300] CHECK failed: it != end(): key not found: serving_default
23-Nov-2024 02:39:39 GMT  Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/dc/7f/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50598687_285_20231204T211904Z_gen_g4_detsim_hitreco__20240509T221106Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                            0.00758231      139.937       279.866       139.937       139.929         2     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00758231     0.0080637     0.0085451     0.0080637    0.000481395       2     
reco:linecluster:LineCluster                           0.0871492     0.130575      0.174001      0.130575      0.043426         2     
reco:trajcluster:TrajCluster                           0.0476307     0.218897      0.390163      0.218897      0.171266         2     
reco:pandora:StandardPandora                           0.992639       138.256       275.52        138.256       137.264         2     
reco:pandoraTrack:LArPandoraTrackCreation             0.000203535   0.00129302     0.0023825    0.00129302    0.00108948        2     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000315777   0.00179087    0.00326597    0.00179087    0.00147509        2     
reco:pandoracalo:Calorimetry                          0.000149262   0.000846059   0.00154286    0.000846059   0.000696797       2     
reco:pandorapid:Chi2ParticleID                        5.0195e-05    0.000444796   0.000839397   0.000444796   0.000394601       2     
reco:trkshowersplit:TrackShowerHits                   0.00235411     0.0156925     0.029031      0.0156925     0.0133384        2     
reco:pmtrack:PMAlgTrackMaker                           0.0149714     0.952047       1.88912      0.952047      0.937075         2     
reco:pmtrackcalo:Calorimetry                          0.00338786    0.00697169     0.0105555    0.00697169    0.00358383        2     
reco:pmtrackpid:Chi2ParticleID                        0.000756581   0.000884923   0.00101327    0.000884923   0.000128342       2     
reco:pmtrajfit:PMAlgTrajFitter                        0.000523691   0.00128443    0.00204517    0.00128443    0.000760739       2     
reco:pmtrajfitcalo:Calorimetry                        0.000112112   0.00011588    0.000119647   0.00011588    3.7675e-06        2     
reco:pmtrajfitpid:Chi2ParticleID                      4.9072e-05    7.54225e-05   0.000101773   7.54225e-05   2.63505e-05       2     
reco:pmtracktc:PMAlgTrackMaker                         0.0053433      1.6383        3.27126       1.6383        1.63296         2     
reco:pmtracktccalo:Calorimetry                        0.000202012   0.00642297     0.0126439    0.00642297    0.00622096        2     
reco:pmtracktcpid:Chi2ParticleID                      6.9872e-05    0.000449385   0.000828897   0.000449385   0.000379512       2     
reco:pmtrajfittc:PMAlgTrajFitter                       0.155221       1.82267       3.49012       1.82267       1.66745         2     
reco:pmtrajfittccalo:Calorimetry                      0.000498012    0.0146033     0.0287085     0.0146033     0.0141053        2     
reco:pmtrajfittcpid:Chi2ParticleID                    0.000109677   0.00075327    0.00139686    0.00075327    0.000643592       2     
reco:blurredcluster:BlurredClustering                  0.273687       1.13621       1.99872       1.13621      0.862519         2     
reco:emtrkmichelid:EmTrackMichelId                     0.987319       15.3322       29.6771       15.3322       14.3449         2     
reco:emshower:EMShower                                0.00151971     0.002837     0.00415429     0.002837     0.00131729        2     
reco:cvnmap:CVNMapper                                 0.00025939     0.0134302     0.0266009     0.0134302     0.0131708        2     
reco:cvneva:CVNEvaluator                              0.000259431   0.000259431   0.000259431   0.000259431        0            1     
reco:energyrecnumu:EnergyReco                         0.00618119    0.00618119    0.00618119    0.00618119         0            1     
reco:energyrecnue:EnergyReco                           0.0030911     0.0030911     0.0030911     0.0030911         0            1     
reco:energyrecnc:EnergyReco                            0.0034987     0.0034987     0.0034987     0.0034987         0            1     
reco:energyrecnumurange:EnergyReco                    0.00335601    0.00335601    0.00335601    0.00335601         0            1     
reco:energyrecnumumcs:EnergyReco                      0.00328336    0.00328336    0.00328336    0.00328336         0            1     
reco:opdec:Deconvolution                               0.254007      0.254007      0.254007      0.254007          0            1     
reco:ophitspe:OpHitFinderDeco                           2.33925       2.33925       2.33925       2.33925          0            1     
reco:opflash:OpFlashFinder                            0.00486891    0.00486891    0.00486891    0.00486891         0            1     
reco:rns:RandomNumberSaver                            0.000311509   0.000311509   0.000311509   0.000311509        0            1     
reco:anglereconue:NuAngularReco                       0.000940158   0.000940158   0.000940158   0.000940158        0            1     
reco:anglereconumu:NuAngularReco                      0.000152458   0.000152458   0.000152458   0.000152458        0            1     
reco:anglereconuepfps:NuAngularReco                   0.00138109    0.00138109    0.00138109    0.00138109         0            1     
reco:anglereconumupfps:NuAngularReco                  0.000190831   0.000190831   0.000190831   0.000190831        0            1     
[art]:TriggerResults:TriggerResultInserter            4.8563e-05    4.8563e-05    4.8563e-05    4.8563e-05         0            1     
end_path:out1:RootOutput                              1.6822e-05    1.6822e-05    1.6822e-05    1.6822e-05         0            1     
end_path:out1:RootOutput(write)                        0.120449      0.120449      0.120449      0.120449          0            1     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 37185 MB
  Peak resident set size usage (VmHWM): 1235.75 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Nov-2024 02:39:40 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- StdException BEGIN
      An exception was thrown while processing module CVNEvaluator/cvneva run: 50598687 subRun: 1 event: 28502
      CHECK failed: it != end(): key not found: serving_default
    ---- StdException END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 1 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-0bbd-4672-eff8-dca9.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50598687_285_20231204T211904Z_gen_g4_detsim_hitreco__20240509T221106Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50598687_285_20231204T211904Z_gen_g4_detsim_hitreco__20240509T221106Z_reco2_reco_2024-11-23T_023104Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work