Jobsub ID 301751.58@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301751.58@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:14:46
Site	UK_Imperial
Entry	DUNE_T2_UK_London_IC_ceprod01
Last heartbeat	2024-11-23 05:28:20
From worker node	Hostname	wh19.grid.hep.ph.ic.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:16:30
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_711_20231203T045522Z_gen_g4_detsim_hitreco__20240508T073232Z_reco2.root
Jobscript	Exit code	0
	Real time	11m (694s)
	CPU time	1m (92s = 13%)
Outputting started	2024-11-23 05:28:05
Output files
Finished	2024-11-23 05:28:20
List job events Wrapper job log
Jobscript log (last 10,000 characters)

rack3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:27:17 GMT run: 66006611 subRun: 1 event: 71102
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:27:17 GMT run: 66006611 subRun: 1 event: 71102
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:27:17 GMT run: 66006611 subRun: 1 event: 71102
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:27:17 GMT run: 66006611 subRun: 1 event: 71102
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:27:17 GMT run: 66006611 subRun: 1 event: 71102
TuneFullTree failed.
%MSG
Boundary wire vector sizes: 384, 573, 385
minwire 0: 808
minwire 1: 1567
minwire 2: 632
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Could not find serving_default in model signatures.
[libprotobuf FATAL /cvmfs/larsoft.opensciencegrid.org/products/protobuf/v3_21_12a/Linux64bit+3.10-2.17-e26/include/google/protobuf/map.h:1300] CHECK failed: it != end(): key not found: serving_default
23-Nov-2024 05:28:04 GMT  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/fd/eb/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_711_20231203T045522Z_gen_g4_detsim_hitreco__20240508T073232Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.020867       122.033       244.046       122.033       122.013         2     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0208384     0.0208527     0.020867      0.0208527    1.4286e-05        2     
reco:linecluster:LineCluster                           0.196653      0.247877      0.299102      0.247877      0.0512245        2     
reco:trajcluster:TrajCluster                           0.0725057     0.156127      0.239748      0.156127      0.083621         2     
reco:pandora:StandardPandora                           0.315281       120.594       240.873       120.594       120.279         2     
reco:pandoraTrack:LArPandoraTrackCreation             0.000243745   0.00115961    0.00207548    0.00115961    0.000915866       2     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000476064   0.00161228    0.00274849    0.00161228    0.00113621        2     
reco:pandoracalo:Calorimetry                          0.000187158   0.000662294   0.00113743    0.000662294   0.000475136       2     
reco:pandorapid:Chi2ParticleID                        0.000103644   0.00036107    0.000618496   0.00036107    0.000257426       2     
reco:trkshowersplit:TrackShowerHits                   0.00181597    0.00610439     0.0103928    0.00610439    0.00428842        2     
reco:pmtrack:PMAlgTrackMaker                           0.0199256     0.683305       1.34668      0.683305      0.663379         2     
reco:pmtrackcalo:Calorimetry                          0.00221251    0.00266659    0.00312067    0.00266659    0.000454083       2     
reco:pmtrackpid:Chi2ParticleID                        0.000467522   0.000599363   0.000731204   0.000599363   0.000131841       2     
reco:pmtrajfit:PMAlgTrajFitter                        0.000685491   0.00112388    0.00156228    0.00112388    0.000438393       2     
reco:pmtrajfitcalo:Calorimetry                        9.9359e-05    0.000124437   0.000149515   0.000124437   2.5078e-05        2     
reco:pmtrajfitpid:Chi2ParticleID                      4.2829e-05    6.31595e-05    8.349e-05    6.31595e-05   2.03305e-05       2     
reco:pmtracktc:PMAlgTrackMaker                        0.00805374     0.461771      0.915489      0.461771      0.453717         2     
reco:pmtracktccalo:Calorimetry                        0.000375722   0.00288999    0.00540426    0.00288999    0.00251427        2     
reco:pmtracktcpid:Chi2ParticleID                      0.000105253   0.000252886   0.000400519   0.000252886   0.000147633       2     
reco:pmtrajfittc:PMAlgTrajFitter                       0.0102483     0.824254       1.63826      0.824254      0.814005         2     
reco:pmtrajfittccalo:Calorimetry                      0.000233291   0.00366657    0.00709986    0.00366657    0.00343328        2     
reco:pmtrajfittcpid:Chi2ParticleID                    7.8363e-05    0.000292408   0.000506452   0.000292408   0.000214045       2     
reco:blurredcluster:BlurredClustering                  0.453284      0.628143      0.803002      0.628143      0.174859         2     
reco:emtrkmichelid:EmTrackMichelId                     0.634129       5.1951        9.75607       5.1951        4.56097         2     
reco:emshower:EMShower                                0.000682088   0.00177364    0.00286519    0.00177364    0.00109155        2     
reco:cvnmap:CVNMapper                                 0.000219563    0.0121784     0.0241373     0.0121784     0.0119589        2     
reco:cvneva:CVNEvaluator                              0.000199606   0.000199606   0.000199606   0.000199606        0            1     
reco:energyrecnumu:EnergyReco                         0.00482006    0.00482006    0.00482006    0.00482006         0            1     
reco:energyrecnue:EnergyReco                           0.0025745     0.0025745     0.0025745     0.0025745         0            1     
reco:energyrecnc:EnergyReco                           0.00254638    0.00254638    0.00254638    0.00254638         0            1     
reco:energyrecnumurange:EnergyReco                    0.00248053    0.00248053    0.00248053    0.00248053         0            1     
reco:energyrecnumumcs:EnergyReco                      0.00249604    0.00249604    0.00249604    0.00249604         0            1     
reco:opdec:Deconvolution                               0.240493      0.240493      0.240493      0.240493          0            1     
reco:ophitspe:OpHitFinderDeco                           1.03818       1.03818       1.03818       1.03818          0            1     
reco:opflash:OpFlashFinder                            0.00330009    0.00330009    0.00330009    0.00330009         0            1     
reco:rns:RandomNumberSaver                            0.00026376    0.00026376    0.00026376    0.00026376         0            1     
reco:anglereconue:NuAngularReco                       0.000747173   0.000747173   0.000747173   0.000747173        0            1     
reco:anglereconumu:NuAngularReco                      0.000167759   0.000167759   0.000167759   0.000167759        0            1     
reco:anglereconuepfps:NuAngularReco                   0.00106711    0.00106711    0.00106711    0.00106711         0            1     
reco:anglereconumupfps:NuAngularReco                  0.000208822   0.000208822   0.000208822   0.000208822        0            1     
[art]:TriggerResults:TriggerResultInserter             4.31e-05      4.31e-05      4.31e-05      4.31e-05          0            1     
end_path:out1:RootOutput                              1.3746e-05    1.3746e-05    1.3746e-05    1.3746e-05         0            1     
end_path:out1:RootOutput(write)                        0.0893763     0.0893763     0.0893763     0.0893763         0            1     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 7071.4 MB
  Peak resident set size usage (VmHWM): 1154.06 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Nov-2024 05:28:05 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- StdException BEGIN
      An exception was thrown while processing module CVNEvaluator/cvneva run: 66006611 subRun: 1 event: 71102
      CHECK failed: it != end(): key not found: serving_default
    ---- StdException END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 1 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-4a52-9be9-bca4-8646.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_711_20231203T045522Z_gen_g4_detsim_hitreco__20240508T073232Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_711_20231203T045522Z_gen_g4_detsim_hitreco__20240508T073232Z_reco2_reco_2024-11-23T_051633Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work