Jobsub ID 108835.11@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108835.11@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:45:04
Site	UK_Imperial
Entry	DUNE_T2_UK_London_IC_ceprod01
Last heartbeat	2024-11-23 05:55:34
From worker node	Hostname	wj47.grid.hep.ph.ic.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:46:25
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_1025_20231203T154556Z_gen_g4_detsim_hitreco__20240508T073340Z_reco2.root
Jobscript	Exit code	0
	Real time	8m (533s)
	CPU time	2m (138s = 25%)
Outputting started	2024-11-23 05:55:18
Output files
Finished	2024-11-23 05:55:34
List job events Wrapper job log
Jobscript log (last 10,000 characters)

G-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:54:24 GMT run: 74511302 subRun: 1 event: 102502
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:54:24 GMT run: 74511302 subRun: 1 event: 102502
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:54:24 GMT run: 74511302 subRun: 1 event: 102502
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:54:24 GMT run: 74511302 subRun: 1 event: 102502
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:54:24 GMT run: 74511302 subRun: 1 event: 102502
TuneFullTree failed.
%MSG
Boundary wire vector sizes: 422, 418, 350
minwire 0: 542
minwire 1: 899
minwire 2: 563
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Could not find serving_default in model signatures.
[libprotobuf FATAL /cvmfs/larsoft.opensciencegrid.org/products/protobuf/v3_21_12a/Linux64bit+3.10-2.17-e26/include/google/protobuf/map.h:1300] CHECK failed: it != end(): key not found: serving_default
23-Nov-2024 05:55:17 GMT  Closed input file "root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/b9/f7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_1025_20231203T154556Z_gen_g4_detsim_hitreco__20240508T073340Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                            0.00935951      123.106       246.203       123.106       123.097         2     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00935951     0.0252342     0.0411089     0.0252342     0.0158747        2     
reco:linecluster:LineCluster                           0.243074      0.322172       0.40127      0.322172      0.079098         2     
reco:trajcluster:TrajCluster                           0.0957892     0.181133      0.266478      0.181133      0.0853441        2     
reco:pandora:StandardPandora                           0.450742       121.456       242.462       121.456       121.005         2     
reco:pandoraTrack:LArPandoraTrackCreation             0.000246633   0.00143894    0.00263124    0.00143894     0.0011923        2     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000342612   0.00213895    0.00393529    0.00213895    0.00179634        2     
reco:pandoracalo:Calorimetry                          0.000210097   0.000874008   0.00153792    0.000874008   0.000663911       2     
reco:pandorapid:Chi2ParticleID                        8.6797e-05    0.000491788   0.000896779   0.000491788   0.000404991       2     
reco:trkshowersplit:TrackShowerHits                   0.00250728    0.00614962    0.00979196    0.00614962    0.00364234        2     
reco:pmtrack:PMAlgTrackMaker                           0.0862433     0.535625      0.985006      0.535625      0.449381         2     
reco:pmtrackcalo:Calorimetry                          0.00310123    0.00450996    0.00591869    0.00450996    0.00140873        2     
reco:pmtrackpid:Chi2ParticleID                        0.00050103    0.000809093   0.00111716    0.000809093   0.000308063       2     
reco:pmtrajfit:PMAlgTrajFitter                        0.000549245   0.00144539    0.00234153    0.00144539    0.000896142       2     
reco:pmtrajfitcalo:Calorimetry                        0.000155711   0.000156556   0.000157401   0.000156556    8.45e-07         2     
reco:pmtrajfitpid:Chi2ParticleID                      6.1835e-05    6.62705e-05   7.0706e-05    6.62705e-05   4.4355e-06        2     
reco:pmtracktc:PMAlgTrackMaker                         0.178088      0.583898      0.989708      0.583898       0.40581         2     
reco:pmtracktccalo:Calorimetry                        0.000517071   0.00393212    0.00734718    0.00393212    0.00341505        2     
reco:pmtracktcpid:Chi2ParticleID                      0.000125493   0.00036317    0.000600847   0.00036317    0.000237677       2     
reco:pmtrajfittc:PMAlgTrajFitter                       0.0313049     0.906022       1.78074      0.906022      0.874717         2     
reco:pmtrajfittccalo:Calorimetry                      0.000462171   0.00483831    0.00921445    0.00483831    0.00437614        2     
reco:pmtrajfittcpid:Chi2ParticleID                    0.000127429   0.000410336   0.000693243   0.000410336   0.000282907       2     
reco:blurredcluster:BlurredClustering                  0.110048       1.6309        3.15175       1.6309        1.52085         2     
reco:emtrkmichelid:EmTrackMichelId                     0.547076       8.42396       16.3008       8.42396       7.87688         2     
reco:emshower:EMShower                                0.00108819    0.00265434     0.0042205    0.00265434    0.00156615        2     
reco:cvnmap:CVNMapper                                 0.000304739    0.0222066     0.0441084     0.0222066     0.0219018        2     
reco:cvneva:CVNEvaluator                              0.000288285   0.000288285   0.000288285   0.000288285        0            1     
reco:energyrecnumu:EnergyReco                         0.00714245    0.00714245    0.00714245    0.00714245         0            1     
reco:energyrecnue:EnergyReco                          0.00398096    0.00398096    0.00398096    0.00398096         0            1     
reco:energyrecnc:EnergyReco                           0.00397516    0.00397516    0.00397516    0.00397516         0            1     
reco:energyrecnumurange:EnergyReco                    0.00397847    0.00397847    0.00397847    0.00397847         0            1     
reco:energyrecnumumcs:EnergyReco                      0.00393231    0.00393231    0.00393231    0.00393231         0            1     
reco:opdec:Deconvolution                                0.50302       0.50302       0.50302       0.50302          0            1     
reco:ophitspe:OpHitFinderDeco                           1.75023       1.75023       1.75023       1.75023          0            1     
reco:opflash:OpFlashFinder                            0.00451322    0.00451322    0.00451322    0.00451322         0            1     
reco:rns:RandomNumberSaver                            0.000367439   0.000367439   0.000367439   0.000367439        0            1     
reco:anglereconue:NuAngularReco                       0.00110265    0.00110265    0.00110265    0.00110265         0            1     
reco:anglereconumu:NuAngularReco                      0.000246263   0.000246263   0.000246263   0.000246263        0            1     
reco:anglereconuepfps:NuAngularReco                   0.00162714    0.00162714    0.00162714    0.00162714         0            1     
reco:anglereconumupfps:NuAngularReco                  0.000322031   0.000322031   0.000322031   0.000322031        0            1     
[art]:TriggerResults:TriggerResultInserter            8.2298e-05    8.2298e-05    8.2298e-05    8.2298e-05         0            1     
end_path:out1:RootOutput                              2.0597e-05    2.0597e-05    2.0597e-05    2.0597e-05         0            1     
end_path:out1:RootOutput(write)                        0.0969546     0.0969546     0.0969546     0.0969546         0            1     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 7163.64 MB
  Peak resident set size usage (VmHWM): 1283.08 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Nov-2024 05:55:17 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- StdException BEGIN
      An exception was thrown while processing module CVNEvaluator/cvneva run: 74511302 subRun: 1 event: 102502
      CHECK failed: it != end(): key not found: serving_default
    ---- StdException END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 1 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-fde2-4005-02c1-07e8.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_1025_20231203T154556Z_gen_g4_detsim_hitreco__20240508T073340Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_1025_20231203T154556Z_gen_g4_detsim_hitreco__20240508T073340Z_reco2_reco_2024-11-23T_054628Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work