Jobsub ID 301820.136@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301820.136@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:41:53
Site	UK_Imperial
Entry	DUNE_T2_UK_London_IC_ceprod03
Last heartbeat	2024-11-23 05:51:34
From worker node	Hostname	wm22.grid.hep.ph.ic.ac.uk
	cpuinfo	AMD EPYC 7742 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:43:21
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6454800_726_20231207T111252Z_gen_g4_detsim_hitreco__20240510T052433Z_reco2.root
Jobscript	Exit code	0
	Real time	7m (478s)
	CPU time	0m (51s = 10%)
Outputting started	2024-11-23 05:51:20
Output files
Finished	2024-11-23 05:51:34
List job events Wrapper job log
Jobscript log (last 10,000 characters)

gTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:50:41 GMT run: 6454800 subRun: 1 event: 72602
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:50:41 GMT run: 6454800 subRun: 1 event: 72602
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:50:41 GMT run: 6454800 subRun: 1 event: 72602
1st wire C:0 T:23 P:2 W:522 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:50:41 GMT run: 6454800 subRun: 1 event: 72602
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 191, 197, 171
minwire 0: 2501
minwire 1: 433
minwire 2: 2529
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Could not find serving_default in model signatures.
[libprotobuf FATAL /cvmfs/larsoft.opensciencegrid.org/products/protobuf/v3_21_12a/Linux64bit+3.10-2.17-e26/include/google/protobuf/map.h:1300] CHECK failed: it != end(): key not found: serving_default
23-Nov-2024 05:51:19 GMT  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/7c/88/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6454800_726_20231207T111252Z_gen_g4_detsim_hitreco__20240510T052433Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.0388936       115.3        230.562        115.3        115.262         2     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0209125     0.0299031     0.0388936     0.0299031    0.00899054        2     
reco:linecluster:LineCluster                           0.203453      0.204548      0.205644      0.204548     0.00109514        2     
reco:trajcluster:TrajCluster                           0.0678672     0.0738649     0.0798626     0.0738649    0.00599773        2     
reco:pandora:StandardPandora                           0.130093       114.344       228.558       114.344       114.214         2     
reco:pandoraTrack:LArPandoraTrackCreation             0.000113935   0.000909472   0.00170501    0.000909472   0.000795537       2     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000297201    0.0012384    0.00217959     0.0012384    0.000941197       2     
reco:pandoracalo:Calorimetry                          9.9338e-05    0.000479716   0.000860094   0.000479716   0.000380378       2     
reco:pandorapid:Chi2ParticleID                        9.4448e-05    0.000282278   0.000470108   0.000282278   0.00018783        2     
reco:trkshowersplit:TrackShowerHits                   0.00116766    0.00127497    0.00138229    0.00127497    0.000107317       2     
reco:pmtrack:PMAlgTrackMaker                          0.00353239     0.188962      0.374391      0.188962      0.185429         2     
reco:pmtrackcalo:Calorimetry                          0.000115869   0.000660051   0.00120423    0.000660051   0.000544182       2     
reco:pmtrackpid:Chi2ParticleID                        2.6771e-05    0.000107117   0.000187463   0.000107117   8.0346e-05        2     
reco:pmtrajfit:PMAlgTrajFitter                        0.000343659   0.000745548   0.00114744    0.000745548   0.000401889       2     
reco:pmtrajfitcalo:Calorimetry                        5.7298e-05    6.0339e-05     6.338e-05    6.0339e-05     3.041e-06        2     
reco:pmtrajfitpid:Chi2ParticleID                      2.4857e-05    2.5869e-05    2.6881e-05    2.5869e-05     1.012e-06        2     
reco:pmtracktc:PMAlgTrackMaker                         0.0106679     0.342172      0.673677      0.342172      0.331504         2     
reco:pmtracktccalo:Calorimetry                        0.000835948    0.0011253    0.00141466     0.0011253    0.000289357       2     
reco:pmtracktcpid:Chi2ParticleID                      0.000134835   0.000339487   0.000544138   0.000339487   0.000204652       2     
reco:pmtrajfittc:PMAlgTrajFitter                      0.00329291     0.204108      0.404922      0.204108      0.200815         2     
reco:pmtrajfittccalo:Calorimetry                      0.000121129   0.000764248   0.00140737    0.000764248   0.00064312        2     
reco:pmtrajfittcpid:Chi2ParticleID                    4.0737e-05    0.000104036   0.000167336   0.000104036   6.32995e-05       2     
reco:blurredcluster:BlurredClustering                  0.0813252      0.25129      0.421255       0.25129      0.169965         2     
reco:emtrkmichelid:EmTrackMichelId                     0.334704       1.88313       3.43157       1.88313       1.54843         2     
reco:emshower:EMShower                                0.000404755   0.00147754    0.00255033    0.00147754    0.00107279        2     
reco:cvnmap:CVNMapper                                 0.000172406    0.0135716     0.0269709     0.0135716     0.0133992        2     
reco:cvneva:CVNEvaluator                              0.000157397   0.000157397   0.000157397   0.000157397        0            1     
reco:energyrecnumu:EnergyReco                         0.00427964    0.00427964    0.00427964    0.00427964         0            1     
reco:energyrecnue:EnergyReco                          0.00204595    0.00204595    0.00204595    0.00204595         0            1     
reco:energyrecnc:EnergyReco                           0.00206219    0.00206219    0.00206219    0.00206219         0            1     
reco:energyrecnumurange:EnergyReco                    0.00200367    0.00200367    0.00200367    0.00200367         0            1     
reco:energyrecnumumcs:EnergyReco                      0.00204133    0.00204133    0.00204133    0.00204133         0            1     
reco:opdec:Deconvolution                               0.222599      0.222599      0.222599      0.222599          0            1     
reco:ophitspe:OpHitFinderDeco                          0.964189      0.964189      0.964189      0.964189          0            1     
reco:opflash:OpFlashFinder                             0.0024431     0.0024431     0.0024431     0.0024431         0            1     
reco:rns:RandomNumberSaver                            0.000228812   0.000228812   0.000228812   0.000228812        0            1     
reco:anglereconue:NuAngularReco                       0.000591176   0.000591176   0.000591176   0.000591176        0            1     
reco:anglereconumu:NuAngularReco                      9.7495e-05    9.7495e-05    9.7495e-05    9.7495e-05         0            1     
reco:anglereconuepfps:NuAngularReco                   0.00084215    0.00084215    0.00084215    0.00084215         0            1     
reco:anglereconumupfps:NuAngularReco                  0.000123894   0.000123894   0.000123894   0.000123894        0            1     
[art]:TriggerResults:TriggerResultInserter            3.3674e-05    3.3674e-05    3.3674e-05    3.3674e-05         0            1     
end_path:out1:RootOutput                              1.2193e-05    1.2193e-05    1.2193e-05    1.2193e-05         0            1     
end_path:out1:RootOutput(write)                        0.060045      0.060045      0.060045      0.060045          0            1     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 37136.2 MB
  Peak resident set size usage (VmHWM): 1152.41 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Nov-2024 05:51:19 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- StdException BEGIN
      An exception was thrown while processing module CVNEvaluator/cvneva run: 6454800 subRun: 1 event: 72602
      CHECK failed: it != end(): key not found: serving_default
    ---- StdException END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 1 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-2359-cb28-12c5-0032.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6454800_726_20231207T111252Z_gen_g4_detsim_hitreco__20240510T052433Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6454800_726_20231207T111252Z_gen_g4_detsim_hitreco__20240510T052433Z_reco2_reco_2024-11-23T_054324Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work