Jobsub ID 301839.107@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301839.107@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 06:18:31
Site	UK_Imperial
Entry	DUNE_T2_UK_London_IC_ceprod03
Last heartbeat	2024-11-23 06:31:25
From worker node	Hostname	wg37.grid.hep.ph.ic.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU X5650 @ 2.67GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 06:19:34
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_712_20231205T031510Z_gen_g4_detsim_hitreco__20240510T020247Z_reco2.root
Jobscript	Exit code	0
	Real time	11m (692s)
	CPU time	2m (150s = 21%)
Outputting started	2024-11-23 06:31:06
Output files
Finished	2024-11-23 06:31:25
List job events Wrapper job log
Jobscript log (last 10,000 characters)

tc@BeginModule  23-Nov-2024 06:30:14 GMT run: 50601160 subRun: 1 event: 71202
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:30:14 GMT run: 50601160 subRun: 1 event: 71202
1st wire C:0 T:10 P:2 W:645 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:30:14 GMT run: 50601160 subRun: 1 event: 71202
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:30:14 GMT run: 50601160 subRun: 1 event: 71202
1st wire C:0 T:14 P:2 W:766 does not exist (max wire number: 480)
%MSG
Boundary wire vector sizes: 269, 415, 328
minwire 0: 1275
minwire 1: 1554
minwire 2: 1127
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Could not find serving_default in model signatures.
[libprotobuf FATAL /cvmfs/larsoft.opensciencegrid.org/products/protobuf/v3_21_12a/Linux64bit+3.10-2.17-e26/include/google/protobuf/map.h:1300] CHECK failed: it != end(): key not found: serving_default
23-Nov-2024 06:31:05 GMT  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/f7/c6/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_712_20231205T031510Z_gen_g4_detsim_hitreco__20240510T020247Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.0208305      122.639       245.257       122.639       122.618         2     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0208305     0.0211668     0.0215031     0.0211668    0.000336289       2     
reco:linecluster:LineCluster                           0.215919      0.237259      0.258599      0.237259      0.0213403        2     
reco:trajcluster:TrajCluster                           0.0789942     0.110509      0.142024      0.110509      0.031515         2     
reco:pandora:StandardPandora                           0.298819       119.223       238.148       119.223       118.924         2     
reco:pandoraTrack:LArPandoraTrackCreation             0.00029436    0.00179119    0.00328801    0.00179119    0.00149683        2     
reco:pandoraShower:LArPandoraModularShowerCreation     0.0003951    0.00233219    0.00426927    0.00233219    0.00193709        2     
reco:pandoracalo:Calorimetry                          0.000240385   0.000992509   0.00174463    0.000992509   0.000752124       2     
reco:pandorapid:Chi2ParticleID                        0.000100644   0.000540438   0.000980232   0.000540438   0.000439794       2     
reco:trkshowersplit:TrackShowerHits                   0.00285635    0.00426144    0.00566653    0.00426144    0.00140509        2     
reco:pmtrack:PMAlgTrackMaker                           0.0311819      2.07529       4.1194        2.07529       2.04411         2     
reco:pmtrackcalo:Calorimetry                          0.00349958    0.00529246    0.00708533    0.00529246    0.00179287        2     
reco:pmtrackpid:Chi2ParticleID                        0.000588929   0.000871406   0.00115388    0.000871406   0.000282478       2     
reco:pmtrajfit:PMAlgTrajFitter                        0.000509898   0.00151795     0.002526     0.00151795    0.00100805        2     
reco:pmtrajfitcalo:Calorimetry                        0.000168645   0.000179105   0.000189565   0.000179105    1.046e-05        2     
reco:pmtrajfitpid:Chi2ParticleID                       7.209e-05    7.32115e-05   7.4333e-05    7.32115e-05   1.1215e-06        2     
reco:pmtracktc:PMAlgTrackMaker                         0.0161406     0.980512       1.94488      0.980512      0.964371         2     
reco:pmtracktccalo:Calorimetry                        0.000525307    0.0040321    0.00753889     0.0040321    0.00350679        2     
reco:pmtracktcpid:Chi2ParticleID                      0.000143118   0.000376643   0.000610168   0.000376643   0.000233525       2     
reco:pmtrajfittc:PMAlgTrajFitter                       0.053788      0.871772       1.68976      0.871772      0.817984         2     
reco:pmtrajfittccalo:Calorimetry                      0.00039314    0.00335012     0.0063071    0.00335012    0.00295698        2     
reco:pmtrajfittcpid:Chi2ParticleID                    0.00011856    0.000340602   0.000562644   0.000340602   0.000222042       2     
reco:blurredcluster:BlurredClustering                   1.09535       1.27526       1.45518       1.27526      0.179914         2     
reco:emtrkmichelid:EmTrackMichelId                      1.39858       8.67637       15.9542       8.67637       7.2778          2     
reco:emshower:EMShower                                0.00112362     0.0028722    0.00462078     0.0028722    0.00174858        2     
reco:cvnmap:CVNMapper                                 0.000334865    0.0205087     0.0406826     0.0205087     0.0201739        2     
reco:cvneva:CVNEvaluator                              0.000363524   0.000363524   0.000363524   0.000363524        0            1     
reco:energyrecnumu:EnergyReco                         0.00805074    0.00805074    0.00805074    0.00805074         0            1     
reco:energyrecnue:EnergyReco                          0.00435144    0.00435144    0.00435144    0.00435144         0            1     
reco:energyrecnc:EnergyReco                           0.00441425    0.00441425    0.00441425    0.00441425         0            1     
reco:energyrecnumurange:EnergyReco                    0.00426237    0.00426237    0.00426237    0.00426237         0            1     
reco:energyrecnumumcs:EnergyReco                      0.00421736    0.00421736    0.00421736    0.00421736         0            1     
reco:opdec:Deconvolution                               0.428739      0.428739      0.428739      0.428739          0            1     
reco:ophitspe:OpHitFinderDeco                           3.55221       3.55221       3.55221       3.55221          0            1     
reco:opflash:OpFlashFinder                            0.00542886    0.00542886    0.00542886    0.00542886         0            1     
reco:rns:RandomNumberSaver                            0.000389668   0.000389668   0.000389668   0.000389668        0            1     
reco:anglereconue:NuAngularReco                       0.00114932    0.00114932    0.00114932    0.00114932         0            1     
reco:anglereconumu:NuAngularReco                      0.000265958   0.000265958   0.000265958   0.000265958        0            1     
reco:anglereconuepfps:NuAngularReco                   0.00169995    0.00169995    0.00169995    0.00169995         0            1     
reco:anglereconumupfps:NuAngularReco                  0.000354124   0.000354124   0.000354124   0.000354124        0            1     
[art]:TriggerResults:TriggerResultInserter            6.5724e-05    6.5724e-05    6.5724e-05    6.5724e-05         0            1     
end_path:out1:RootOutput                              2.3362e-05    2.3362e-05    2.3362e-05    2.3362e-05         0            1     
end_path:out1:RootOutput(write)                        0.152599      0.152599      0.152599      0.152599          0            1     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6088.41 MB
  Peak resident set size usage (VmHWM): 1278.22 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Nov-2024 06:31:05 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- StdException BEGIN
      An exception was thrown while processing module CVNEvaluator/cvneva run: 50601160 subRun: 1 event: 71202
      CHECK failed: it != end(): key not found: serving_default
    ---- StdException END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 1 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-5119-bad5-0137-d8c1.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_712_20231205T031510Z_gen_g4_detsim_hitreco__20240510T020247Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50601160_712_20231205T031510Z_gen_g4_detsim_hitreco__20240510T020247Z_reco2_reco_2024-11-23T_061938Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work