Jobsub ID 301639.149@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301639.149@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 02:10:16
Site	UK_Imperial
Entry	DUNE_T2_UK_London_IC_ceprod00
Last heartbeat	2024-11-23 02:36:40
From worker node	Hostname	wg62.grid.hep.ph.ic.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU X5650 @ 2.67GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 02:19:04
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50584512_246_20231204T062533Z_gen_g4_detsim_hitreco__20240509T214425Z_reco2.root
Jobscript	Exit code	0
	Real time	17m (1036s)
	CPU time	10m (628s = 60%)
Outputting started	2024-11-23 02:36:21
Output files
Finished	2024-11-23 02:36:40
List job events Wrapper job log
Jobscript log (last 10,000 characters)

MAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 02:31:50 GMT run: 50584512 subRun: 1 event: 24602
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 02:31:50 GMT run: 50584512 subRun: 1 event: 24602
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 02:31:50 GMT run: 50584512 subRun: 1 event: 24602
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 02:31:50 GMT run: 50584512 subRun: 1 event: 24602
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 5430, 5800, 4176
minwire 0: 1068
minwire 1: 466
minwire 2: 670
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Could not find serving_default in model signatures.
[libprotobuf FATAL /cvmfs/larsoft.opensciencegrid.org/products/protobuf/v3_21_12a/Linux64bit+3.10-2.17-e26/include/google/protobuf/map.h:1300] CHECK failed: it != end(): key not found: serving_default
23-Nov-2024 02:36:19 GMT  Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/6b/29/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50584512_246_20231204T062533Z_gen_g4_detsim_hitreco__20240509T214425Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.0296879      121.308       242.587       121.308       121.279         2     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00814469     0.0189163     0.0296879     0.0189163     0.0107716        2     
reco:linecluster:LineCluster                           0.136066       2.98879       5.84152       2.98879       2.85273         2     
reco:trajcluster:TrajCluster                           0.0418514      15.8853       31.7287       15.8853       15.8434         2     
reco:pandora:StandardPandora                            26.0235       132.876       239.728       132.876       106.852         2     
reco:pandoraTrack:LArPandoraTrackCreation             0.000286873   0.00166324     0.0030396    0.00166324    0.00137636        2     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000551344   0.00233216    0.00411298    0.00233216    0.00178082        2     
reco:pandoracalo:Calorimetry                          0.000176098   0.000964243   0.00175239    0.000964243   0.000788146       2     
reco:pandorapid:Chi2ParticleID                        7.5063e-05    0.000513931    0.0009528    0.000513931   0.000438868       2     
reco:trkshowersplit:TrackShowerHits                   0.00273227     0.330063      0.657394      0.330063      0.327331         2     
reco:pmtrack:PMAlgTrackMaker                          0.00766869      25.6324       51.2572       25.6324       25.6248         2     
reco:pmtrackcalo:Calorimetry                          0.000183546    0.170028      0.339872      0.170028      0.169844         2     
reco:pmtrackpid:Chi2ParticleID                        8.6605e-05    0.00192853    0.00377045    0.00192853    0.00184192        2     
reco:pmtrajfit:PMAlgTrajFitter                        0.00172887    0.00203349    0.00233811    0.00203349    0.000304623       2     
reco:pmtrajfitcalo:Calorimetry                        0.000135682   0.000141507   0.000147332   0.000141507    5.825e-06        2     
reco:pmtrajfitpid:Chi2ParticleID                      5.1352e-05    5.77335e-05   6.4115e-05    5.77335e-05   6.3815e-06        2     
reco:pmtracktc:PMAlgTrackMaker                         0.0135708      52.8516       105.69        52.8516       52.838          2     
reco:pmtracktccalo:Calorimetry                        0.00340761     0.104275      0.205142      0.104275      0.100867         2     
reco:pmtracktcpid:Chi2ParticleID                      0.00120084    0.00276596    0.00433107    0.00276596    0.00156512        2     
reco:pmtrajfittc:PMAlgTrajFitter                        0.12893       25.0545        49.98        25.0545       24.9255         2     
reco:pmtrajfittccalo:Calorimetry                      0.000669406    0.301809      0.602949      0.301809       0.30114         2     
reco:pmtrajfittcpid:Chi2ParticleID                    0.000161127   0.00382385    0.00748657    0.00382385    0.00366272        2     
reco:blurredcluster:BlurredClustering                  0.0191989      6.21943       12.4197       6.21943       6.20023         2     
reco:emtrkmichelid:EmTrackMichelId                      0.81048       111.335       221.859       111.335       110.524         2     
reco:emshower:EMShower                                0.00462692    0.00664384    0.00866077    0.00664384    0.00201693        2     
reco:cvnmap:CVNMapper                                 0.000366314    0.283421      0.566475      0.283421      0.283054         2     
reco:cvneva:CVNEvaluator                              0.000314705   0.000314705   0.000314705   0.000314705        0            1     
reco:energyrecnumu:EnergyReco                         0.00865896    0.00865896    0.00865896    0.00865896         0            1     
reco:energyrecnue:EnergyReco                          0.00443628    0.00443628    0.00443628    0.00443628         0            1     
reco:energyrecnc:EnergyReco                           0.00434794    0.00434794    0.00434794    0.00434794         0            1     
reco:energyrecnumurange:EnergyReco                    0.00411888    0.00411888    0.00411888    0.00411888         0            1     
reco:energyrecnumumcs:EnergyReco                      0.00407761    0.00407761    0.00407761    0.00407761         0            1     
reco:opdec:Deconvolution                               0.175956      0.175956      0.175956      0.175956          0            1     
reco:ophitspe:OpHitFinderDeco                           1.37667       1.37667       1.37667       1.37667          0            1     
reco:opflash:OpFlashFinder                            0.00534985    0.00534985    0.00534985    0.00534985         0            1     
reco:rns:RandomNumberSaver                            0.000517108   0.000517108   0.000517108   0.000517108        0            1     
reco:anglereconue:NuAngularReco                       0.00151754    0.00151754    0.00151754    0.00151754         0            1     
reco:anglereconumu:NuAngularReco                      0.000274097   0.000274097   0.000274097   0.000274097        0            1     
reco:anglereconuepfps:NuAngularReco                   0.00211888    0.00211888    0.00211888    0.00211888         0            1     
reco:anglereconumupfps:NuAngularReco                  0.000363453   0.000363453   0.000363453   0.000363453        0            1     
[art]:TriggerResults:TriggerResultInserter            0.000106602   0.000106602   0.000106602   0.000106602        0            1     
end_path:out1:RootOutput                              3.2018e-05    3.2018e-05    3.2018e-05    3.2018e-05         0            1     
end_path:out1:RootOutput(write)                        0.0756153     0.0756153     0.0756153     0.0756153         0            1     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 7029.17 MB
  Peak resident set size usage (VmHWM): 2119.09 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Nov-2024 02:36:20 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- StdException BEGIN
      An exception was thrown while processing module CVNEvaluator/cvneva run: 50584512 subRun: 1 event: 24602
      CHECK failed: it != end(): key not found: serving_default
    ---- StdException END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 1 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-7695-a695-c399-4380.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50584512_246_20231204T062533Z_gen_g4_detsim_hitreco__20240509T214425Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50584512_246_20231204T062533Z_gen_g4_detsim_hitreco__20240509T214425Z_reco2_reco_2024-11-23T_021909Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work