Jobsub ID 301751.100@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301751.100@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:14:46
Site	UK_Imperial
Entry	DUNE_T2_UK_London_IC_ceprod00
Last heartbeat	2024-11-23 05:26:19
From worker node	Hostname	wj36.grid.hep.ph.ic.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:16:13
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406500_961_20231202T131336Z_gen_g4_detsim_hitreco__20240507T222628Z_reco2.root
Jobscript	Exit code	0
	Real time	9m (583s)
	CPU time	2m (134s = 22%)
Outputting started	2024-11-23 05:25:57
Output files
Finished	2024-11-23 05:26:19
List job events Wrapper job log
Jobscript log (last 10,000 characters)

ireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:24:55 GMT run: 6406500 subRun: 1 event: 96102
1st wire C:0 T:19 P:2 W:666 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:24:55 GMT run: 6406500 subRun: 1 event: 96102
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:24:55 GMT run: 6406500 subRun: 1 event: 96102
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:24:55 GMT run: 6406500 subRun: 1 event: 96102
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 509, 609, 341
minwire 0: 1875
minwire 1: 943
minwire 2: 1884
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Could not find serving_default in model signatures.
[libprotobuf FATAL /cvmfs/larsoft.opensciencegrid.org/products/protobuf/v3_21_12a/Linux64bit+3.10-2.17-e26/include/google/protobuf/map.h:1300] CHECK failed: it != end(): key not found: serving_default
23-Nov-2024 05:25:56 GMT  Closed input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/d7/58/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406500_961_20231202T131336Z_gen_g4_detsim_hitreco__20240507T222628Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                            0.00396097      130.787       261.571       130.787       130.783         2     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00396097    0.00412318    0.00428539    0.00412318    0.000162211       2     
reco:linecluster:LineCluster                           0.0580368     0.394844      0.731652      0.394844      0.336808         2     
reco:trajcluster:TrajCluster                           0.0260124     0.0793657     0.132719      0.0793657     0.0533533        2     
reco:pandora:StandardPandora                           0.380041       130.134       259.888       130.134       129.754         2     
reco:pandoraTrack:LArPandoraTrackCreation             0.000255257   0.00169101    0.00312677    0.00169101    0.00143575        2     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000570162   0.00240783     0.0042455    0.00240783    0.00183767        2     
reco:pandoracalo:Calorimetry                          0.000202495   0.000968491   0.00173449    0.000968491   0.000765996       2     
reco:pandorapid:Chi2ParticleID                        0.000104806   0.000511945   0.000919083   0.000511945   0.000407139       2     
reco:trkshowersplit:TrackShowerHits                   0.00260958    0.00693094     0.0112523    0.00693094    0.00432136        2     
reco:pmtrack:PMAlgTrackMaker                           0.0112234     0.797726       1.58423      0.797726      0.786503         2     
reco:pmtrackcalo:Calorimetry                          0.000182699   0.00375285    0.00732301    0.00375285    0.00357015        2     
reco:pmtrackpid:Chi2ParticleID                        8.7695e-05    0.00027604    0.000464385   0.00027604    0.000188345       2     
reco:pmtrajfit:PMAlgTrajFitter                        0.000793446   0.00147212     0.0021508    0.00147212    0.000678679       2     
reco:pmtrajfitcalo:Calorimetry                        0.000163626   0.000166574   0.000169521   0.000166574   2.9475e-06        2     
reco:pmtrajfitpid:Chi2ParticleID                       6.491e-05    6.90655e-05   7.3221e-05    6.90655e-05   4.1555e-06        2     
reco:pmtracktc:PMAlgTrackMaker                        0.00821309     0.811386       1.61456      0.811386      0.803173         2     
reco:pmtracktccalo:Calorimetry                        0.00292222    0.00404869    0.00517516    0.00404869    0.00112647        2     
reco:pmtracktcpid:Chi2ParticleID                      0.000368038   0.000682489   0.00099694    0.000682489   0.000314451       2     
reco:pmtrajfittc:PMAlgTrajFitter                       0.0933625      1.31352       2.53367       1.31352       1.22016         2     
reco:pmtrajfittccalo:Calorimetry                      0.000370006   0.00481341    0.00925681    0.00481341     0.0044434        2     
reco:pmtrajfittcpid:Chi2ParticleID                    0.000108357   0.000348078   0.000587799   0.000348078   0.000239721       2     
reco:blurredcluster:BlurredClustering                  0.053142       1.84749       3.64183       1.84749       1.79434         2     
reco:emtrkmichelid:EmTrackMichelId                     0.434103       9.94927       19.4644       9.94927       9.51517         2     
reco:emshower:EMShower                                0.00126655    0.00223269    0.00319882    0.00223269    0.000966135       2     
reco:cvnmap:CVNMapper                                 0.000239505    0.0233657     0.0464919     0.0233657     0.0231262        2     
reco:cvneva:CVNEvaluator                              0.000234151   0.000234151   0.000234151   0.000234151        0            1     
reco:energyrecnumu:EnergyReco                         0.00546285    0.00546285    0.00546285    0.00546285         0            1     
reco:energyrecnue:EnergyReco                           0.0028914     0.0028914     0.0028914     0.0028914         0            1     
reco:energyrecnc:EnergyReco                           0.00298367    0.00298367    0.00298367    0.00298367         0            1     
reco:energyrecnumurange:EnergyReco                    0.00292258    0.00292258    0.00292258    0.00292258         0            1     
reco:energyrecnumumcs:EnergyReco                      0.00291629    0.00291629    0.00291629    0.00291629         0            1     
reco:opdec:Deconvolution                               0.078153      0.078153      0.078153      0.078153          0            1     
reco:ophitspe:OpHitFinderDeco                          0.804839      0.804839      0.804839      0.804839          0            1     
reco:opflash:OpFlashFinder                            0.00318438    0.00318438    0.00318438    0.00318438         0            1     
reco:rns:RandomNumberSaver                            0.000282349   0.000282349   0.000282349   0.000282349        0            1     
reco:anglereconue:NuAngularReco                       0.000812267   0.000812267   0.000812267   0.000812267        0            1     
reco:anglereconumu:NuAngularReco                      0.000160407   0.000160407   0.000160407   0.000160407        0            1     
reco:anglereconuepfps:NuAngularReco                   0.00116326    0.00116326    0.00116326    0.00116326         0            1     
reco:anglereconumupfps:NuAngularReco                  0.000206648   0.000206648   0.000206648   0.000206648        0            1     
[art]:TriggerResults:TriggerResultInserter            4.7589e-05    4.7589e-05    4.7589e-05    4.7589e-05         0            1     
end_path:out1:RootOutput                              1.5947e-05    1.5947e-05    1.5947e-05    1.5947e-05         0            1     
end_path:out1:RootOutput(write)                        0.062364      0.062364      0.062364      0.062364          0            1     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 7429.73 MB
  Peak resident set size usage (VmHWM): 1509.08 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Nov-2024 05:25:56 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- StdException BEGIN
      An exception was thrown while processing module CVNEvaluator/cvneva run: 6406500 subRun: 1 event: 96102
      CHECK failed: it != end(): key not found: serving_default
    ---- StdException END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 1 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-1fae-69a2-68ac-26ac.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406500_961_20231202T131336Z_gen_g4_detsim_hitreco__20240507T222628Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406500_961_20231202T131336Z_gen_g4_detsim_hitreco__20240507T222628Z_reco2_reco_2024-11-23T_051618Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work