Jobsub ID 108779.1@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108779.1@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:04:07
Site	UK_Imperial
Entry	DUNE_T2_UK_London_IC_ceprod01
Last heartbeat	2024-11-23 05:17:58
From worker node	Hostname	wj47.grid.hep.ph.ic.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:04:49
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6416859_1047_20231203T085211Z_gen_g4_detsim_hitreco__20240509T202747Z_reco2.root
Jobscript	Exit code	0
	Real time	12m (768s)
	CPU time	2m (129s = 16%)
Outputting started	2024-11-23 05:17:38
Output files
Finished	2024-11-23 05:17:58
List job events Wrapper job log
Jobscript log (last 10,000 characters)

PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:17:04 GMT run: 6416859 subRun: 1 event: 104702
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:17:04 GMT run: 6416859 subRun: 1 event: 104702
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:17:04 GMT run: 6416859 subRun: 1 event: 104702
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:17:04 GMT run: 6416859 subRun: 1 event: 104702
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 145, 198, 117
minwire 0: 657
minwire 1: 1758
minwire 2: 746
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Could not find serving_default in model signatures.
[libprotobuf FATAL /cvmfs/larsoft.opensciencegrid.org/products/protobuf/v3_21_12a/Linux64bit+3.10-2.17-e26/include/google/protobuf/map.h:1300] CHECK failed: it != end(): key not found: serving_default
23-Nov-2024 05:17:36 GMT  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/b3/5b/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6416859_1047_20231203T085211Z_gen_g4_detsim_hitreco__20240509T202747Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.0334239      121.044       242.055       121.044       121.011         2     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0216185     0.0275212     0.0334239     0.0275212    0.00590266        2     
reco:linecluster:LineCluster                           0.279662      0.360106      0.440549      0.360106      0.0804435        2     
reco:trajcluster:TrajCluster                           0.0790598     0.0842387     0.0894175     0.0842387    0.00517886        2     
reco:pandora:StandardPandora                           0.184222       118.482       236.779       118.482       118.298         2     
reco:pandoraTrack:LArPandoraTrackCreation             0.000257792   0.00161153    0.00296527    0.00161153    0.00135374        2     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000568429   0.00230726    0.00404609    0.00230726    0.00173883        2     
reco:pandoracalo:Calorimetry                          0.000211652   0.000948747   0.00168584    0.000948747   0.000737096       2     
reco:pandorapid:Chi2ParticleID                        0.000100794   0.000520097   0.000939399   0.000520097   0.000419302       2     
reco:trkshowersplit:TrackShowerHits                   0.00215765    0.00244205    0.00272645    0.00244205    0.000284401       2     
reco:pmtrack:PMAlgTrackMaker                           0.0416805     0.236442      0.431203      0.236442      0.194761         2     
reco:pmtrackcalo:Calorimetry                          0.00152688    0.00244053    0.00335418    0.00244053    0.000913649       2     
reco:pmtrackpid:Chi2ParticleID                        0.000310277   0.000691425   0.00107257    0.000691425   0.000381147       2     
reco:pmtrajfit:PMAlgTrajFitter                        0.000642224   0.00147649    0.00231075    0.00147649    0.000834266       2     
reco:pmtrajfitcalo:Calorimetry                        0.000161311   0.00017338    0.000185449   0.00017338    1.2069e-05        2     
reco:pmtrajfitpid:Chi2ParticleID                      6.7323e-05    7.64525e-05   8.5582e-05    7.64525e-05   9.1295e-06        2     
reco:pmtracktc:PMAlgTrackMaker                         0.0206914      0.30035       0.58001       0.30035      0.279659         2     
reco:pmtracktccalo:Calorimetry                        0.000513754   0.00103014    0.00154652    0.00103014    0.000516385       2     
reco:pmtracktcpid:Chi2ParticleID                      0.000137343   0.000202421   0.000267498   0.000202421   6.50775e-05       2     
reco:pmtrajfittc:PMAlgTrajFitter                       0.0240927     0.561182       1.09827      0.561182       0.53709         2     
reco:pmtrajfittccalo:Calorimetry                      0.000461788   0.00186512    0.00326846    0.00186512    0.00140333        2     
reco:pmtrajfittcpid:Chi2ParticleID                    0.000125601   0.000251512   0.000377422   0.000251512   0.000125911       2     
reco:blurredcluster:BlurredClustering                  0.118448       0.66797       1.21749       0.66797      0.549523         2     
reco:emtrkmichelid:EmTrackMichelId                      1.15415       3.57771       6.00128       3.57771       2.42356         2     
reco:emshower:EMShower                                0.000779395   0.00260429    0.00442918    0.00260429    0.00182489        2     
reco:cvnmap:CVNMapper                                 0.000377539    0.0167975     0.0332175     0.0167975      0.01642         2     
reco:cvneva:CVNEvaluator                              0.000310169   0.000310169   0.000310169   0.000310169        0            1     
reco:energyrecnumu:EnergyReco                         0.00754449    0.00754449    0.00754449    0.00754449         0            1     
reco:energyrecnue:EnergyReco                           0.0041393     0.0041393     0.0041393     0.0041393         0            1     
reco:energyrecnc:EnergyReco                           0.00408753    0.00408753    0.00408753    0.00408753         0            1     
reco:energyrecnumurange:EnergyReco                    0.00387745    0.00387745    0.00387745    0.00387745         0            1     
reco:energyrecnumumcs:EnergyReco                      0.00397864    0.00397864    0.00397864    0.00397864         0            1     
reco:opdec:Deconvolution                               0.387316      0.387316      0.387316      0.387316          0            1     
reco:ophitspe:OpHitFinderDeco                           1.75725       1.75725       1.75725       1.75725          0            1     
reco:opflash:OpFlashFinder                            0.00481395    0.00481395    0.00481395    0.00481395         0            1     
reco:rns:RandomNumberSaver                            0.000365683   0.000365683   0.000365683   0.000365683        0            1     
reco:anglereconue:NuAngularReco                        0.0011235     0.0011235     0.0011235     0.0011235         0            1     
reco:anglereconumu:NuAngularReco                      0.000246913   0.000246913   0.000246913   0.000246913        0            1     
reco:anglereconuepfps:NuAngularReco                   0.00166642    0.00166642    0.00166642    0.00166642         0            1     
reco:anglereconumupfps:NuAngularReco                  0.000347538   0.000347538   0.000347538   0.000347538        0            1     
[art]:TriggerResults:TriggerResultInserter            6.4809e-05    6.4809e-05    6.4809e-05    6.4809e-05         0            1     
end_path:out1:RootOutput                              2.2449e-05    2.2449e-05    2.2449e-05    2.2449e-05         0            1     
end_path:out1:RootOutput(write)                        0.0710822     0.0710822     0.0710822     0.0710822         0            1     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 7071.41 MB
  Peak resident set size usage (VmHWM): 1155.18 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Nov-2024 05:17:37 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- StdException BEGIN
      An exception was thrown while processing module CVNEvaluator/cvneva run: 6416859 subRun: 1 event: 104702
      CHECK failed: it != end(): key not found: serving_default
    ---- StdException END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 1 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-7aec-ac13-3f57-aefa.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6416859_1047_20231203T085211Z_gen_g4_detsim_hitreco__20240509T202747Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6416859_1047_20231203T085211Z_gen_g4_detsim_hitreco__20240509T202747Z_reco2_reco_2024-11-23T_050457Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work