Jobsub ID 301751.42@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301751.42@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:14:46
Site	UK_Imperial
Entry	DUNE_T2_UK_London_IC_ceprod03
Last heartbeat	2024-11-23 05:25:01
From worker node	Hostname	wj51.grid.hep.ph.ic.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:16:16
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472771_105_20231208T085018Z_gen_g4_detsim_hitreco__20240510T061132Z_reco2.root
Jobscript	Exit code	0
	Real time	8m (508s)
	CPU time	2m (121s = 23%)
Outputting started	2024-11-23 05:24:45
Output files
Finished	2024-11-23 05:25:01
List job events Wrapper job log
Jobscript log (last 10,000 characters)

er:pmtrajfittc@BeginModule  23-Nov-2024 05:23:58 GMT run: 6472771 subRun: 1 event: 10502
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:23:58 GMT run: 6472771 subRun: 1 event: 10502
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:23:58 GMT run: 6472771 subRun: 1 event: 10502
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:23:58 GMT run: 6472771 subRun: 1 event: 10502
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:23:58 GMT run: 6472771 subRun: 1 event: 10502
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 762, 790, 370
minwire 0: 566
minwire 1: 2189
minwire 2: 73
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Could not find serving_default in model signatures.
[libprotobuf FATAL /cvmfs/larsoft.opensciencegrid.org/products/protobuf/v3_21_12a/Linux64bit+3.10-2.17-e26/include/google/protobuf/map.h:1300] CHECK failed: it != end(): key not found: serving_default
23-Nov-2024 05:24:44 GMT  Closed input file "root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/59/cf/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472771_105_20231208T085018Z_gen_g4_detsim_hitreco__20240510T061132Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                            0.00936894      119.211       238.413       119.211       119.202         2     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00936894     0.0447987     0.0802284     0.0447987     0.0354297        2     
reco:linecluster:LineCluster                           0.501004      0.505814      0.510625      0.505814     0.00481049        2     
reco:trajcluster:TrajCluster                           0.0848882     0.207164       0.32944      0.207164      0.122276         2     
reco:pandora:StandardPandora                           0.583999       118.151       235.719       118.151       117.567         2     
reco:pandoraTrack:LArPandoraTrackCreation             0.000241994    0.0011507     0.0020594     0.0011507    0.000908702       2     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000484333    0.001676     0.00286767     0.001676     0.00119167        2     
reco:pandoracalo:Calorimetry                          0.000204574   0.000689473   0.00117437    0.000689473    0.0004849        2     
reco:pandorapid:Chi2ParticleID                        9.2622e-05    0.000375113   0.000657604   0.000375113   0.000282491       2     
reco:trkshowersplit:TrackShowerHits                    0.0018334    0.00718412     0.0125349    0.00718412    0.00535073        2     
reco:pmtrack:PMAlgTrackMaker                           0.0049559      1.80013       3.59529       1.80013       1.79517         2     
reco:pmtrackcalo:Calorimetry                          0.000112919   0.00591754     0.0117222    0.00591754    0.00580462        2     
reco:pmtrackpid:Chi2ParticleID                        4.4941e-05    0.000208074   0.000371207   0.000208074   0.000163133       2     
reco:pmtrajfit:PMAlgTrajFitter                        0.000639643   0.00114735    0.00165505    0.00114735    0.000507703       2     
reco:pmtrajfitcalo:Calorimetry                        0.000102266   0.000104484   0.000106701   0.000104484   2.2175e-06        2     
reco:pmtrajfitpid:Chi2ParticleID                      4.1478e-05    4.1806e-05    4.2134e-05    4.1806e-05     3.28e-07         2     
reco:pmtracktc:PMAlgTrackMaker                         0.0117229      1.00236       1.99299       1.00236      0.990632         2     
reco:pmtracktccalo:Calorimetry                        0.00204316    0.00648297     0.0109228    0.00648297     0.0044398        2     
reco:pmtracktcpid:Chi2ParticleID                      0.000380068   0.000556936   0.000733803   0.000556936   0.000176867       2     
reco:pmtrajfittc:PMAlgTrajFitter                       0.0916653     0.792715       1.49376      0.792715       0.70105         2     
reco:pmtrajfittccalo:Calorimetry                      0.000267079   0.00615217     0.0120373    0.00615217    0.00588509        2     
reco:pmtrajfittcpid:Chi2ParticleID                    8.0197e-05    0.000265207   0.000450217   0.000265207   0.00018501        2     
reco:blurredcluster:BlurredClustering                  0.0142469      1.16107       2.30789       1.16107       1.14682         2     
reco:emtrkmichelid:EmTrackMichelId                     0.387444       8.99014       17.5928       8.99014       8.6027          2     
reco:emshower:EMShower                                0.00111914    0.00202015    0.00292116    0.00202015    0.000901012       2     
reco:cvnmap:CVNMapper                                 0.000208066    0.0169525     0.0336969     0.0169525     0.0167444        2     
reco:cvneva:CVNEvaluator                              0.000209321   0.000209321   0.000209321   0.000209321        0            1     
reco:energyrecnumu:EnergyReco                         0.00514601    0.00514601    0.00514601    0.00514601         0            1     
reco:energyrecnue:EnergyReco                           0.0028193     0.0028193     0.0028193     0.0028193         0            1     
reco:energyrecnc:EnergyReco                           0.00297543    0.00297543    0.00297543    0.00297543         0            1     
reco:energyrecnumurange:EnergyReco                    0.00292224    0.00292224    0.00292224    0.00292224         0            1     
reco:energyrecnumumcs:EnergyReco                      0.00292223    0.00292223    0.00292223    0.00292223         0            1     
reco:opdec:Deconvolution                               0.393157      0.393157      0.393157      0.393157          0            1     
reco:ophitspe:OpHitFinderDeco                          0.997342      0.997342      0.997342      0.997342          0            1     
reco:opflash:OpFlashFinder                            0.00318103    0.00318103    0.00318103    0.00318103         0            1     
reco:rns:RandomNumberSaver                            0.000263831   0.000263831   0.000263831   0.000263831        0            1     
reco:anglereconue:NuAngularReco                       0.000777738   0.000777738   0.000777738   0.000777738        0            1     
reco:anglereconumu:NuAngularReco                      0.00015416    0.00015416    0.00015416    0.00015416         0            1     
reco:anglereconuepfps:NuAngularReco                   0.00121117    0.00121117    0.00121117    0.00121117         0            1     
reco:anglereconumupfps:NuAngularReco                  0.000201025   0.000201025   0.000201025   0.000201025        0            1     
[art]:TriggerResults:TriggerResultInserter            4.4205e-05    4.4205e-05    4.4205e-05    4.4205e-05         0            1     
end_path:out1:RootOutput                              1.3998e-05    1.3998e-05    1.3998e-05    1.3998e-05         0            1     
end_path:out1:RootOutput(write)                        0.0855526     0.0855526     0.0855526     0.0855526         0            1     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 7313.3 MB
  Peak resident set size usage (VmHWM): 1397.2 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Nov-2024 05:24:44 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- StdException BEGIN
      An exception was thrown while processing module CVNEvaluator/cvneva run: 6472771 subRun: 1 event: 10502
      CHECK failed: it != end(): key not found: serving_default
    ---- StdException END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 1 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-f779-ffa9-53f8-5ae1.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472771_105_20231208T085018Z_gen_g4_detsim_hitreco__20240510T061132Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6472771_105_20231208T085018Z_gen_g4_detsim_hitreco__20240510T061132Z_reco2_reco_2024-11-23T_051620Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work