Jobsub ID 108784.11@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108784.11@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:15:53
Site	UK_Imperial
Entry	DUNE_T2_UK_London_IC_ceprod01
Last heartbeat	2024-11-23 05:33:43
From worker node	Hostname	wj47.grid.hep.ph.ic.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:18:53
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_86_20231203T125043Z_gen_g4_detsim_hitreco__20240508T022819Z_reco2.root
Jobscript	Exit code	0
	Real time	14m (876s)
	CPU time	1m (106s = 12%)
Outputting started	2024-11-23 05:33:30
Output files
Finished	2024-11-23 05:33:43
List job events Wrapper job log
Jobscript log (last 10,000 characters)

GMT run: 74511302 subRun: 1 event: 8603
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 05:32:41 GMT run: 74511302 subRun: 1 event: 8603
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 05:32:41 GMT run: 74511302 subRun: 1 event: 8603
Hit sorting problem.
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:32:41 GMT run: 74511302 subRun: 1 event: 8603
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:32:41 GMT run: 74511302 subRun: 1 event: 8603
1st wire C:0 T:6 P:2 W:827 does not exist (max wire number: 480)
%MSG
Boundary wire vector sizes: 411, 605, 598
minwire 0: 602
minwire 1: 2442
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Could not find serving_default in model signatures.
[libprotobuf FATAL /cvmfs/larsoft.opensciencegrid.org/products/protobuf/v3_21_12a/Linux64bit+3.10-2.17-e26/include/google/protobuf/map.h:1300] CHECK failed: it != end(): key not found: serving_default
23-Nov-2024 05:33:29 GMT  Closed input file "root://xrootd.pic.es:1094/pnfs/pic.es/data/dune/RSE/fardet-hd/8b/99/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_86_20231203T125043Z_gen_g4_detsim_hitreco__20240508T022819Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.0684161      82.1502       243.037        3.345        113.772         3     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0346885     0.0461631     0.0684161     0.0353846     0.0157379        3     
reco:linecluster:LineCluster                           0.491524      0.521669      0.579788      0.493696      0.0411058        3     
reco:trajcluster:TrajCluster                           0.105512      0.184779      0.336386       0.11244       0.10724         3     
reco:pandora:StandardPandora                           0.143232       80.6728       241.625      0.250298       113.81          3     
reco:pandoraTrack:LArPandoraTrackCreation             0.000153732   0.00076535    0.00198811    0.000154203   0.000864625       3     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000202956   0.00112321    0.00283609    0.000330587   0.00121231        3     
reco:pandoracalo:Calorimetry                          0.000117344   0.000468919   0.00115153    0.00013788    0.000482753       3     
reco:pandorapid:Chi2ParticleID                        4.9633e-05    0.000259809   0.000662409   6.7385e-05    0.000284773       3     
reco:trkshowersplit:TrackShowerHits                   0.000845351   0.00237762    0.00507664    0.00121089    0.00191431        3     
reco:pmtrack:PMAlgTrackMaker                          0.00448695      1.58458       4.7358       0.0134429      2.22825         3     
reco:pmtrackcalo:Calorimetry                          0.000124666   0.00101057    0.00199161    0.000915424   0.00076514        3     
reco:pmtrackpid:Chi2ParticleID                        4.5316e-05    0.000304776   0.000747677   0.000121336   0.000314712       3     
reco:pmtrajfit:PMAlgTrajFitter                        0.000363853   0.000765705    0.0015396    0.000393662   0.000547361       3     
reco:pmtrajfitcalo:Calorimetry                        9.5927e-05    9.90977e-05   0.000103342   9.8024e-05    3.12091e-06       3     
reco:pmtrajfitpid:Chi2ParticleID                      4.2525e-05    4.75483e-05   5.5428e-05    4.4692e-05    5.64156e-06       3     
reco:pmtracktc:PMAlgTrackMaker                        0.00140642      0.63983       1.88764      0.0304383     0.882418         3     
reco:pmtracktccalo:Calorimetry                         9.989e-05     0.0036394     0.0106304    0.000187891   0.00494353        3     
reco:pmtracktcpid:Chi2ParticleID                       4.322e-05    8.97983e-05   0.000160744   6.5431e-05    5.09791e-05       3     
reco:pmtrajfittc:PMAlgTrajFitter                      0.000223449    0.346279       1.01844      0.0201758     0.475358         3     
reco:pmtrajfittccalo:Calorimetry                      0.000105472   0.00345471     0.0100519    0.000206769   0.00466509        3     
reco:pmtrajfittcpid:Chi2ParticleID                    5.8361e-05    0.000116395   0.000220527   7.0296e-05    7.37937e-05       3     
reco:blurredcluster:BlurredClustering                 0.000514067    0.421269      0.666614      0.596677      0.298885         3     
reco:emtrkmichelid:EmTrackMichelId                     0.0302268      4.89842       14.2864      0.378685       6.6398          3     
reco:emshower:EMShower                                0.000557953   0.00147583    0.00292186    0.000947663    0.0010348        3     
reco:cvnmap:CVNMapper                                 5.4428e-05     0.0114623     0.0340958    0.000236817    0.0160044        3     
reco:cvneva:CVNEvaluator                              2.3121e-05    0.000119836   0.000216551   0.000119836   9.6715e-05        2     
reco:energyrecnumu:EnergyReco                         0.00298138    0.00414554     0.0053097    0.00414554    0.00116416        2     
reco:energyrecnue:EnergyReco                          0.00273579    0.00278238    0.00282898    0.00278238    4.65945e-05       2     
reco:energyrecnc:EnergyReco                           0.00277713    0.00280139    0.00282566    0.00280139    2.4265e-05        2     
reco:energyrecnumurange:EnergyReco                    0.00274825    0.00275785    0.00276745    0.00275785     9.604e-06        2     
reco:energyrecnumumcs:EnergyReco                      0.00270983    0.00271543    0.00272103    0.00271543    5.5985e-06        2     
reco:opdec:Deconvolution                               0.120333      0.267093      0.413852      0.267093       0.14676         2     
reco:ophitspe:OpHitFinderDeco                          0.480051      0.755502       1.03095      0.755502      0.275451         2     
reco:opflash:OpFlashFinder                            0.000721513   0.00172725    0.00273298    0.00172725    0.00100573        2     
reco:rns:RandomNumberSaver                            2.3719e-05    0.000140437   0.000257154   0.000140437   0.000116718       2     
reco:anglereconue:NuAngularReco                       0.000216019   0.000492705   0.000769391   0.000492705   0.000276686       2     
reco:anglereconumu:NuAngularReco                      0.000101719   0.000127975   0.000154231   0.000127975   2.6256e-05        2     
reco:anglereconuepfps:NuAngularReco                   0.000146869   0.000614552   0.00108223    0.000614552   0.000467683       2     
reco:anglereconumupfps:NuAngularReco                  0.000155231   0.000177225   0.000199219   0.000177225   2.1994e-05        2     
[art]:TriggerResults:TriggerResultInserter            1.2672e-05    2.9641e-05     4.661e-05    2.9641e-05    1.6969e-05        2     
end_path:out1:RootOutput                               2.888e-06    8.7315e-06    1.4575e-05    8.7315e-06    5.8435e-06        2     
end_path:out1:RootOutput(write)                        0.0174836     0.0356968     0.0539099     0.0356968     0.0182132        2     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 7080.12 MB
  Peak resident set size usage (VmHWM): 1200.23 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Nov-2024 05:33:29 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- StdException BEGIN
      An exception was thrown while processing module CVNEvaluator/cvneva run: 74511302 subRun: 1 event: 8603
      CHECK failed: it != end(): key not found: serving_default
    ---- StdException END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 2 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-2c26-e5c4-9feb-fe46.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_86_20231203T125043Z_gen_g4_detsim_hitreco__20240508T022819Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_86_20231203T125043Z_gen_g4_detsim_hitreco__20240508T022819Z_reco2_reco_2024-11-23T_051857Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work