Jobsub ID 301716.92@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301716.92@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:49:33
Site	UK_Imperial
Entry	DUNE_T2_UK_London_IC_ceprod01
Last heartbeat	2024-11-23 04:58:03
From worker node	Hostname	wh19.grid.hep.ph.ic.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:50:41
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6400858_383_20231202T080255Z_gen_g4_detsim_hitreco__20240507T210306Z_reco2.root
Jobscript	Exit code	0
	Real time	7m (428s)
	CPU time	1m (76s = 17%)
Outputting started	2024-11-23 04:57:50
Output files
Finished	2024-11-23 04:58:03
List job events Wrapper job log
Jobscript log (last 10,000 characters)

v-2024 04:57:23 GMT run: 6400858 subRun: 1 event: 38303
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 04:57:23 GMT run: 6400858 subRun: 1 event: 38303
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 04:57:23 GMT run: 6400858 subRun: 1 event: 38303
Hit sorting problem.
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:57:23 GMT run: 6400858 subRun: 1 event: 38303
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:57:23 GMT run: 6400858 subRun: 1 event: 38303
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 107, 84, 111
minwire 0: 1408
minwire 1: 1372
minwire 2: 1423
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Could not find serving_default in model signatures.
[libprotobuf FATAL /cvmfs/larsoft.opensciencegrid.org/products/protobuf/v3_21_12a/Linux64bit+3.10-2.17-e26/include/google/protobuf/map.h:1300] CHECK failed: it != end(): key not found: serving_default
23-Nov-2024 04:57:49 GMT  Closed input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/1a/81/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6400858_383_20231202T080255Z_gen_g4_detsim_hitreco__20240507T210306Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                            0.00603324      79.207        236.078       1.53742       110.926         3     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00290848    0.00410535    0.00603324    0.00337432    0.00137643        3     
reco:linecluster:LineCluster                           0.0408042     0.0510925     0.0703654     0.0421079     0.0136384        3     
reco:trajcluster:TrajCluster                          0.00946915     0.0149953     0.0191345     0.0163824    0.00406592        3     
reco:pandora:StandardPandora                           0.0445048      78.0852       234.164      0.0466482      110.365         3     
reco:pandoraTrack:LArPandoraTrackCreation             0.000161735   0.000771868   0.00198341    0.00017046    0.000856696       3     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000195066   0.00109477    0.00275927    0.000329961   0.00117827        3     
reco:pandoracalo:Calorimetry                          0.000119303   0.00046804    0.00115163    0.000133186   0.000483406       3     
reco:pandorapid:Chi2ParticleID                        5.0765e-05    0.00025542    0.000647816    6.768e-05    0.000277552       3     
reco:trkshowersplit:TrackShowerHits                   0.000148407   0.000807754   0.00180073    0.000474124   0.000714621       3     
reco:pmtrack:PMAlgTrackMaker                          0.00294988     0.102516      0.299829      0.0047688     0.139523         3     
reco:pmtrackcalo:Calorimetry                          0.000110483   0.000478661   0.00106399    0.000261509   0.000418458       3     
reco:pmtrackpid:Chi2ParticleID                        4.6865e-05    8.14663e-05   0.000118611   7.8923e-05    2.93453e-05       3     
reco:pmtrajfit:PMAlgTrajFitter                        0.000272316   0.000731463   0.00155846    0.000363608   0.000585965       3     
reco:pmtrajfitcalo:Calorimetry                         9.597e-05    0.000102013   0.000110705   9.9364e-05    6.30042e-06       3     
reco:pmtrajfitpid:Chi2ParticleID                       4.145e-05    4.25673e-05   4.3697e-05    4.2555e-05    9.17375e-07       3     
reco:pmtracktc:PMAlgTrackMaker                        0.00329009     0.0639413     0.177004      0.0115294     0.0800184        3     
reco:pmtracktccalo:Calorimetry                        0.000170878   0.00114572    0.00218672    0.00107956    0.000824291       3     
reco:pmtracktcpid:Chi2ParticleID                      6.4903e-05    0.000305684   0.000733695   0.000118455   0.000303438       3     
reco:pmtrajfittc:PMAlgTrajFitter                      0.00274827     0.227752      0.671883      0.0086245     0.314057         3     
reco:pmtrajfittccalo:Calorimetry                      0.000186647   0.000532101   0.00118639    0.000223262   0.000462896       3     
reco:pmtrajfittcpid:Chi2ParticleID                    6.6954e-05    9.3505e-05    0.000142397   7.1164e-05    3.46146e-05       3     
reco:blurredcluster:BlurredClustering                  0.0806841     0.275014      0.569778       0.17458      0.211925         3     
reco:emtrkmichelid:EmTrackMichelId                     0.252663       0.99385       2.39334      0.335544      0.990168         3     
reco:emshower:EMShower                                0.000285963   0.00116832    0.00284306    0.000375931   0.00118479        3     
reco:cvnmap:CVNMapper                                 4.3474e-05    0.00879418     0.0261346    0.000204492    0.0122617        3     
reco:cvneva:CVNEvaluator                              2.4166e-05    0.000110865   0.000197565   0.000110865   8.66995e-05       2     
reco:energyrecnumu:EnergyReco                         0.00273199    0.00372974    0.00472749    0.00372974    0.000997754       2     
reco:energyrecnue:EnergyReco                           0.0024369    0.00247972    0.00252255    0.00247972    4.28225e-05       2     
reco:energyrecnc:EnergyReco                           0.00243151     0.0024683    0.00250509     0.0024683     3.679e-05        2     
reco:energyrecnumurange:EnergyReco                     0.0024378    0.00244387    0.00244995    0.00244387     6.074e-06        2     
reco:energyrecnumumcs:EnergyReco                       0.0024395    0.00246725    0.00249501    0.00246725    2.77535e-05       2     
reco:opdec:Deconvolution                               0.100264      0.108747       0.11723      0.108747     0.00848289        2     
reco:ophitspe:OpHitFinderDeco                          0.734645      0.759129      0.783613      0.759129      0.0244837        2     
reco:opflash:OpFlashFinder                            0.00059854    0.00177629    0.00295404    0.00177629    0.00117775        2     
reco:rns:RandomNumberSaver                            3.5426e-05    0.000149368   0.00026331    0.000149368   0.000113942       2     
reco:anglereconue:NuAngularReco                       0.000319641   0.000536079   0.000752516   0.000536079   0.000216437       2     
reco:anglereconumu:NuAngularReco                      0.000158256   0.000159657   0.000161058   0.000159657    1.401e-06        2     
reco:anglereconuepfps:NuAngularReco                   0.00022772    0.000659856   0.00109199    0.000659856   0.000432137       2     
reco:anglereconumupfps:NuAngularReco                  0.000212455   0.000228205   0.000243956   0.000228205   1.57505e-05       2     
[art]:TriggerResults:TriggerResultInserter            2.0302e-05    3.1852e-05    4.3402e-05    3.1852e-05     1.155e-05        2     
end_path:out1:RootOutput                               3.689e-06    8.8325e-06    1.3976e-05    8.8325e-06    5.1435e-06        2     
end_path:out1:RootOutput(write)                        0.046634      0.0505309     0.0544277     0.0505309    0.00389683        2     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 7071.4 MB
  Peak resident set size usage (VmHWM): 1154.75 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Nov-2024 04:57:49 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- StdException BEGIN
      An exception was thrown while processing module CVNEvaluator/cvneva run: 6400858 subRun: 1 event: 38303
      CHECK failed: it != end(): key not found: serving_default
    ---- StdException END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 2 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-063f-fef6-66b9-212d.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6400858_383_20231202T080255Z_gen_g4_detsim_hitreco__20240507T210306Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6400858_383_20231202T080255Z_gen_g4_detsim_hitreco__20240507T210306Z_reco2_reco_2024-11-23T_045044Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work