Jobsub ID 301820.124@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301820.124@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:41:53
Site	UK_Imperial
Entry	DUNE_T2_UK_London_IC_ceprod03
Last heartbeat	2024-11-23 05:57:51
From worker node	Hostname	wj17.grid.hep.ph.ic.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:43:38
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467222_374_20231207T042356Z_gen_g4_detsim_hitreco__20240510T033614Z_reco2.root
Jobscript	Exit code	0
	Real time	13m (830s)
	CPU time	1m (102s = 12%)
Outputting started	2024-11-23 05:57:29
Output files
Finished	2024-11-23 05:57:51
List job events Wrapper job log
Jobscript log (last 10,000 characters)

vent: 37405
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 05:56:50 GMT run: 6467222 subRun: 1 event: 37405
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:56:50 GMT run: 6467222 subRun: 1 event: 37405
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:56:50 GMT run: 6467222 subRun: 1 event: 37405
1st wire C:0 T:20 P:2 W:595 does not exist (max wire number: 480)
%MSG
Boundary wire vector sizes: 54, 45, 35
minwire 0: 2606
minwire 1: 579
minwire 2: 2606
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max wires due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Could not find serving_default in model signatures.
[libprotobuf FATAL /cvmfs/larsoft.opensciencegrid.org/products/protobuf/v3_21_12a/Linux64bit+3.10-2.17-e26/include/google/protobuf/map.h:1300] CHECK failed: it != end(): key not found: serving_default
23-Nov-2024 05:57:28 GMT  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/19/df/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467222_374_20231207T042356Z_gen_g4_detsim_hitreco__20240510T033614Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.0617239      50.9452       240.256       4.08805       94.6895         5     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0214075     0.040783      0.0617239     0.0392355     0.0140574        5     
reco:linecluster:LineCluster                           0.276809      0.330548      0.389922      0.323049      0.0367473        5     
reco:trajcluster:TrajCluster                           0.0610471     0.112315      0.291059      0.0643807     0.0896664        5     
reco:pandora:StandardPandora                           0.0788807      47.3172       235.97       0.123306       94.3267         5     
reco:pandoraTrack:LArPandoraTrackCreation             0.000162314   0.000659063    0.0023907    0.000262092   0.000867089       5     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000255569   0.000914349    0.0032119    0.000370046   0.00114973        5     
reco:pandoracalo:Calorimetry                          0.000131075   0.000390418   0.00128935    0.000188206   0.000450402       5     
reco:pandorapid:Chi2ParticleID                        5.1774e-05    0.000200164   0.000715889   8.0025e-05    0.000258322       5     
reco:trkshowersplit:TrackShowerHits                   0.000136244   0.00065362    0.00208905    0.000277653   0.000729931       5     
reco:pmtrack:PMAlgTrackMaker                          0.00150248     0.0215019     0.0426305     0.0178912     0.017571         5     
reco:pmtrackcalo:Calorimetry                          0.000101376   0.000809768   0.00264619    0.000377007   0.000931856       5     
reco:pmtrackpid:Chi2ParticleID                         4.179e-05    0.00027671    0.000901807   0.00013668    0.000315893       5     
reco:pmtrajfit:PMAlgTrajFitter                        0.000269411   0.000706783   0.00198358    0.000431436   0.000641344       5     
reco:pmtrajfitcalo:Calorimetry                         9.546e-05    0.000124563   0.000165719   0.000116785   2.72661e-05       5     
reco:pmtrajfitpid:Chi2ParticleID                      4.4737e-05    5.72286e-05    8.549e-05    4.5819e-05    1.61395e-05       5     
reco:pmtracktc:PMAlgTrackMaker                         0.0013462     0.0411656     0.146208      0.0175674     0.053268         5     
reco:pmtracktccalo:Calorimetry                        0.000103247   0.000423233   0.000962683   0.000350595   0.000286599       5     
reco:pmtracktcpid:Chi2ParticleID                      4.2206e-05    0.000118828   0.000218958   0.000107215   5.76781e-05       5     
reco:pmtrajfittc:PMAlgTrajFitter                      0.000275771    0.0600613     0.119269      0.0521819     0.0390354        5     
reco:pmtrajfittccalo:Calorimetry                      9.6644e-05    0.00048697    0.000888751   0.000422053   0.000264138       5     
reco:pmtrajfittcpid:Chi2ParticleID                    4.1915e-05    0.000125857    0.0002051    0.000111757   5.48703e-05       5     
reco:blurredcluster:BlurredClustering                  0.192141       1.45215       3.98948      0.442661       1.46253         5     
reco:emtrkmichelid:EmTrackMichelId                     0.381852      0.996488       2.18351      0.807728      0.615805         5     
reco:emshower:EMShower                                0.00028642    0.000989779   0.00323911    0.000437808    0.0011287        5     
reco:cvnmap:CVNMapper                                  3.912e-05     0.0294798     0.147008     4.5682e-05     0.058764         5     
reco:cvneva:CVNEvaluator                              2.3343e-05    7.8512e-05    0.000233057   2.8824e-05    8.92887e-05       4     
reco:energyrecnumu:EnergyReco                          0.0028266     0.0038628    0.00560827    0.00350816    0.00112453        4     
reco:energyrecnue:EnergyReco                          0.00276258    0.00313525    0.00410602    0.00283621    0.000562658       4     
reco:energyrecnc:EnergyReco                           0.00279755    0.00319035     0.0040133    0.00297528    0.000482283       4     
reco:energyrecnumurange:EnergyReco                     0.0027511    0.00311595    0.00395282    0.00287994    0.000491169       4     
reco:energyrecnumumcs:EnergyReco                      0.00285548    0.00317952    0.00397162     0.0029455    0.000459165       4     
reco:opdec:Deconvolution                               0.268956      0.702954      0.976207      0.783326      0.271824         4     
reco:ophitspe:OpHitFinderDeco                          0.547488       1.19703       1.55198       1.34433      0.392849         4     
reco:opflash:OpFlashFinder                            0.000560545   0.00175066    0.00411721    0.00116245    0.00140538        4     
reco:rns:RandomNumberSaver                            3.2541e-05    9.9272e-05    0.000279537   4.2505e-05    0.000104156       4     
reco:anglereconue:NuAngularReco                       0.000230844   0.00489604     0.0181988    0.000577259   0.00768382        4     
reco:anglereconumu:NuAngularReco                      0.00010579    0.000140045   0.000174591    0.0001399    2.79679e-05       4     
reco:anglereconuepfps:NuAngularReco                   0.000169996   0.000463159   0.00116535    0.000258643   0.000408251       4     
reco:anglereconumupfps:NuAngularReco                  0.000155496   0.000191703   0.000241272   0.000185022   3.60072e-05       4     
[art]:TriggerResults:TriggerResultInserter            1.4853e-05    2.80255e-05   5.9824e-05    1.87125e-05   1.85286e-05       4     
end_path:out1:RootOutput                               4.295e-06    8.2395e-06    1.6207e-05     6.228e-06    4.66724e-06       4     
end_path:out1:RootOutput(write)                        0.0338802     0.0766822     0.0987263     0.0870611     0.0260868        4     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 7279.96 MB
  Peak resident set size usage (VmHWM): 1366.12 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Nov-2024 05:57:28 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- StdException BEGIN
      An exception was thrown while processing module CVNEvaluator/cvneva run: 6467222 subRun: 1 event: 37405
      CHECK failed: it != end(): key not found: serving_default
    ---- StdException END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 4 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-bf40-b86a-e60b-c7d1.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467222_374_20231207T042356Z_gen_g4_detsim_hitreco__20240510T033614Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6467222_374_20231207T042356Z_gen_g4_detsim_hitreco__20240510T033614Z_reco2_reco_2024-11-23T_054342Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work