Jobsub ID 108787.50@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108787.50@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:19:04
Site	UK_Imperial
Entry	DUNE_T2_UK_London_IC_ceprod01
Last heartbeat	2024-11-23 05:32:16
From worker node	Hostname	wj54.grid.hep.ph.ic.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:19:54
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_411_20231204T140343Z_gen_g4_detsim_hitreco__20240509T221439Z_reco2.root
Jobscript	Exit code	0
	Real time	12m (721s)
	CPU time	3m (186s = 25%)
Outputting started	2024-11-23 05:31:56
Output files
Finished	2024-11-23 05:32:16
List job events Wrapper job log
Jobscript log (last 10,000 characters)

event: 41104
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:30:47 GMT run: 74517882 subRun: 1 event: 41104
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:30:47 GMT run: 74517882 subRun: 1 event: 41104
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:30:47 GMT run: 74517882 subRun: 1 event: 41104
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:30:48 GMT run: 74517882 subRun: 1 event: 41104
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 1500, 1604, 1217
minwire 0: 1616
minwire 1: 235
minwire 2: 1940
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Could not find serving_default in model signatures.
[libprotobuf FATAL /cvmfs/larsoft.opensciencegrid.org/products/protobuf/v3_21_12a/Linux64bit+3.10-2.17-e26/include/google/protobuf/map.h:1300] CHECK failed: it != end(): key not found: serving_default
23-Nov-2024 05:31:54 GMT  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/06/1a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_411_20231204T140343Z_gen_g4_detsim_hitreco__20240509T221439Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.0272311      63.5984       241.768       6.29932       102.898         4     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0245917     0.0363133     0.0648798     0.0278909     0.0165544        4     
reco:linecluster:LineCluster                           0.212286      0.291686      0.459573      0.247442      0.0988451        4     
reco:trajcluster:TrajCluster                           0.0628687     0.743469       2.76898      0.0710117      1.16944         4     
reco:pandora:StandardPandora                           0.0766102      60.2368       238.88       0.995279       103.141         4     
reco:pandoraTrack:LArPandoraTrackCreation             0.000208456   0.00110146     0.0035395    0.000328949   0.00140847        4     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000263488   0.00155475    0.00488431    0.000535599    0.0019298        4     
reco:pandoracalo:Calorimetry                          0.00016707    0.000685948   0.00204458    0.00026607    0.000785634       4     
reco:pandorapid:Chi2ParticleID                        5.5492e-05    0.000352369   0.00114522    0.000104381   0.000458516       4     
reco:trkshowersplit:TrackShowerHits                   0.000219453    0.0158422     0.0594045    0.00187245     0.0251793        4     
reco:pmtrack:PMAlgTrackMaker                          0.00904196      1.88258       7.39043      0.0654239      3.18026         4     
reco:pmtrackcalo:Calorimetry                          0.00021996    0.00869238     0.0336589    0.000445334    0.0144152        4     
reco:pmtrackpid:Chi2ParticleID                        8.6963e-05    0.00036767    0.00115497    0.00011437    0.000454716       4     
reco:pmtrajfit:PMAlgTrajFitter                        0.000336839   0.00117163     0.0028494    0.000750148   0.000984655       4     
reco:pmtrajfitcalo:Calorimetry                        0.000126962   0.000162552   0.000204839   0.000159203   3.54199e-05       4     
reco:pmtrajfitpid:Chi2ParticleID                      4.6293e-05    6.59758e-05    8.707e-05     6.527e-05    1.81207e-05       4     
reco:pmtracktc:PMAlgTrackMaker                         0.0205683      2.66954       10.6066      0.0255043      4.58246         4     
reco:pmtracktccalo:Calorimetry                        0.00031531    0.00688987     0.0231154    0.00206438    0.00946728        4     
reco:pmtracktcpid:Chi2ParticleID                      9.0164e-05    0.000622329   0.00133913    0.000530013   0.000542683       4     
reco:pmtrajfittc:PMAlgTrajFitter                       0.0213851      1.12548       4.30972      0.0854053      1.83872         4     
reco:pmtrajfittccalo:Calorimetry                      0.000391067    0.0124117     0.0479406    0.000657597    0.0205135        4     
reco:pmtrajfittcpid:Chi2ParticleID                    0.000110879    0.000611     0.00128093    0.000526097   0.000497572       4     
reco:blurredcluster:BlurredClustering                  0.0460508      1.85456       4.43619        1.468        1.79471         4     
reco:emtrkmichelid:EmTrackMichelId                      0.6448        9.93164       36.773        1.15437       15.4984         4     
reco:emshower:EMShower                                0.000627535   0.00188159    0.00415936    0.00136973    0.00138264        4     
reco:cvnmap:CVNMapper                                 6.9257e-05     0.008429      0.0332318    0.000207473    0.0143202        4     
reco:cvneva:CVNEvaluator                              4.2606e-05    0.000139426   0.000312822   6.2849e-05    0.000122888       3     
reco:energyrecnumu:EnergyReco                         0.00545614    0.00707928    0.00791059    0.00787112    0.00114785        3     
reco:energyrecnue:EnergyReco                          0.00438754    0.00538917    0.00733576    0.00444421    0.00137664        3     
reco:energyrecnc:EnergyReco                           0.00398431    0.00530385    0.00728367    0.00464357    0.00142558        3     
reco:energyrecnumurange:EnergyReco                    0.00399884     0.0052932    0.00731557    0.00456519     0.0014486        3     
reco:energyrecnumumcs:EnergyReco                      0.00410531    0.00532682    0.00730203    0.00457313    0.00140968        3     
reco:opdec:Deconvolution                               0.286825      0.421166      0.610194      0.366478      0.137562         3     
reco:ophitspe:OpHitFinderDeco                           1.26061       1.98582       2.97811       1.71873      0.726157         3     
reco:opflash:OpFlashFinder                            0.00126079    0.00226321    0.00367763     0.0018512    0.00102878        3     
reco:rns:RandomNumberSaver                            5.1349e-05    0.000139854   0.000315276   5.2938e-05    0.000124044       3     
reco:anglereconue:NuAngularReco                       0.000312339   0.000519627   0.000882814   0.000363728   0.000257667       3     
reco:anglereconumu:NuAngularReco                      0.000124159   0.000146211   0.000171739   0.000142734   1.95794e-05       3     
reco:anglereconuepfps:NuAngularReco                   0.000193707   0.000576541   0.00125769    0.000278228   0.000482879       3     
reco:anglereconumupfps:NuAngularReco                  0.00014771    0.000204332   0.000246551   0.000218735   4.16171e-05       3     
[art]:TriggerResults:TriggerResultInserter             2.158e-05    3.1543e-05    4.9066e-05    2.3983e-05    1.24294e-05       3     
end_path:out1:RootOutput                               6.786e-06    1.0169e-05    1.5683e-05     8.038e-06    3.93235e-06       3     
end_path:out1:RootOutput(write)                        0.0905244     0.116392      0.136311      0.122341      0.0191598        3     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 7481 MB
  Peak resident set size usage (VmHWM): 1568.38 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Nov-2024 05:31:54 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- StdException BEGIN
      An exception was thrown while processing module CVNEvaluator/cvneva run: 74517882 subRun: 1 event: 41104
      CHECK failed: it != end(): key not found: serving_default
    ---- StdException END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 3 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-258a-7722-b008-7948.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_411_20231204T140343Z_gen_g4_detsim_hitreco__20240509T221439Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74517882_411_20231204T140343Z_gen_g4_detsim_hitreco__20240509T221439Z_reco2_reco_2024-11-23T_052000Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work