Jobsub ID 301695.6@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301695.6@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:08:26
Site	UK_Imperial
Entry	DUNE_T2_UK_London_IC_ceprod00
Last heartbeat	2024-11-23 04:17:43
From worker node	Hostname	wm04.grid.hep.ph.ic.ac.uk
	cpuinfo	AMD EPYC 7742 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:09:30
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602992_691_20231206T233016Z_gen_g4_detsim_hitreco__20240510T031218Z_reco2.root
Jobscript	Exit code	0
	Real time	7m (474s)
	CPU time	1m (92s = 19%)
Outputting started	2024-11-23 04:17:25
Output files
Finished	2024-11-23 04:17:43
List job events Wrapper job log
Jobscript log (last 10,000 characters)

Check:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:16:52 GMT run: 50602992 subRun: 1 event: 69102
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:16:52 GMT run: 50602992 subRun: 1 event: 69102
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:16:52 GMT run: 50602992 subRun: 1 event: 69102
1st wire C:0 T:8 P:2 W:760 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:16:52 GMT run: 50602992 subRun: 1 event: 69102
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 173, 224, 204
minwire 0: 1173
minwire 1: 1877
minwire 2: 883
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Could not find serving_default in model signatures.
[libprotobuf FATAL /cvmfs/larsoft.opensciencegrid.org/products/protobuf/v3_21_12a/Linux64bit+3.10-2.17-e26/include/google/protobuf/map.h:1300] CHECK failed: it != end(): key not found: serving_default
23-Nov-2024 04:17:24 GMT  Closed input file "root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/ab/0f/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602992_691_20231206T233016Z_gen_g4_detsim_hitreco__20240510T031218Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.0776248      127.216       254.354       127.216       127.138         2     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0745759     0.0761004     0.0776248     0.0761004    0.00152445        2     
reco:linecluster:LineCluster                           0.231914      0.293547      0.355181      0.293547      0.0616332        2     
reco:trajcluster:TrajCluster                           0.0750944     0.110379      0.145663      0.110379      0.0352843        2     
reco:pandora:StandardPandora                           0.292685       125.347       250.402       125.347       125.055         2     
reco:pandoraTrack:LArPandoraTrackCreation             0.000186303   0.00148001    0.00277371    0.00148001     0.0012937        2     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000775737    0.0023391    0.00390247     0.0023391    0.00156337        2     
reco:pandoracalo:Calorimetry                          0.000179059   0.000897634   0.00161621    0.000897634   0.000718576       2     
reco:pandorapid:Chi2ParticleID                         9.537e-05    0.000514629   0.000933888   0.000514629   0.000419259       2     
reco:trkshowersplit:TrackShowerHits                   0.00182734    0.00234631    0.00286528    0.00234631    0.000518974       2     
reco:pmtrack:PMAlgTrackMaker                           0.0179564     0.534237       1.05052      0.534237       0.51628         2     
reco:pmtrackcalo:Calorimetry                          0.00226719    0.00265827    0.00304935    0.00265827    0.000391083       2     
reco:pmtrackpid:Chi2ParticleID                        0.000244302   0.00066528    0.00108626    0.00066528    0.000420978       2     
reco:pmtrajfit:PMAlgTrajFitter                        0.000718439   0.00137554    0.00203265    0.00137554    0.000657105       2     
reco:pmtrajfitcalo:Calorimetry                        0.000101452   0.000118053   0.000134655   0.000118053   1.66015e-05       2     
reco:pmtrajfitpid:Chi2ParticleID                      4.5235e-05    4.7114e-05    4.8993e-05    4.7114e-05     1.879e-06        2     
reco:pmtracktc:PMAlgTrackMaker                         0.0275815     0.298432      0.569283      0.298432      0.270851         2     
reco:pmtracktccalo:Calorimetry                        0.000280231   0.00123002     0.0021798    0.00123002    0.000949786       2     
reco:pmtracktcpid:Chi2ParticleID                      8.1113e-05    0.00016669    0.000252267   0.00016669    8.5577e-05        2     
reco:pmtrajfittc:PMAlgTrajFitter                       0.0148736     0.617518       1.22016      0.617518      0.602645         2     
reco:pmtrajfittccalo:Calorimetry                      0.000255504    0.0016358     0.0030161     0.0016358     0.0013803        2     
reco:pmtrajfittcpid:Chi2ParticleID                    8.0312e-05    0.000207793   0.000335274   0.000207793   0.000127481       2     
reco:blurredcluster:BlurredClustering                  0.110626       0.2293       0.347973       0.2293       0.118674         2     
reco:emtrkmichelid:EmTrackMichelId                     0.664696       3.14733       5.62996       3.14733       2.48263         2     
reco:emshower:EMShower                                0.000592751   0.00300742     0.0054221    0.00300742    0.00241467        2     
reco:cvnmap:CVNMapper                                 0.000392202    0.0131915     0.0259908     0.0131915     0.0127993        2     
reco:cvneva:CVNEvaluator                              0.00030643    0.00030643    0.00030643    0.00030643         0            1     
reco:energyrecnumu:EnergyReco                         0.00758375    0.00758375    0.00758375    0.00758375         0            1     
reco:energyrecnue:EnergyReco                          0.00353845    0.00353845    0.00353845    0.00353845         0            1     
reco:energyrecnc:EnergyReco                           0.00359968    0.00359968    0.00359968    0.00359968         0            1     
reco:energyrecnumurange:EnergyReco                     0.0034752     0.0034752     0.0034752     0.0034752         0            1     
reco:energyrecnumumcs:EnergyReco                      0.00357453    0.00357453    0.00357453    0.00357453         0            1     
reco:opdec:Deconvolution                                0.68145       0.68145       0.68145       0.68145          0            1     
reco:ophitspe:OpHitFinderDeco                           1.89295       1.89295       1.89295       1.89295          0            1     
reco:opflash:OpFlashFinder                            0.00601343    0.00601343    0.00601343    0.00601343         0            1     
reco:rns:RandomNumberSaver                            0.000429864   0.000429864   0.000429864   0.000429864        0            1     
reco:anglereconue:NuAngularReco                       0.00127881    0.00127881    0.00127881    0.00127881         0            1     
reco:anglereconumu:NuAngularReco                      0.000193106   0.000193106   0.000193106   0.000193106        0            1     
reco:anglereconuepfps:NuAngularReco                   0.00176432    0.00176432    0.00176432    0.00176432         0            1     
reco:anglereconumupfps:NuAngularReco                  0.000209416   0.000209416   0.000209416   0.000209416        0            1     
[art]:TriggerResults:TriggerResultInserter            5.3922e-05    5.3922e-05    5.3922e-05    5.3922e-05         0            1     
end_path:out1:RootOutput                              2.2904e-05    2.2904e-05    2.2904e-05    2.2904e-05         0            1     
end_path:out1:RootOutput(write)                        0.100585      0.100585      0.100585      0.100585          0            1     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 37136.2 MB
  Peak resident set size usage (VmHWM): 1147.54 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Nov-2024 04:17:24 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- StdException BEGIN
      An exception was thrown while processing module CVNEvaluator/cvneva run: 50602992 subRun: 1 event: 69102
      CHECK failed: it != end(): key not found: serving_default
    ---- StdException END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 1 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-b216-3c80-cfe7-c0d9.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602992_691_20231206T233016Z_gen_g4_detsim_hitreco__20240510T031218Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602992_691_20231206T233016Z_gen_g4_detsim_hitreco__20240510T031218Z_reco2_reco_2024-11-23T_040934Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work