Jobsub ID 108727.48@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108727.48@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 03:57:40
Site	UK_Imperial
Entry	DUNE_T2_UK_London_IC_ceprod01
Last heartbeat	2024-11-23 04:16:41
From worker node	Hostname	wj54.grid.hep.ph.ic.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:02:35
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420351_52_20231204T014451Z_gen_g4_detsim_hitreco__20240509T195411Z_reco2.root
Jobscript	Exit code	0
	Real time	13m (826s)
	CPU time	3m (204s = 24%)
Outputting started	2024-11-23 04:16:21
Output files
Finished	2024-11-23 04:16:41
List job events Wrapper job log
Jobscript log (last 10,000 characters)

ter:pmtrajfittc@BeginModule  23-Nov-2024 04:14:46 GMT run: 6420351 subRun: 1 event: 5204
1st wire C:0 T:23 P:2 W:669 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:14:46 GMT run: 6420351 subRun: 1 event: 5204
1st wire C:0 T:23 P:2 W:764 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:14:46 GMT run: 6420351 subRun: 1 event: 5204
1st wire C:0 T:23 P:2 W:768 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:14:48 GMT run: 6420351 subRun: 1 event: 5204
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 1432, 1099, 1244
minwire 0: 2415
minwire 1: 597
minwire 2: 2392
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Could not find serving_default in model signatures.
[libprotobuf FATAL /cvmfs/larsoft.opensciencegrid.org/products/protobuf/v3_21_12a/Linux64bit+3.10-2.17-e26/include/google/protobuf/map.h:1300] CHECK failed: it != end(): key not found: serving_default
23-Nov-2024 04:16:20 GMT  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/fb/60/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420351_52_20231204T014451Z_gen_g4_detsim_hitreco__20240509T195411Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.0198558      65.3945       250.003       5.77733       106.614         4     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0198558     0.0252883     0.0390229     0.0211372    0.00794698        4     
reco:linecluster:LineCluster                           0.211729      0.283841      0.481312      0.221162      0.114248         4     
reco:trajcluster:TrajCluster                           0.0639524     0.911559       3.43984      0.0712202      1.45972         4     
reco:pandora:StandardPandora                           0.106263       61.5871       244.661      0.790304       105.699         4     
reco:pandoraTrack:LArPandoraTrackCreation             0.000289253   0.00101188    0.00313114    0.000313554    0.0012237        4     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000390531   0.00140782    0.00442743    0.00040666    0.00174341        4     
reco:pandoracalo:Calorimetry                          0.000216603   0.000677722   0.00196295    0.00026567    0.000742506       4     
reco:pandorapid:Chi2ParticleID                        8.7171e-05    0.000362411    0.0011324    0.000115036   0.000444838       4     
reco:trkshowersplit:TrackShowerHits                   0.000455365    0.0195818     0.0740155    0.00192811     0.0314501        4     
reco:pmtrack:PMAlgTrackMaker                          0.00601798      3.10672       12.3596      0.0306464      5.34215         4     
reco:pmtrackcalo:Calorimetry                          0.000399178    0.0100189     0.0353831    0.00214664     0.0147017        4     
reco:pmtrackpid:Chi2ParticleID                        0.000141756   0.000721105   0.00133366    0.000704504   0.000557665       4     
reco:pmtrajfit:PMAlgTrajFitter                        0.000476729   0.00116076     0.0026804    0.000742964   0.00088563        4     
reco:pmtrajfitcalo:Calorimetry                        0.000152968   0.000181573   0.000204542   0.000184392   2.03319e-05       4     
reco:pmtrajfitpid:Chi2ParticleID                      6.8647e-05    7.39505e-05   7.8911e-05    7.4122e-05    3.71555e-06       4     
reco:pmtracktc:PMAlgTrackMaker                        0.00999342      3.42481       13.4584      0.115426       5.79346         4     
reco:pmtracktccalo:Calorimetry                        0.000385714    0.0103175     0.0396976    0.000593317    0.0169633        4     
reco:pmtracktcpid:Chi2ParticleID                      0.000124707   0.000412467    0.0012446    0.000140283   0.000480582       4     
reco:pmtrajfittc:PMAlgTrajFitter                       0.0346815      1.7775        6.72953       0.17289       2.85962         4     
reco:pmtrajfittccalo:Calorimetry                      0.000444863    0.012072      0.0466556    0.00059382     0.019967         4     
reco:pmtrajfittcpid:Chi2ParticleID                    0.000131109   0.000432823   0.00129014    0.00015502    0.000495137       4     
reco:blurredcluster:BlurredClustering                  0.104443       1.28855       2.52387       1.26294      0.957171         4     
reco:emtrkmichelid:EmTrackMichelId                      1.02756       13.564        50.322        1.4533        21.2232         4     
reco:emshower:EMShower                                0.000447372   0.00181486    0.00338642    0.00171283    0.00132482        4     
reco:cvnmap:CVNMapper                                 4.4442e-05     0.0117893     0.0468095    0.000151661    0.0202191        4     
reco:cvneva:CVNEvaluator                              3.1414e-05    0.000108552   0.000253833    4.041e-05    0.000102795       3     
reco:energyrecnumu:EnergyReco                         0.00337774    0.00452792    0.00570153    0.00450448    0.000948831       3     
reco:energyrecnue:EnergyReco                          0.00287435    0.00325395    0.00381384    0.00307367    0.000404176       3     
reco:energyrecnc:EnergyReco                           0.00284492    0.00325535    0.00386919    0.00305195    0.000442199       3     
reco:energyrecnumurange:EnergyReco                    0.00279238      0.00326     0.00387142    0.00311619    0.000452102       3     
reco:energyrecnumumcs:EnergyReco                      0.00278198    0.00323044    0.00386598    0.00304337    0.000461889       3     
reco:opdec:Deconvolution                               0.396353      0.449106      0.485547      0.465419      0.0381969        3     
reco:ophitspe:OpHitFinderDeco                           2.00615       2.1953        2.45841       2.12133      0.191897         3     
reco:opflash:OpFlashFinder                            0.00158725    0.00278699    0.00469112    0.00208262    0.00136152        3     
reco:rns:RandomNumberSaver                             4.83e-05     0.00014305    0.000326608   5.4243e-05    0.000129818       3     
reco:anglereconue:NuAngularReco                       0.000259311   0.000508164   0.000935791   0.000329391   0.000303728       3     
reco:anglereconumu:NuAngularReco                      0.000124798   0.000164854   0.000185519   0.000184244   2.83284e-05       3     
reco:anglereconuepfps:NuAngularReco                   0.000185918   0.00058116    0.00128495    0.000272607   0.000498915       3     
reco:anglereconumupfps:NuAngularReco                  0.000162461   0.000207425   0.000255244   0.000204569   3.79323e-05       3     
[art]:TriggerResults:TriggerResultInserter             1.529e-05    2.78993e-05   4.7156e-05    2.1252e-05    1.38323e-05       3     
end_path:out1:RootOutput                               6.486e-06    1.03687e-05   1.5448e-05     9.172e-06    3.7553e-06        3     
end_path:out1:RootOutput(write)                        0.100021      0.143316      0.190369       0.13956       0.03698         3     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 7310.23 MB
  Peak resident set size usage (VmHWM): 1397.71 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Nov-2024 04:16:20 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- StdException BEGIN
      An exception was thrown while processing module CVNEvaluator/cvneva run: 6420351 subRun: 1 event: 5204
      CHECK failed: it != end(): key not found: serving_default
    ---- StdException END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 3 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-9d8b-7f63-8e44-4cb6.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420351_52_20231204T014451Z_gen_g4_detsim_hitreco__20240509T195411Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420351_52_20231204T014451Z_gen_g4_detsim_hitreco__20240509T195411Z_reco2_reco_2024-11-23T_040241Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work