Jobsub ID 108779.47@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108779.47@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:04:07
Site	UK_Imperial
Entry	DUNE_T2_UK_London_IC_ceprod01
Last heartbeat	2024-11-23 05:16:08
From worker node	Hostname	wj54.grid.hep.ph.ic.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:05:03
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50604274_37_20231207T001757Z_gen_g4_detsim_hitreco__20240510T044328Z_reco2.root
Jobscript	Exit code	0
	Real time	10m (643s)
	CPU time	1m (101s = 15%)
Outputting started	2024-11-23 05:15:46
Output files
Finished	2024-11-23 05:16:08
List job events Wrapper job log
Jobscript log (last 10,000 characters)

odule  23-Nov-2024 05:15:06 GMT run: 50604274 subRun: 1 event: 3702
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:15:06 GMT run: 50604274 subRun: 1 event: 3702
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:15:06 GMT run: 50604274 subRun: 1 event: 3702
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:15:06 GMT run: 50604274 subRun: 1 event: 3702
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:15:06 GMT run: 50604274 subRun: 1 event: 3702
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 432, 681, 381
minwire 0: 591
minwire 1: 797
minwire 2: 253
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Could not find serving_default in model signatures.
[libprotobuf FATAL /cvmfs/larsoft.opensciencegrid.org/products/protobuf/v3_21_12a/Linux64bit+3.10-2.17-e26/include/google/protobuf/map.h:1300] CHECK failed: it != end(): key not found: serving_default
23-Nov-2024 05:15:45 GMT  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/5f/46/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50604274_37_20231207T001757Z_gen_g4_detsim_hitreco__20240510T044328Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.0309964      121.756       243.482       121.756       121.725         2     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0309964     0.0331237     0.035251      0.0331237    0.00212729        2     
reco:linecluster:LineCluster                            0.23269      0.273218      0.313747      0.273218      0.0405284        2     
reco:trajcluster:TrajCluster                           0.0822791     0.127688      0.173098      0.127688      0.0454093        2     
reco:pandora:StandardPandora                           0.388109       119.665       238.942       119.665       119.277         2     
reco:pandoraTrack:LArPandoraTrackCreation             0.000264535   0.00196436    0.00366418    0.00196436    0.00169982        2     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000649689   0.00280429    0.00495889    0.00280429     0.0021546        2     
reco:pandoracalo:Calorimetry                          0.000221042   0.00112805    0.00203506    0.00112805    0.000907012       2     
reco:pandorapid:Chi2ParticleID                        0.00010455    0.000634304   0.00116406    0.000634304   0.000529754       2     
reco:trkshowersplit:TrackShowerHits                   0.00339571    0.00472965    0.00606358    0.00472965    0.00133394        2     
reco:pmtrack:PMAlgTrackMaker                           0.021873      0.810324       1.59878      0.810324      0.788451         2     
reco:pmtrackcalo:Calorimetry                           0.0038411    0.00600476    0.00816841    0.00600476    0.00216365        2     
reco:pmtrackpid:Chi2ParticleID                        0.00022872    0.000790427   0.00135213    0.000790427   0.000561708       2     
reco:pmtrajfit:PMAlgTrajFitter                        0.000658782   0.00171873    0.00277868    0.00171873    0.00105995        2     
reco:pmtrajfitcalo:Calorimetry                        0.000106667   0.000150158   0.00019365    0.000150158   4.34915e-05       2     
reco:pmtrajfitpid:Chi2ParticleID                      4.3998e-05    6.16545e-05   7.9311e-05    6.16545e-05   1.76565e-05       2     
reco:pmtracktc:PMAlgTrackMaker                         0.0110378     0.646203       1.28137      0.646203      0.635165         2     
reco:pmtracktccalo:Calorimetry                        0.000534536   0.00416084    0.00778714    0.00416084     0.0036263        2     
reco:pmtracktcpid:Chi2ParticleID                      0.00015559    0.000197493   0.000239396   0.000197493   4.1903e-05        2     
reco:pmtrajfittc:PMAlgTrajFitter                       0.0585125     0.867613       1.67671      0.867613       0.8091          2     
reco:pmtrajfittccalo:Calorimetry                      0.000545406   0.00302064    0.00549588    0.00302064    0.00247524        2     
reco:pmtrajfittcpid:Chi2ParticleID                    0.000156127   0.000206968   0.000257809   0.000206968   5.0841e-05        2     
reco:blurredcluster:BlurredClustering                  0.242973       1.35734       2.47172       1.35734       1.11437         2     
reco:emtrkmichelid:EmTrackMichelId                      1.15458       6.85318       12.5518       6.85318       5.6986          2     
reco:emshower:EMShower                                0.000863537   0.00233418    0.00380483    0.00233418    0.00147065        2     
reco:cvnmap:CVNMapper                                 0.00028283     0.0127786     0.0252743     0.0127786     0.0124957        2     
reco:cvneva:CVNEvaluator                              0.000267884   0.000267884   0.000267884   0.000267884        0            1     
reco:energyrecnumu:EnergyReco                         0.00622007    0.00622007    0.00622007    0.00622007         0            1     
reco:energyrecnue:EnergyReco                           0.0032267     0.0032267     0.0032267     0.0032267         0            1     
reco:energyrecnc:EnergyReco                            0.0031052     0.0031052     0.0031052     0.0031052         0            1     
reco:energyrecnumurange:EnergyReco                    0.00324002    0.00324002    0.00324002    0.00324002         0            1     
reco:energyrecnumumcs:EnergyReco                      0.00303846    0.00303846    0.00303846    0.00303846         0            1     
reco:opdec:Deconvolution                               0.453697      0.453697      0.453697      0.453697          0            1     
reco:ophitspe:OpHitFinderDeco                           2.00599       2.00599       2.00599       2.00599          0            1     
reco:opflash:OpFlashFinder                            0.00453582    0.00453582    0.00453582    0.00453582         0            1     
reco:rns:RandomNumberSaver                            0.000341662   0.000341662   0.000341662   0.000341662        0            1     
reco:anglereconue:NuAngularReco                       0.00101664    0.00101664    0.00101664    0.00101664         0            1     
reco:anglereconumu:NuAngularReco                      0.000180203   0.000180203   0.000180203   0.000180203        0            1     
reco:anglereconuepfps:NuAngularReco                   0.00137715    0.00137715    0.00137715    0.00137715         0            1     
reco:anglereconumupfps:NuAngularReco                  0.000268159   0.000268159   0.000268159   0.000268159        0            1     
[art]:TriggerResults:TriggerResultInserter            5.4015e-05    5.4015e-05    5.4015e-05    5.4015e-05         0            1     
end_path:out1:RootOutput                              3.1925e-05    3.1925e-05    3.1925e-05    3.1925e-05         0            1     
end_path:out1:RootOutput(write)                        0.182178      0.182178      0.182178      0.182178          0            1     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 7321.24 MB
  Peak resident set size usage (VmHWM): 1403.79 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Nov-2024 05:15:45 GMT ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- StdException BEGIN
      An exception was thrown while processing module CVNEvaluator/cvneva run: 50604274 subRun: 1 event: 3702
      CHECK failed: it != end(): key not found: serving_default
    ---- StdException END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 1 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-477b-ed69-10c1-8cac.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50604274_37_20231207T001757Z_gen_g4_detsim_hitreco__20240510T044328Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50604274_37_20231207T001757Z_gen_g4_detsim_hitreco__20240510T044328Z_reco2_reco_2024-11-23T_050509Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work