Jobsub ID 301865.0@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301865.0@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 07:17:21
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_klomp
Last heartbeat	2024-11-23 07:20:56
From worker node	Hostname	wn-choc-034.farm.nikhef.nl
	cpuinfo	Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 07:18:14
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_333_20231203T045744Z_gen_g4_detsim_hitreco__20240508T075138Z_reco2.root
Jobscript	Exit code	0
	Real time	2m (149s)
	CPU time	1m (97s = 65%)
Outputting started	2024-11-23 07:20:43
Output files
Finished	2024-11-23 07:20:56
List job events Wrapper job log
Jobscript log (last 10,000 characters)

n: 1 event: 33302
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:20:42 CET run: 50572032 subRun: 1 event: 33302
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:20:42 CET run: 50572032 subRun: 1 event: 33302
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:20:42 CET run: 50572032 subRun: 1 event: 33302
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:20:42 CET run: 50572032 subRun: 1 event: 33302
Comparing two wires in the same plane: return failure
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:20:42 CET run: 50572032 subRun: 1 event: 33302
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:20:42 CET run: 50572032 subRun: 1 event: 33302
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:20:42 CET run: 50572032 subRun: 1 event: 33302
TuneFullTree failed.
%MSG
23-Nov-2024 08:20:42 CET  Closed input file "root://penguin12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/e0/b8/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_333_20231203T045744Z_gen_g4_detsim_hitreco__20240508T075138Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                            0.00127492      8.31488       16.6285       8.31488       8.3136          2     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00127492    0.00134472    0.00141452    0.00134472    6.9798e-05        2     
reco:linecluster:LineCluster                           0.0475679     0.0488293     0.0500907     0.0488293    0.00126139        2     
reco:trajcluster:TrajCluster                           0.072281      0.133912      0.195543      0.133912      0.0616308        2     
reco:pandora:StandardPandora                            2.41083       2.78998       3.16912       2.78998      0.379143         2     
reco:pandoraTrack:LArPandoraTrackCreation             0.00844856     0.0090471    0.00964565     0.0090471    0.000598545       2     
reco:pandoraShower:LArPandoraModularShowerCreation     0.0191082     0.0197802     0.0204522     0.0197802    0.000672005       2     
reco:pandoracalo:Calorimetry                          0.00588895    0.00639722    0.00690548    0.00639722    0.000508266       2     
reco:pandorapid:Chi2ParticleID                        0.000328847   0.000987993   0.00164714    0.000987993   0.000659146       2     
reco:trkshowersplit:TrackShowerHits                   0.00401599    0.00645046    0.00888492    0.00645046    0.00243446        2     
reco:pmtrack:PMAlgTrackMaker                           0.462272       1.63614        2.81         1.63614       1.17387         2     
reco:pmtrackcalo:Calorimetry                           0.0024131    0.00543413    0.00845517    0.00543413    0.00302103        2     
reco:pmtrackpid:Chi2ParticleID                        0.000211128   0.000326872   0.000442615   0.000326872   0.000115744       2     
reco:pmtrajfit:PMAlgTrajFitter                          1.10745       1.11698       1.12652       1.11698     0.00953486        2     
reco:pmtrajfitcalo:Calorimetry                        0.00505401    0.00581045    0.00656689    0.00581045    0.000756442       2     
reco:pmtrajfitpid:Chi2ParticleID                      0.000229696   0.000303769   0.000377841   0.000303769   7.40725e-05       2     
reco:pmtracktc:PMAlgTrackMaker                         0.908875       1.7251        2.54132       1.7251       0.816221         2     
reco:pmtracktccalo:Calorimetry                        0.00231001    0.00370754    0.00510506    0.00370754    0.00139753        2     
reco:pmtracktcpid:Chi2ParticleID                      0.000214513   0.000281918   0.000349324   0.000281918   6.74055e-05       2     
reco:pmtrajfittc:PMAlgTrajFitter                       0.881538       1.11933       1.35713       1.11933      0.237795         2     
reco:pmtrajfittccalo:Calorimetry                      0.00334121    0.00537991    0.00741861    0.00537991     0.0020387        2     
reco:pmtrajfittcpid:Chi2ParticleID                    0.000251965   0.000333799   0.000415634   0.000333799   8.18345e-05       2     
reco:blurredcluster:BlurredClustering                  0.180661      0.180661      0.180661      0.180661          0            1     
reco:emtrkmichelid:EmTrackMichelId                       6.421         6.421         6.421         6.421           0            1     
reco:emshower:EMShower                                 0.0680978     0.0680978     0.0680978     0.0680978         0            1     
reco:cvnmap:CVNMapper                                  0.0194359     0.0194359     0.0194359     0.0194359         0            1     
reco:cvneva:CVNEvaluator                                1.20326       1.20326       1.20326       1.20326          0            1     
reco:energyrecnumu:EnergyReco                          0.0103251     0.0103251     0.0103251     0.0103251         0            1     
reco:energyrecnue:EnergyReco                          0.00156088    0.00156088    0.00156088    0.00156088         0            1     
reco:energyrecnc:EnergyReco                           0.00336194    0.00336194    0.00336194    0.00336194         0            1     
reco:energyrecnumurange:EnergyReco                     0.0014591     0.0014591     0.0014591     0.0014591         0            1     
reco:energyrecnumumcs:EnergyReco                       0.0039484     0.0039484     0.0039484     0.0039484         0            1     
reco:opdec:Deconvolution                               0.212933      0.212933      0.212933      0.212933          0            1     
reco:ophitspe:OpHitFinderDeco                           1.47131       1.47131       1.47131       1.47131          0            1     
reco:opflash:OpFlashFinder                            0.00422441    0.00422441    0.00422441    0.00422441         0            1     
reco:rns:RandomNumberSaver                            0.000287929   0.000287929   0.000287929   0.000287929        0            1     
reco:anglereconue:NuAngularReco                       0.00151686    0.00151686    0.00151686    0.00151686         0            1     
reco:anglereconumu:NuAngularReco                      0.000679158   0.000679158   0.000679158   0.000679158        0            1     
reco:anglereconuepfps:NuAngularReco                   0.00291394    0.00291394    0.00291394    0.00291394         0            1     
reco:anglereconumupfps:NuAngularReco                  0.000714745   0.000714745   0.000714745   0.000714745        0            1     
[art]:TriggerResults:TriggerResultInserter            5.0282e-05    5.0282e-05    5.0282e-05    5.0282e-05         0            1     
end_path:out1:RootOutput                               1.421e-05     1.421e-05     1.421e-05     1.421e-05         0            1     
end_path:out1:RootOutput(write)                        0.294869      0.294869      0.294869      0.294869          0            1     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6425.3 MB
  Peak resident set size usage (VmHWM): 1677.25 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Nov-2024 08:20:43 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module BlurredClustering/blurredcluster run: 50572032 subRun: 1 event: 33302
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 1 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-8603-f634-5b03-d132.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_333_20231203T045744Z_gen_g4_detsim_hitreco__20240508T075138Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_333_20231203T045744Z_gen_g4_detsim_hitreco__20240508T075138Z_reco2_reco_2024-11-23T_071817Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work