Jobsub ID 108878.15@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108878.15@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 07:16:18
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_brug
Last heartbeat	2024-11-23 07:24:00
From worker node	Hostname	wn-choc-039.farm.nikhef.nl
	cpuinfo	Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 07:17:51
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574864_270_20231203T154035Z_gen_g4_detsim_hitreco__20240508T031850Z_reco2.root
Jobscript	Exit code	0
	Real time	5m (356s)
	CPU time	5m (301s = 84%)
Outputting started	2024-11-23 07:23:47
Output files
Finished	2024-11-23 07:24:00
List job events Wrapper job log
Jobscript log (last 10,000 characters)

AlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:23:45 CET run: 50574864 subRun: 1 event: 27012
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:23:45 CET run: 50574864 subRun: 1 event: 27012
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:23:45 CET run: 50574864 subRun: 1 event: 27012
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:23:45 CET run: 50574864 subRun: 1 event: 27012
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:23:45 CET run: 50574864 subRun: 1 event: 27012
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:23:45 CET run: 50574864 subRun: 1 event: 27012
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:23:45 CET run: 50574864 subRun: 1 event: 27012
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:23:45 CET run: 50574864 subRun: 1 event: 27012
Comparing two wires in the same plane: return failure
%MSG
23-Nov-2024 08:23:46 CET  Closed input file "root://penguin12.grid.surfsara.nl:21094/pnfs/grid.sara.nl/data/dune/disk/RSE/fardet-hd/c4/0c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574864_270_20231203T154035Z_gen_g4_detsim_hitreco__20240508T031850Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                            0.00220546      17.0933       73.2395       8.21545       21.3774        12     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00120702    0.00203009    0.00618188    0.00168558    0.00127595       12     
reco:linecluster:LineCluster                           0.0114759     0.0496057     0.187134      0.0229968     0.0551259       12     
reco:trajcluster:TrajCluster                          0.00169238     0.401175       2.26454      0.0214398     0.715741        12     
reco:pandora:StandardPandora                            1.37459       2.07103       5.02296       1.54715       1.10639        12     
reco:pandoraTrack:LArPandoraTrackCreation              0.0001635    0.00647408     0.0249156    0.00225253    0.00871206       12     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000232543    0.0097647     0.0361955    0.00413665     0.0120869       12     
reco:pandoracalo:Calorimetry                          0.000118272    0.0111476     0.0594151    0.00179329     0.0187587       12     
reco:pandorapid:Chi2ParticleID                        4.8521e-05    0.000291007   0.00187368    0.000129401   0.000483296      12     
reco:trkshowersplit:TrackShowerHits                   0.000153444    0.0162225     0.091431     0.00123067     0.0287494       12     
reco:pmtrack:PMAlgTrackMaker                          0.00144652      1.32998       5.89149      0.214018       1.92587        12     
reco:pmtrackcalo:Calorimetry                          0.000106776   0.00635658     0.0274553    0.00130651    0.00909728       12     
reco:pmtrackpid:Chi2ParticleID                        4.2479e-05    0.000300782   0.000819492   0.000185505   0.000273091      12     
reco:pmtrajfit:PMAlgTrajFitter                        0.000276623     1.77199       11.0267      0.401064       3.11828        12     
reco:pmtrajfitcalo:Calorimetry                        9.4993e-05     0.0119287     0.0571383    0.00202796     0.0191195       12     
reco:pmtrajfitpid:Chi2ParticleID                      4.2516e-05    0.000232282    0.0008049    0.000178079   0.000199988      12     
reco:pmtracktc:PMAlgTrackMaker                        0.00132601      1.60087       7.44431      0.261448       2.37206        12     
reco:pmtracktccalo:Calorimetry                        9.7442e-05    0.00615758     0.0271884    0.00146175    0.00927014       12     
reco:pmtracktcpid:Chi2ParticleID                      4.1829e-05    0.000290355   0.000821005   0.000185361   0.000275162      12     
reco:pmtrajfittc:PMAlgTrajFitter                      0.000251129     1.38846       5.82251      0.327854       1.98973        12     
reco:pmtrajfittccalo:Calorimetry                      9.5895e-05     0.0123449     0.0612311    0.00172706     0.0202832       12     
reco:pmtrajfittcpid:Chi2ParticleID                    4.0436e-05    0.000404414    0.0012862    0.000193283   0.000429237      12     
reco:blurredcluster:BlurredClustering                  0.0160397     0.696511       2.23937      0.356454      0.766901        11     
reco:emtrkmichelid:EmTrackMichelId                     0.309181       8.44046       33.7889       3.52346       10.9876        11     
reco:emshower:EMShower                                0.000291567    0.0233417     0.0711855     0.0139435     0.0239867       11     
reco:cvnmap:CVNMapper                                 3.5416e-05     0.0100139     0.0243476     0.0111204    0.00843266       11     
reco:cvneva:CVNEvaluator                              2.3042e-05     0.746149       1.20906       1.16187       0.56416        11     
reco:energyrecnumu:EnergyReco                         0.000497877    0.0287016     0.154755     0.00274007     0.0525263       11     
reco:energyrecnue:EnergyReco                          0.000204282   0.00173273    0.00739307    0.000790288   0.00221504       11     
reco:energyrecnc:EnergyReco                           0.00262882    0.00394085    0.00973457    0.00291087    0.00222537       11     
reco:energyrecnumurange:EnergyReco                    0.000238487    0.0271554     0.154092     0.00267414     0.050486        11     
reco:energyrecnumumcs:EnergyReco                      0.00259361     0.0288462     0.153068     0.00346151     0.0496035       11     
reco:opdec:Deconvolution                               0.054055      0.189927      0.287424      0.225503      0.0714717       11     
reco:ophitspe:OpHitFinderDeco                          0.323453       1.38477       2.1284        1.63389      0.534505        11     
reco:opflash:OpFlashFinder                            0.000365637   0.00160715    0.00480173    0.00145287    0.00111213       11     
reco:rns:RandomNumberSaver                            2.0706e-05    5.07829e-05   0.000295782   2.5231e-05    7.77028e-05      11     
reco:anglereconue:NuAngularReco                       0.000178598   0.00120742     0.0061631    0.000354104   0.00185303       11     
reco:anglereconumu:NuAngularReco                      0.000104735   0.00102769    0.00467485    0.00027221    0.00150379       11     
reco:anglereconuepfps:NuAngularReco                   0.000153754    0.0013236    0.00740013    0.000368501   0.00215698       11     
reco:anglereconumupfps:NuAngularReco                  0.00014755    0.00111114    0.00527265    0.000350696   0.00163925       11     
[art]:TriggerResults:TriggerResultInserter            1.2176e-05    1.81065e-05   7.0804e-05    1.2925e-05    1.66709e-05      11     
end_path:out1:RootOutput                               3.209e-06    6.37573e-06   2.4771e-05     3.511e-06    6.05812e-06      11     
end_path:out1:RootOutput(write)                        0.0297398     0.232981      0.459075      0.276337      0.145818        11     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6419.07 MB
  Peak resident set size usage (VmHWM): 1627.61 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Nov-2024 08:23:46 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module BlurredClustering/blurredcluster run: 50574864 subRun: 1 event: 27012
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 11 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-ff3e-fade-5429-9931.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574864_270_20231203T154035Z_gen_g4_detsim_hitreco__20240508T031850Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574864_270_20231203T154035Z_gen_g4_detsim_hitreco__20240508T031850Z_reco2_reco_2024-11-23T_071754Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work