Jobsub ID 108876.0@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108876.0@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 07:14:13
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_klomp
Last heartbeat	2024-11-23 07:17:49
From worker node	Hostname	wn-choc-039.farm.nikhef.nl
	cpuinfo	Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 07:14:37
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50557588_898_20231202T110348Z_gen_g4_detsim_hitreco__20240507T200749Z_reco2.root
Jobscript	Exit code	0
	Real time	2m (178s)
	CPU time	1m (85s = 47%)
Outputting started	2024-11-23 07:17:35
Output files
Finished	2024-11-23 07:17:49
List job events Wrapper job log
Jobscript log (last 10,000 characters)

mtrajfittc@BeginModule  23-Nov-2024 08:17:33 CET run: 50557588 subRun: 1 event: 89802
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:17:33 CET run: 50557588 subRun: 1 event: 89802
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:17:33 CET run: 50557588 subRun: 1 event: 89802
1st wire C:0 T:21 P:2 W:821 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:17:33 CET run: 50557588 subRun: 1 event: 89802
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:17:33 CET run: 50557588 subRun: 1 event: 89802
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:17:33 CET run: 50557588 subRun: 1 event: 89802
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:17:33 CET run: 50557588 subRun: 1 event: 89802
Comparing two wires in the same plane: return failure
%MSG
23-Nov-2024 08:17:34 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/15/01/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50557588_898_20231202T110348Z_gen_g4_detsim_hitreco__20240507T200749Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.0270507      5.14369       10.2603       5.14369       5.11664         2     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0154662     0.0212584     0.0270507     0.0212584    0.00579226        2     
reco:linecluster:LineCluster                            0.1497       0.154355       0.15901      0.154355     0.00465505        2     
reco:trajcluster:TrajCluster                           0.0592541     0.0622155     0.0651769     0.0622155    0.00296145        2     
reco:pandora:StandardPandora                            2.27708       2.74274       3.2084        2.74274       0.46566         2     
reco:pandoraTrack:LArPandoraTrackCreation             0.00296635    0.00437111    0.00577586    0.00437111    0.00140475        2     
reco:pandoraShower:LArPandoraModularShowerCreation     0.0100255     0.0123393     0.0146531     0.0123393    0.00231379        2     
reco:pandoracalo:Calorimetry                          0.00221624    0.00320908    0.00420192    0.00320908    0.000992843       2     
reco:pandorapid:Chi2ParticleID                        0.000212134   0.000830342   0.00144855    0.000830342   0.000618208       2     
reco:trkshowersplit:TrackShowerHits                   0.00112758    0.00172865    0.00232972    0.00172865    0.000601068       2     
reco:pmtrack:PMAlgTrackMaker                           0.212641      0.243378      0.274115      0.243378      0.030737         2     
reco:pmtrackcalo:Calorimetry                          0.000860698    0.0014121    0.00196351     0.0014121    0.000551405       2     
reco:pmtrackpid:Chi2ParticleID                        0.000136193   0.000169132   0.000202071   0.000169132   3.2939e-05        2     
reco:pmtrajfit:PMAlgTrajFitter                         0.346977       0.35171      0.356443       0.35171     0.00473275        2     
reco:pmtrajfitcalo:Calorimetry                        0.00111911    0.00154997    0.00198083    0.00154997    0.000430859       2     
reco:pmtrajfitpid:Chi2ParticleID                      0.000120707   0.000162645   0.000204583   0.000162645   4.1938e-05        2     
reco:pmtracktc:PMAlgTrackMaker                         0.191669      0.276399      0.361129      0.276399       0.08473         2     
reco:pmtracktccalo:Calorimetry                        0.000762505   0.00140844    0.00205438    0.00140844    0.000645939       2     
reco:pmtracktcpid:Chi2ParticleID                      0.000120154   0.000169692   0.000219229   0.000169692   4.95375e-05       2     
reco:pmtrajfittc:PMAlgTrajFitter                       0.227067      0.320716      0.414364      0.320716      0.0936481        2     
reco:pmtrajfittccalo:Calorimetry                      0.000751254   0.00146236    0.00217347    0.00146236    0.000711108       2     
reco:pmtrajfittcpid:Chi2ParticleID                    0.000119803   0.000178485   0.000237166   0.000178485   5.86815e-05       2     
reco:blurredcluster:BlurredClustering                  0.395266      0.395266      0.395266      0.395266          0            1     
reco:emtrkmichelid:EmTrackMichelId                      2.29695       2.29695       2.29695       2.29695          0            1     
reco:emshower:EMShower                                 0.0047144     0.0047144     0.0047144     0.0047144         0            1     
reco:cvnmap:CVNMapper                                  0.0140371     0.0140371     0.0140371     0.0140371         0            1     
reco:cvneva:CVNEvaluator                                1.1939        1.1939        1.1939        1.1939           0            1     
reco:energyrecnumu:EnergyReco                          0.0108667     0.0108667     0.0108667     0.0108667         0            1     
reco:energyrecnue:EnergyReco                          0.000634878   0.000634878   0.000634878   0.000634878        0            1     
reco:energyrecnc:EnergyReco                           0.00324738    0.00324738    0.00324738    0.00324738         0            1     
reco:energyrecnumurange:EnergyReco                    0.00538452    0.00538452    0.00538452    0.00538452         0            1     
reco:energyrecnumumcs:EnergyReco                      0.00829636    0.00829636    0.00829636    0.00829636         0            1     
reco:opdec:Deconvolution                               0.259946      0.259946      0.259946      0.259946          0            1     
reco:ophitspe:OpHitFinderDeco                           1.32802       1.32802       1.32802       1.32802          0            1     
reco:opflash:OpFlashFinder                            0.00403503    0.00403503    0.00403503    0.00403503         0            1     
reco:rns:RandomNumberSaver                            0.000287823   0.000287823   0.000287823   0.000287823        0            1     
reco:anglereconue:NuAngularReco                       0.00137246    0.00137246    0.00137246    0.00137246         0            1     
reco:anglereconumu:NuAngularReco                      0.00035278    0.00035278    0.00035278    0.00035278         0            1     
reco:anglereconuepfps:NuAngularReco                   0.00253073    0.00253073    0.00253073    0.00253073         0            1     
reco:anglereconumupfps:NuAngularReco                  0.000443475   0.000443475   0.000443475   0.000443475        0            1     
[art]:TriggerResults:TriggerResultInserter            4.6142e-05    4.6142e-05    4.6142e-05    4.6142e-05         0            1     
end_path:out1:RootOutput                              1.4859e-05    1.4859e-05    1.4859e-05    1.4859e-05         0            1     
end_path:out1:RootOutput(write)                        0.268968      0.268968      0.268968      0.268968          0            1     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6412.29 MB
  Peak resident set size usage (VmHWM): 1663.25 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Nov-2024 08:17:34 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module BlurredClustering/blurredcluster run: 50557588 subRun: 1 event: 89802
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 1 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-1ec9-b479-644d-7524.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50557588_898_20231202T110348Z_gen_g4_detsim_hitreco__20240507T200749Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50557588_898_20231202T110348Z_gen_g4_detsim_hitreco__20240507T200749Z_reco2_reco_2024-11-23T_071440Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work