Jobsub ID 108878.30@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108878.30@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 07:16:18
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_brug
Last heartbeat	2024-11-23 07:21:36
From worker node	Hostname	wn-choc-032.farm.nikhef.nl
	cpuinfo	Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 07:17:17
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6210996_415_20231121T150250Z_gen_g4_detsim_hitreco__20240507T181250Z_reco2.root
Jobscript	Exit code	0
	Real time	4m (245s)
	CPU time	3m (237s = 96%)
Outputting started	2024-11-23 07:21:23
Output files
Finished	2024-11-23 07:21:36
List job events Wrapper job log
Jobscript log (last 10,000 characters)

Check:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:21:21 CET run: 6210996 subRun: 1 event: 83002
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:21:21 CET run: 6210996 subRun: 1 event: 83002
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:21:21 CET run: 6210996 subRun: 1 event: 83002
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:21:21 CET run: 6210996 subRun: 1 event: 83002
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:21:21 CET run: 6210996 subRun: 1 event: 83002
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:21:21 CET run: 6210996 subRun: 1 event: 83002
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:21:21 CET run: 6210996 subRun: 1 event: 83002
Comparing two wires in the same plane: return failure
%MSG
23-Nov-2024 08:21:22 CET  Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/ab/61/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6210996_415_20231121T150250Z_gen_g4_detsim_hitreco__20240507T181250Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                            0.000948188     2.22189       4.44283       2.22189       2.22094         2     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000948188   0.00116445    0.00138071    0.00116445    0.000216259       2     
reco:linecluster:LineCluster                           0.0372918      1.6697        3.30211       1.6697        1.63241         2     
reco:trajcluster:TrajCluster                          0.00847467      8.17624       16.344        8.17624       8.16776         2     
reco:pandora:StandardPandora                            3.11714       9.29722       15.4773       9.29722       6.18008         2     
reco:pandoraTrack:LArPandoraTrackCreation             0.00194732     0.0344418     0.0669362     0.0344418     0.0324944        2     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00287406     0.0306719     0.0584698     0.0306719     0.0277979        2     
reco:pandoracalo:Calorimetry                          0.00132175     0.204086       0.40685      0.204086      0.202764         2     
reco:pandorapid:Chi2ParticleID                        0.000698206   0.00122282    0.00174743    0.00122282    0.000524612       2     
reco:trkshowersplit:TrackShowerHits                   0.00214122     0.119605      0.237069      0.119605      0.117464         2     
reco:pmtrack:PMAlgTrackMaker                          0.00509936      15.0633       30.1215       15.0633       15.0582         2     
reco:pmtrackcalo:Calorimetry                          0.000118839    0.103008      0.205898      0.103008      0.102889         2     
reco:pmtrackpid:Chi2ParticleID                        4.5674e-05    0.00124856    0.00245144    0.00124856    0.00120288        2     
reco:pmtrajfit:PMAlgTrajFitter                        0.00152328      15.0583       30.115        15.0583       15.0567         2     
reco:pmtrajfitcalo:Calorimetry                        9.6761e-05     0.211233       0.42237      0.211233      0.211136         2     
reco:pmtrajfitpid:Chi2ParticleID                      4.1259e-05    0.000452615   0.00086397    0.000452615   0.000411355       2     
reco:pmtracktc:PMAlgTrackMaker                        0.00140211      22.4879       44.9745       22.4879       22.4865         2     
reco:pmtracktccalo:Calorimetry                        9.4493e-05     0.0531892     0.106284      0.0531892     0.0530947        2     
reco:pmtracktcpid:Chi2ParticleID                      4.1341e-05    0.000946699   0.00185206    0.000946699   0.000905358       2     
reco:pmtrajfittc:PMAlgTrajFitter                      0.000237301     8.96068       17.9211       8.96068       8.96044         2     
reco:pmtrajfittccalo:Calorimetry                      9.0663e-05     0.134555       0.26902      0.134555      0.134465         2     
reco:pmtrajfittcpid:Chi2ParticleID                    4.0417e-05    0.00145994    0.00287947    0.00145994    0.00141953        2     
reco:blurredcluster:BlurredClustering                   0.35295       0.35295       0.35295       0.35295          0            1     
reco:emtrkmichelid:EmTrackMichelId                     0.424094      0.424094      0.424094      0.424094          0            1     
reco:emshower:EMShower                                0.00347425    0.00347425    0.00347425    0.00347425         0            1     
reco:cvnmap:CVNMapper                                 0.000244195   0.000244195   0.000244195   0.000244195        0            1     
reco:cvneva:CVNEvaluator                              0.000254261   0.000254261   0.000254261   0.000254261        0            1     
reco:energyrecnumu:EnergyReco                         0.00527575    0.00527575    0.00527575    0.00527575         0            1     
reco:energyrecnue:EnergyReco                          0.00269371    0.00269371    0.00269371    0.00269371         0            1     
reco:energyrecnc:EnergyReco                           0.00267564    0.00267564    0.00267564    0.00267564         0            1     
reco:energyrecnumurange:EnergyReco                    0.00268993    0.00268993    0.00268993    0.00268993         0            1     
reco:energyrecnumumcs:EnergyReco                      0.00269238    0.00269238    0.00269238    0.00269238         0            1     
reco:opdec:Deconvolution                               0.0538269     0.0538269     0.0538269     0.0538269         0            1     
reco:ophitspe:OpHitFinderDeco                          0.366322      0.366322      0.366322      0.366322          0            1     
reco:opflash:OpFlashFinder                            0.00395869    0.00395869    0.00395869    0.00395869         0            1     
reco:rns:RandomNumberSaver                            0.000336866   0.000336866   0.000336866   0.000336866        0            1     
reco:anglereconue:NuAngularReco                       0.000946576   0.000946576   0.000946576   0.000946576        0            1     
reco:anglereconumu:NuAngularReco                      0.000169629   0.000169629   0.000169629   0.000169629        0            1     
reco:anglereconuepfps:NuAngularReco                   0.00143885    0.00143885    0.00143885    0.00143885         0            1     
reco:anglereconumupfps:NuAngularReco                  0.00020436    0.00020436    0.00020436    0.00020436         0            1     
[art]:TriggerResults:TriggerResultInserter             5.888e-05     5.888e-05     5.888e-05     5.888e-05         0            1     
end_path:out1:RootOutput                              1.9373e-05    1.9373e-05    1.9373e-05    1.9373e-05         0            1     
end_path:out1:RootOutput(write)                        0.0345944     0.0345944     0.0345944     0.0345944         0            1     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6424.85 MB
  Peak resident set size usage (VmHWM): 1672.7 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Nov-2024 08:21:22 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module BlurredClustering/blurredcluster run: 6210996 subRun: 1 event: 83002
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 1 entries while sim::OpDetDivRecs_opdigi__detsim. has 200 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-0edf-d811-f0da-d71e.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6210996_415_20231121T150250Z_gen_g4_detsim_hitreco__20240507T181250Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6210996_415_20231121T150250Z_gen_g4_detsim_hitreco__20240507T181250Z_reco2_reco_2024-11-23T_071720Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work