Jobsub ID 301864.33@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301864.33@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 07:13:11
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_dissel
Last heartbeat	2024-11-23 07:18:03
From worker node	Hostname	wn-choc-036.farm.nikhef.nl
	cpuinfo	Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 07:14:43
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492179_593_20231208T045944Z_gen_g4_detsim_hitreco__20240510T041245Z_reco2.root
Jobscript	Exit code	0
	Real time	3m (185s)
	CPU time	1m (92s = 49%)
Outputting started	2024-11-23 07:17:49
Output files
Finished	2024-11-23 07:18:03
List job events Wrapper job log
Jobscript log (last 10,000 characters)

ma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:17:48 CET run: 6492179 subRun: 1 event: 59302
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:17:48 CET run: 6492179 subRun: 1 event: 59302
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:17:48 CET run: 6492179 subRun: 1 event: 59302
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:17:48 CET run: 6492179 subRun: 1 event: 59302
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:17:48 CET run: 6492179 subRun: 1 event: 59302
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:17:48 CET run: 6492179 subRun: 1 event: 59302
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:17:48 CET run: 6492179 subRun: 1 event: 59302
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:17:48 CET run: 6492179 subRun: 1 event: 59302
Comparing two wires in the same plane: return failure
%MSG
23-Nov-2024 08:17:48 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/e4/f4/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492179_593_20231208T045944Z_gen_g4_detsim_hitreco__20240510T041245Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.0243598      3.05803       6.0917        3.05803       3.03367         2     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0155769     0.0199683     0.0243598     0.0199683    0.00439145        2     
reco:linecluster:LineCluster                           0.164157      0.193013      0.221869      0.193013      0.0288563        2     
reco:trajcluster:TrajCluster                           0.0558043     0.264184      0.472564      0.264184       0.20838         2     
reco:pandora:StandardPandora                            2.77899       2.95674       3.13449       2.95674      0.177749         2     
reco:pandoraTrack:LArPandoraTrackCreation             0.00212146    0.00757464     0.0130278    0.00757464    0.00545318        2     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00299305     0.013565      0.0241369     0.013565      0.0105719        2     
reco:pandoracalo:Calorimetry                          0.00137721     0.0134348     0.0254925     0.0134348     0.0120576        2     
reco:pandorapid:Chi2ParticleID                        0.000734792   0.00105293    0.00137108    0.00105293    0.000318142       2     
reco:trkshowersplit:TrackShowerHits                   0.00224641     0.0115862     0.020926      0.0115862    0.00933978        2     
reco:pmtrack:PMAlgTrackMaker                          0.00567301      1.08581       2.16596       1.08581       1.08014         2     
reco:pmtrackcalo:Calorimetry                          0.00013382     0.0035409    0.00694799     0.0035409    0.00340708        2     
reco:pmtrackpid:Chi2ParticleID                        4.9572e-05    0.000256997   0.000464423   0.000256997   0.000207425       2     
reco:pmtrajfit:PMAlgTrajFitter                         0.0016576     0.723207       1.44476      0.723207      0.721549         2     
reco:pmtrajfitcalo:Calorimetry                        0.00011681     0.0121449     0.0241731     0.0121449     0.0120281        2     
reco:pmtrajfitpid:Chi2ParticleID                      4.5171e-05    0.000170594   0.000296018   0.000170594   0.000125424       2     
reco:pmtracktc:PMAlgTrackMaker                        0.00151269      1.23788       2.47425       1.23788       1.23637         2     
reco:pmtracktccalo:Calorimetry                        0.000100026   0.00200788    0.00391572    0.00200788    0.00190785        2     
reco:pmtracktcpid:Chi2ParticleID                       4.282e-05    0.000203827   0.000364835   0.000203827   0.000161007       2     
reco:pmtrajfittc:PMAlgTrajFitter                      0.00031099     0.875831       1.75135      0.875831       0.87552         2     
reco:pmtrajfittccalo:Calorimetry                      9.2018e-05    0.00684373     0.0135954    0.00684373    0.00675171        2     
reco:pmtrajfittcpid:Chi2ParticleID                     4.016e-05    0.000359302   0.000678444   0.000359302   0.000319142       2     
reco:blurredcluster:BlurredClustering                   1.02228       1.02228       1.02228       1.02228          0            1     
reco:emtrkmichelid:EmTrackMichelId                     0.434414      0.434414      0.434414      0.434414          0            1     
reco:emshower:EMShower                                0.00340874    0.00340874    0.00340874    0.00340874         0            1     
reco:cvnmap:CVNMapper                                 0.000246379   0.000246379   0.000246379   0.000246379        0            1     
reco:cvneva:CVNEvaluator                              0.000238311   0.000238311   0.000238311   0.000238311        0            1     
reco:energyrecnumu:EnergyReco                         0.00530782    0.00530782    0.00530782    0.00530782         0            1     
reco:energyrecnue:EnergyReco                          0.00271777    0.00271777    0.00271777    0.00271777         0            1     
reco:energyrecnc:EnergyReco                           0.00272178    0.00272178    0.00272178    0.00272178         0            1     
reco:energyrecnumurange:EnergyReco                    0.00273395    0.00273395    0.00273395    0.00273395         0            1     
reco:energyrecnumumcs:EnergyReco                      0.00256269    0.00256269    0.00256269    0.00256269         0            1     
reco:opdec:Deconvolution                               0.212379      0.212379      0.212379      0.212379          0            1     
reco:ophitspe:OpHitFinderDeco                          0.937366      0.937366      0.937366      0.937366          0            1     
reco:opflash:OpFlashFinder                             0.0038813     0.0038813     0.0038813     0.0038813         0            1     
reco:rns:RandomNumberSaver                            0.000308137   0.000308137   0.000308137   0.000308137        0            1     
reco:anglereconue:NuAngularReco                       0.00111278    0.00111278    0.00111278    0.00111278         0            1     
reco:anglereconumu:NuAngularReco                      0.000171103   0.000171103   0.000171103   0.000171103        0            1     
reco:anglereconuepfps:NuAngularReco                   0.00128516    0.00128516    0.00128516    0.00128516         0            1     
reco:anglereconumupfps:NuAngularReco                  0.000200514   0.000200514   0.000200514   0.000200514        0            1     
[art]:TriggerResults:TriggerResultInserter            4.8649e-05    4.8649e-05    4.8649e-05    4.8649e-05         0            1     
end_path:out1:RootOutput                              1.9166e-05    1.9166e-05    1.9166e-05    1.9166e-05         0            1     
end_path:out1:RootOutput(write)                        0.0669304     0.0669304     0.0669304     0.0669304         0            1     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6421.28 MB
  Peak resident set size usage (VmHWM): 1635.64 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Nov-2024 08:17:48 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module BlurredClustering/blurredcluster run: 6492179 subRun: 1 event: 59302
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 1 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-bc71-2f77-b0df-74eb.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492179_593_20231208T045944Z_gen_g4_detsim_hitreco__20240510T041245Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492179_593_20231208T045944Z_gen_g4_detsim_hitreco__20240510T041245Z_reco2_reco_2024-11-23T_071447Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work