Jobsub ID 301865.3@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301865.3@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 07:17:21
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_dissel
Last heartbeat	2024-11-23 07:29:20
From worker node	Hostname	wn-choc-036.farm.nikhef.nl
	cpuinfo	Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 07:18:36
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1497_126_20231027T144150Z_gen_g4_detsim_hitreco__20240503T035450Z_reco2.root
Jobscript	Exit code	0
	Real time	10m (630s)
	CPU time	8m (521s = 82%)
Outputting started	2024-11-23 07:29:07
Output files
Finished	2024-11-23 07:29:20
List job events Wrapper job log
Jobscript log (last 10,000 characters)

97 subRun: 1 event: 25204
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:29:05 CET run: 1497 subRun: 1 event: 25204
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:29:05 CET run: 1497 subRun: 1 event: 25204
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:29:05 CET run: 1497 subRun: 1 event: 25204
Cannot attach back to inner node of other track.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:29:05 CET run: 1497 subRun: 1 event: 25204
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:29:05 CET run: 1497 subRun: 1 event: 25204
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:29:05 CET run: 1497 subRun: 1 event: 25204
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:29:05 CET run: 1497 subRun: 1 event: 25204
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:29:05 CET run: 1497 subRun: 1 event: 25204
Comparing two wires in the same plane: return failure
%MSG
23-Nov-2024 08:29:06 CET  Closed input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/3e/f5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1497_126_20231027T144150Z_gen_g4_detsim_hitreco__20240503T035450Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.0119715      7.81157       21.5884       4.82294       8.19315         4     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00881883     0.0158293     0.0323036     0.0110973    0.00957679        4     
reco:linecluster:LineCluster                           0.115495       6.38011       25.0934      0.155786       10.8041         4     
reco:trajcluster:TrajCluster                           0.0244659      19.7625       78.9114      0.0569894      34.1497         4     
reco:pandora:StandardPandora                            1.54179       8.68532       27.6588       2.77036       10.9727         4     
reco:pandoraTrack:LArPandoraTrackCreation             0.000177443    0.0180049     0.062417     0.00471251     0.0258915        4     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000736394    0.0207391     0.0525495     0.0148353     0.021608         4     
reco:pandoracalo:Calorimetry                          0.000134933    0.259951       1.03273     0.00346882     0.446172         4     
reco:pandorapid:Chi2ParticleID                        6.0724e-05    0.00060139    0.00186614    0.000239351   0.000742678       4     
reco:trkshowersplit:TrackShowerHits                   0.000183505    0.222767      0.885548     0.00266904     0.382662         4     
reco:pmtrack:PMAlgTrackMaker                          0.00202082      17.6212       70.1031      0.189845       30.3008         4     
reco:pmtrackcalo:Calorimetry                          0.000298254    0.0511455      0.20148      0.001402      0.0867995        4     
reco:pmtrackpid:Chi2ParticleID                        8.0995e-05    0.000740679   0.00256072    0.000160503   0.00105186        4     
reco:pmtrajfit:PMAlgTrajFitter                         0.0149683      18.0565       71.2911      0.459883       30.7366         4     
reco:pmtrajfitcalo:Calorimetry                        0.000267804     0.25853       1.03024     0.00180552     0.445549         4     
reco:pmtrajfitpid:Chi2ParticleID                      7.8492e-05    0.000218599   0.000492446   0.000151729   0.000165476       4     
reco:pmtracktc:PMAlgTrackMaker                        0.00276185      29.7857       118.61       0.265183       51.2829         4     
reco:pmtracktccalo:Calorimetry                        0.000264314    0.0158926     0.0596844    0.00181076     0.0253102        4     
reco:pmtracktcpid:Chi2ParticleID                      7.7275e-05    0.000506964   0.00156344    0.000193569   0.000615341       4     
reco:pmtrajfittc:PMAlgTrajFitter                       0.0570992      4.98896       19.2652      0.316764       8.24372         4     
reco:pmtrajfittccalo:Calorimetry                      0.000318996    0.108137      0.428897     0.00166709     0.185193         4     
reco:pmtrajfittcpid:Chi2ParticleID                    8.8836e-05     0.0008853     0.0030893    0.000181532   0.00127426        4     
reco:blurredcluster:BlurredClustering                   0.19752       1.25224       2.79677      0.762435       1.11623         3     
reco:emtrkmichelid:EmTrackMichelId                     0.664983       3.41803       8.76717      0.821953       3.78295         3     
reco:emshower:EMShower                                0.00166205     0.0395801     0.108538     0.00854038     0.0488413        3     
reco:cvnmap:CVNMapper                                 3.6074e-05     0.0116769     0.0349392    5.5392e-05     0.0164489        3     
reco:cvneva:CVNEvaluator                              2.1441e-05     0.409418       1.22821     2.2866e-05     0.578973         3     
reco:energyrecnumu:EnergyReco                         0.000598686   0.00545556     0.0128098    0.00295825    0.00528867        3     
reco:energyrecnue:EnergyReco                          0.000212358   0.000606469   0.00134799    0.000259063   0.000524679       3     
reco:energyrecnc:EnergyReco                           0.00260921    0.00307734    0.00398275    0.00264006    0.000640344       3     
reco:energyrecnumurange:EnergyReco                    0.00048754    0.00188442    0.00269738    0.00246836    0.000992161       3     
reco:energyrecnumumcs:EnergyReco                       0.0025718    0.00271739    0.00295514    0.00262522    0.000169524       3     
reco:opdec:Deconvolution                               0.187641      0.242376      0.330783      0.208706      0.0631015        3     
reco:ophitspe:OpHitFinderDeco                          0.857742       1.41609       2.16869       1.22183      0.552539         3     
reco:opflash:OpFlashFinder                            0.00085535    0.00245951    0.00554645    0.000976738   0.00218336        3     
reco:rns:RandomNumberSaver                            2.1193e-05    0.000124734   0.000324866   2.8143e-05    0.000141543       3     
reco:anglereconue:NuAngularReco                       0.000211537   0.000820935   0.00201063    0.000240643   0.000841322       3     
reco:anglereconumu:NuAngularReco                      0.000124252   0.00034579    0.000763598   0.000149521   0.000295615       3     
reco:anglereconuepfps:NuAngularReco                   0.000171133   0.00127805    0.00344366    0.000219368   0.00153144        3     
reco:anglereconumupfps:NuAngularReco                  0.000176059   0.000432406   0.000924234   0.000196926   0.000347879       3     
[art]:TriggerResults:TriggerResultInserter            1.2573e-05    2.7594e-05    5.7331e-05    1.2878e-05    2.10276e-05       3     
end_path:out1:RootOutput                               3.344e-06     8.315e-06     1.819e-05     3.411e-06    6.98273e-06       3     
end_path:out1:RootOutput(write)                        0.0588884     0.164046      0.357818      0.0754304     0.137184         3     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6425.3 MB
  Peak resident set size usage (VmHWM): 1676.1 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Nov-2024 08:29:06 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module BlurredClustering/blurredcluster run: 1497 subRun: 1 event: 25204
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 3 entries while sim::OpDetDivRecs_opdigi__detsim. has 200 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-9dd5-11e8-5ebe-97de.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1497_126_20231027T144150Z_gen_g4_detsim_hitreco__20240503T035450Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1497_126_20231027T144150Z_gen_g4_detsim_hitreco__20240503T035450Z_reco2_reco_2024-11-23T_071840Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work