Jobsub ID 301870.0@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301870.0@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 07:27:52
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_brug
Last heartbeat	2024-11-23 07:35:32
From worker node	Hostname	wn-choc-026.farm.nikhef.nl
	cpuinfo	Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 07:30:02
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50640579_252_20231207T162353Z_gen_g4_detsim_hitreco__20240510T051426Z_reco2.root
Jobscript	Exit code	0
	Real time	5m (316s)
	CPU time	3m (216s = 68%)
Outputting started	2024-11-23 07:35:19
Output files
Finished	2024-11-23 07:35:32
List job events Wrapper job log
Jobscript log (last 10,000 characters)

gTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:35:17 CET run: 50640579 subRun: 1 event: 25210
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:35:17 CET run: 50640579 subRun: 1 event: 25210
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:35:17 CET run: 50640579 subRun: 1 event: 25210
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:35:18 CET run: 50640579 subRun: 1 event: 25210
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:35:18 CET run: 50640579 subRun: 1 event: 25210
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:35:18 CET run: 50640579 subRun: 1 event: 25210
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:35:18 CET run: 50640579 subRun: 1 event: 25210
Comparing two wires in the same plane: return failure
%MSG
23-Nov-2024 08:35:18 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/f3/98/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50640579_252_20231207T162353Z_gen_g4_detsim_hitreco__20240510T051426Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.0297347      8.79401       32.1313       5.36748       9.37338        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0158513     0.033644      0.0511017     0.0338049     0.0114823       10     
reco:linecluster:LineCluster                           0.138389      0.257992        1.205       0.142592      0.316339        10     
reco:trajcluster:TrajCluster                           0.043829      0.496889       4.35062      0.0480499      1.28542        10     
reco:pandora:StandardPandora                           0.0588552      2.02114       5.52455       1.54233       1.40374        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000164702   0.00600277     0.0383662    0.00077466     0.0112782       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000222207    0.0149466     0.083937     0.00236234     0.0251387       10     
reco:pandoracalo:Calorimetry                          0.000127353    0.0186104     0.161978     0.000582667    0.047974        10     
reco:pandorapid:Chi2ParticleID                         5.341e-05    0.000308451   0.00184522    0.000120012   0.000519288      10     
reco:trkshowersplit:TrackShowerHits                   0.000120182    0.0145411      0.13384     0.000292411    0.0398124       10     
reco:pmtrack:PMAlgTrackMaker                          0.00144432      1.96111       14.4768      0.0681479      4.3006         10     
reco:pmtrackcalo:Calorimetry                          0.000100962   0.00628284     0.0457285    0.000721234    0.0135618       10     
reco:pmtrackpid:Chi2ParticleID                         4.323e-05    0.000239153   0.00130572    0.000126516   0.000361151      10     
reco:pmtrajfit:PMAlgTrajFitter                        0.000258961     1.25074       9.86399      0.152629        2.892         10     
reco:pmtrajfitcalo:Calorimetry                        9.5817e-05     0.0189784     0.169478     0.00059034     0.0503108       10     
reco:pmtrajfitpid:Chi2ParticleID                      4.2669e-05    0.000156305   0.000533808   0.00011586    0.00013563       10     
reco:pmtracktc:PMAlgTrackMaker                        0.00128721      2.04089       17.3305      0.132025       5.13391        10     
reco:pmtracktccalo:Calorimetry                        0.000100511   0.00404916     0.0217688    0.000822928   0.00678032       10     
reco:pmtracktcpid:Chi2ParticleID                      4.3483e-05    0.000219944   0.000966466   0.000146057   0.00025765       10     
reco:pmtrajfittc:PMAlgTrajFitter                      0.000227304    0.962955       6.09051      0.321684       1.77824        10     
reco:pmtrajfittccalo:Calorimetry                      9.6607e-05     0.0112416     0.0966813    0.000873141    0.0285978       10     
reco:pmtrajfittcpid:Chi2ParticleID                    4.2602e-05    0.000315049    0.001908     0.000153426   0.000535498      10     
reco:blurredcluster:BlurredClustering                  8.63e-05      0.288525       1.34925      0.134896      0.400638         9     
reco:emtrkmichelid:EmTrackMichelId                     0.0480754      3.5586        16.7205       1.41123       5.11508         9     
reco:emshower:EMShower                                0.000295609   0.00741255     0.0317514    0.000575208    0.0114371        9     
reco:cvnmap:CVNMapper                                 3.3732e-05    0.00964232     0.0330232     0.0106457     0.0106815        9     
reco:cvneva:CVNEvaluator                              2.3489e-05     0.693047       1.26041       1.21748      0.619968         9     
reco:energyrecnumu:EnergyReco                         0.000435115    0.012291      0.0452785    0.00315085     0.0173381        9     
reco:energyrecnue:EnergyReco                          0.000206863   0.00157394    0.00658751    0.000373843   0.00197586        9     
reco:energyrecnc:EnergyReco                           0.00260642    0.00306085    0.00484035    0.00270449    0.000694644       9     
reco:energyrecnumurange:EnergyReco                    0.000335458    0.0102667     0.0416467    0.00259522     0.0148012        9     
reco:energyrecnumumcs:EnergyReco                      0.00269503     0.0117454     0.0421208    0.00376169     0.0142631        9     
reco:opdec:Deconvolution                               0.0875309     0.242198      0.413516      0.207332      0.105898         9     
reco:ophitspe:OpHitFinderDeco                          0.359164       1.22416       2.25213       1.04017      0.626636         9     
reco:opflash:OpFlashFinder                            0.000292163   0.00156705    0.00625233    0.000941507   0.00172662        9     
reco:rns:RandomNumberSaver                            2.3013e-05    9.19364e-05   0.000605534   2.6209e-05    0.000181631       9     
reco:anglereconue:NuAngularReco                       0.000170831   0.000680003   0.00291262    0.000250039   0.000897115       9     
reco:anglereconumu:NuAngularReco                      0.00010621    0.000346662   0.00148531    0.000158877   0.000431781       9     
reco:anglereconuepfps:NuAngularReco                   0.000159734   0.000734144   0.00336723    0.00025955    0.00102656        9     
reco:anglereconumupfps:NuAngularReco                  0.000148721   0.000425216   0.00161067    0.000250229   0.000447878       9     
[art]:TriggerResults:TriggerResultInserter            1.3837e-05    1.98583e-05   6.4097e-05    1.4276e-05    1.56452e-05       9     
end_path:out1:RootOutput                               3.487e-06    6.01333e-06   1.8855e-05     3.611e-06    4.73891e-06       9     
end_path:out1:RootOutput(write)                        0.0240378     0.183774      0.365861       0.22311      0.130911         9     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6425.37 MB
  Peak resident set size usage (VmHWM): 1677.9 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Nov-2024 08:35:18 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module BlurredClustering/blurredcluster run: 50640579 subRun: 1 event: 25210
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 9 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-688b-41c4-2974-1684.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50640579_252_20231207T162353Z_gen_g4_detsim_hitreco__20240510T051426Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50640579_252_20231207T162353Z_gen_g4_detsim_hitreco__20240510T051426Z_reco2_reco_2024-11-23T_073005Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work