Jobsub ID 301868.46@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301868.46@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 07:20:29
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_juk
Last heartbeat	2024-11-23 07:30:13
From worker node	Hostname	wn-choc-036.farm.nikhef.nl
	cpuinfo	Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 07:21:47
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_1197_20231203T073525Z_gen_g4_detsim_hitreco__20240508T073140Z_reco2.root
Jobscript	Exit code	0
	Real time	8m (493s)
	CPU time	5m (353s = 71%)
Outputting started	2024-11-23 07:30:00
Output files
Finished	2024-11-23 07:30:13
List job events Wrapper job log
Jobscript log (last 10,000 characters)

CET run: 50572032 subRun: 1 event: 119702
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:29:59 CET run: 50572032 subRun: 1 event: 119702
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:29:59 CET run: 50572032 subRun: 1 event: 119702
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:29:59 CET run: 50572032 subRun: 1 event: 119702
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:29:59 CET run: 50572032 subRun: 1 event: 119702
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:29:59 CET run: 50572032 subRun: 1 event: 119702
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:29:59 CET run: 50572032 subRun: 1 event: 119702
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:29:59 CET run: 50572032 subRun: 1 event: 119702
Comparing two wires in the same plane: return failure
%MSG
23-Nov-2024 08:29:59 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/ba/e2/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_1197_20231203T073525Z_gen_g4_detsim_hitreco__20240508T073140Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.0278443      12.1714       24.3149       12.1714       12.1435         2     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0154781     0.0216612     0.0278443     0.0216612    0.00618308        2     
reco:linecluster:LineCluster                           0.190928       11.4918       22.7927       11.4918       11.3009         2     
reco:trajcluster:TrajCluster                           0.198506       23.4366       46.6747       23.4366       23.2381         2     
reco:pandora:StandardPandora                            3.3615        9.89871       16.4359       9.89871       6.53721         2     
reco:pandoraTrack:LArPandoraTrackCreation             0.00978903     0.0260176     0.0422462     0.0260176     0.0162286        2     
reco:pandoraShower:LArPandoraModularShowerCreation     0.019429      0.0275216     0.0356143     0.0275216    0.00809265        2     
reco:pandoracalo:Calorimetry                          0.00613071     0.280714      0.555297      0.280714      0.274583         2     
reco:pandorapid:Chi2ParticleID                        0.000269872   0.000879982   0.00149009    0.000879982   0.00061011        2     
reco:trkshowersplit:TrackShowerHits                   0.00627627     0.293114      0.579951      0.293114      0.286837         2     
reco:pmtrack:PMAlgTrackMaker                            1.62103       24.7919       47.9628       24.7919       23.1709         2     
reco:pmtrackcalo:Calorimetry                          0.00424419     0.0439508     0.0836575     0.0439508     0.0397066        2     
reco:pmtrackpid:Chi2ParticleID                        0.000371446   0.00115142     0.0019314    0.00115142    0.000779979       2     
reco:pmtrajfit:PMAlgTrajFitter                         0.747557       19.3958       38.044        19.3958       18.6482         2     
reco:pmtrajfitcalo:Calorimetry                         0.0066581     0.276852      0.547046      0.276852      0.270194         2     
reco:pmtrajfitpid:Chi2ParticleID                      0.000329401   0.000371111   0.000412822   0.000371111   4.17105e-05       2     
reco:pmtracktc:PMAlgTrackMaker                          1.26765       36.537        71.8064       36.537        35.2694         2     
reco:pmtracktccalo:Calorimetry                        0.00656389     0.024187      0.0418101     0.024187      0.0176231        2     
reco:pmtracktcpid:Chi2ParticleID                      0.000465333   0.000887033   0.00130873    0.000887033    0.0004217        2     
reco:pmtrajfittc:PMAlgTrajFitter                        1.30624       8.45647       15.6067       8.45647       7.15023         2     
reco:pmtrajfittccalo:Calorimetry                      0.00684073     0.121233      0.235626      0.121233      0.114393         2     
reco:pmtrajfittcpid:Chi2ParticleID                    0.00045716    0.00152718    0.00259719    0.00152718    0.00107002        2     
reco:blurredcluster:BlurredClustering                  0.997802      0.997802      0.997802      0.997802          0            1     
reco:emtrkmichelid:EmTrackMichelId                      10.668        10.668        10.668        10.668           0            1     
reco:emshower:EMShower                                 0.193196      0.193196      0.193196      0.193196          0            1     
reco:cvnmap:CVNMapper                                  0.0259384     0.0259384     0.0259384     0.0259384         0            1     
reco:cvneva:CVNEvaluator                                1.18236       1.18236       1.18236       1.18236          0            1     
reco:energyrecnumu:EnergyReco                          0.0177944     0.0177944     0.0177944     0.0177944         0            1     
reco:energyrecnue:EnergyReco                          0.00199349    0.00199349    0.00199349    0.00199349         0            1     
reco:energyrecnc:EnergyReco                            0.0046753     0.0046753     0.0046753     0.0046753         0            1     
reco:energyrecnumurange:EnergyReco                     0.012598      0.012598      0.012598      0.012598          0            1     
reco:energyrecnumumcs:EnergyReco                       0.0156874     0.0156874     0.0156874     0.0156874         0            1     
reco:opdec:Deconvolution                               0.320182      0.320182      0.320182      0.320182          0            1     
reco:ophitspe:OpHitFinderDeco                           1.74595       1.74595       1.74595       1.74595          0            1     
reco:opflash:OpFlashFinder                            0.00392142    0.00392142    0.00392142    0.00392142         0            1     
reco:rns:RandomNumberSaver                            0.000253338   0.000253338   0.000253338   0.000253338        0            1     
reco:anglereconue:NuAngularReco                       0.00217254    0.00217254    0.00217254    0.00217254         0            1     
reco:anglereconumu:NuAngularReco                      0.00136978    0.00136978    0.00136978    0.00136978         0            1     
reco:anglereconuepfps:NuAngularReco                   0.00339062    0.00339062    0.00339062    0.00339062         0            1     
reco:anglereconumupfps:NuAngularReco                  0.00144528    0.00144528    0.00144528    0.00144528         0            1     
[art]:TriggerResults:TriggerResultInserter            4.4369e-05    4.4369e-05    4.4369e-05    4.4369e-05         0            1     
end_path:out1:RootOutput                              1.6136e-05    1.6136e-05    1.6136e-05    1.6136e-05         0            1     
end_path:out1:RootOutput(write)                        0.335989      0.335989      0.335989      0.335989          0            1     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6425.3 MB
  Peak resident set size usage (VmHWM): 1677.03 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Nov-2024 08:29:59 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module BlurredClustering/blurredcluster run: 50572032 subRun: 1 event: 119702
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 1 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-9975-0f55-616f-2548.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_1197_20231203T073525Z_gen_g4_detsim_hitreco__20240508T073140Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_1197_20231203T073525Z_gen_g4_detsim_hitreco__20240508T073140Z_reco2_reco_2024-11-23T_072150Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work