Jobsub ID 301865.10@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301865.10@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 07:17:21
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_brug
Last heartbeat	2024-11-23 07:28:46
From worker node	Hostname	wn-choc-026.farm.nikhef.nl
	cpuinfo	Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 07:19:14
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50620606_299_20231207T105124Z_gen_g4_detsim_hitreco__20240510T035842Z_reco2.root
Jobscript	Exit code	0
	Real time	9m (557s)
	CPU time	7m (450s = 80%)
Outputting started	2024-11-23 07:28:32
Output files
Finished	2024-11-23 07:28:46
List job events Wrapper job log
Jobscript log (last 10,000 characters)

rajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:28:31 CET run: 50620606 subRun: 1 event: 29914
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:28:31 CET run: 50620606 subRun: 1 event: 29914
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:28:31 CET run: 50620606 subRun: 1 event: 29914
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:28:31 CET run: 50620606 subRun: 1 event: 29914
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:28:31 CET run: 50620606 subRun: 1 event: 29914
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:28:31 CET run: 50620606 subRun: 1 event: 29914
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:28:31 CET run: 50620606 subRun: 1 event: 29914
Comparing two wires in the same plane: return failure
%MSG
23-Nov-2024 08:28:31 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/13/a6/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50620606_299_20231207T105124Z_gen_g4_detsim_hitreco__20240510T035842Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.0334297      15.7659       73.9138       7.31576       19.6165        14     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0155032     0.0370072     0.051546      0.037265      0.0102532       14     
reco:linecluster:LineCluster                           0.139085      0.362467       2.51908      0.165737      0.608826        14     
reco:trajcluster:TrajCluster                           0.0439838      1.4144        16.5878      0.0635019      4.23687        14     
reco:pandora:StandardPandora                            1.42525       5.22543       48.3204       1.62703       11.9692        14     
reco:pandoraTrack:LArPandoraTrackCreation             0.000164381   0.00707123     0.0542353    0.00122957     0.0137014       14     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000240655   0.00807118     0.0298742    0.00314385    0.00954432       14     
reco:pandoracalo:Calorimetry                          0.000134283    0.0455511     0.525463     0.000860464    0.134344        14     
reco:pandorapid:Chi2ParticleID                        5.0522e-05    0.000248875   0.00164958    0.000132924   0.00039816       14     
reco:trkshowersplit:TrackShowerHits                   0.000130044    0.0330587     0.343817      0.0004445     0.0890368       14     
reco:pmtrack:PMAlgTrackMaker                          0.00147346      2.03909       18.6273      0.258037       4.74948        14     
reco:pmtrackcalo:Calorimetry                          0.000102186   0.00931014     0.0848104    0.000648247    0.0214685       14     
reco:pmtrackpid:Chi2ParticleID                        4.2404e-05    0.000337087   0.00170179    0.000134648   0.000446504      14     
reco:pmtrajfit:PMAlgTrajFitter                        0.000265981     3.86711       34.3671      0.297897       9.04668        14     
reco:pmtrajfitcalo:Calorimetry                        9.5273e-05     0.0339286     0.360308     0.00108809     0.0923493       14     
reco:pmtrajfitpid:Chi2ParticleID                      4.0443e-05    0.000189018   0.00061668    0.000151975   0.000149276      14     
reco:pmtracktc:PMAlgTrackMaker                        0.00129657      2.80422       27.8061      0.275726       7.09795        14     
reco:pmtracktccalo:Calorimetry                        9.9366e-05    0.00507997     0.0291408    0.00084627    0.00790859       14     
reco:pmtracktcpid:Chi2ParticleID                       4.185e-05    0.000299087   0.00118784    0.000151223   0.000332684      14     
reco:pmtrajfittc:PMAlgTrajFitter                      0.00024559      1.63689       11.8677      0.253288       3.06571        14     
reco:pmtrajfittccalo:Calorimetry                      9.3181e-05     0.0185815      0.17804     0.000784221    0.045399        14     
reco:pmtrajfittcpid:Chi2ParticleID                    3.9993e-05    0.000465461   0.00245638    0.000146665   0.000664763      14     
reco:blurredcluster:BlurredClustering                 0.00719326     0.551731       1.96777       0.32918       0.61854        13     
reco:emtrkmichelid:EmTrackMichelId                     0.284729       7.30674       35.5912       1.70089       10.4723        13     
reco:emshower:EMShower                                0.00029816     0.0179536     0.0812272    0.000638878    0.0301386       13     
reco:cvnmap:CVNMapper                                 3.1403e-05    0.00895006     0.0212461     0.0107016    0.00774066       13     
reco:cvneva:CVNEvaluator                              2.2116e-05     0.770423       1.45773       1.21159       0.61216        13     
reco:energyrecnumu:EnergyReco                         0.000429747    0.0177331     0.179999     0.00263919     0.0469771       13     
reco:energyrecnue:EnergyReco                          0.000202561   0.00157529    0.00491364    0.00142462    0.00136036       13     
reco:energyrecnc:EnergyReco                           0.00254434    0.00339127    0.00719758    0.00265819    0.00129216       13     
reco:energyrecnumurange:EnergyReco                    0.000296117    0.0160743     0.177491     0.00253455     0.0466467       13     
reco:energyrecnumumcs:EnergyReco                      0.00255767     0.0175554     0.178854     0.00367646     0.0466075       13     
reco:opdec:Deconvolution                               0.124849      0.270973       0.4159       0.250305      0.0931184       13     
reco:ophitspe:OpHitFinderDeco                          0.481405        1.316        2.38541       1.16307      0.592394        13     
reco:opflash:OpFlashFinder                            0.000455213   0.00143978    0.00402823    0.00106224    0.00094263       13     
reco:rns:RandomNumberSaver                            2.2371e-05    5.51534e-05   0.00035871    2.8978e-05    8.7877e-05       13     
reco:anglereconue:NuAngularReco                       0.000165169   0.000732923   0.00283468    0.000233151   0.000800622      13     
reco:anglereconumu:NuAngularReco                      0.00010055    0.00053683    0.00274106    0.000152189   0.000726374      13     
reco:anglereconuepfps:NuAngularReco                   0.000149775   0.000817905   0.00294637    0.000265396   0.000958197      13     
reco:anglereconumupfps:NuAngularReco                  0.000142453   0.000615827   0.00281847    0.00024943    0.000730026      13     
[art]:TriggerResults:TriggerResultInserter            1.3081e-05    1.75901e-05   5.3669e-05    1.4076e-05    1.05119e-05      13     
end_path:out1:RootOutput                               3.274e-06    5.86915e-06   1.8436e-05     3.654e-06    3.96141e-06      13     
end_path:out1:RootOutput(write)                        0.031653      0.213728      0.451146      0.236451      0.139956        13     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6415.54 MB
  Peak resident set size usage (VmHWM): 1650.38 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Nov-2024 08:28:31 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module BlurredClustering/blurredcluster run: 50620606 subRun: 1 event: 29914
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 13 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-fdc6-796e-f9c5-50e8.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50620606_299_20231207T105124Z_gen_g4_detsim_hitreco__20240510T035842Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50620606_299_20231207T105124Z_gen_g4_detsim_hitreco__20240510T035842Z_reco2_reco_2024-11-23T_071917Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work