Jobsub ID 301864.1@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301864.1@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 07:13:11
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_klomp
Last heartbeat	2024-11-23 07:17:24
From worker node	Hostname	wn-choc-034.farm.nikhef.nl
	cpuinfo	Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 07:13:57
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496970_37_20231202T004235Z_gen_g4_detsim_hitreco__20240508T053127Z_reco2.root
Jobscript	Exit code	0
	Real time	3m (193s)
	CPU time	1m (99s = 51%)
Outputting started	2024-11-23 07:17:11
Output files
Finished	2024-11-23 07:17:24
List job events Wrapper job log
Jobscript log (last 10,000 characters)

970 subRun: 1 event: 3703
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:17:08 CET run: 74496970 subRun: 1 event: 3703
1st wire C:0 T:13 P:2 W:890 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:17:09 CET run: 74496970 subRun: 1 event: 3703
1st wire C:0 T:13 P:2 W:1012 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:17:09 CET run: 74496970 subRun: 1 event: 3703
1st wire C:0 T:15 P:2 W:837 does not exist (max wire number: 480)
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:17:10 CET run: 74496970 subRun: 1 event: 3703
Flip, endpoint closer to vStart.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:17:10 CET run: 74496970 subRun: 1 event: 3703
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:17:10 CET run: 74496970 subRun: 1 event: 3703
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:17:10 CET run: 74496970 subRun: 1 event: 3703
TuneFullTree failed.
%MSG
23-Nov-2024 08:17:10 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/83/1b/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496970_37_20231202T004235Z_gen_g4_detsim_hitreco__20240508T053127Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.0389361      4.64045       8.22483       5.6576        3.4184          3     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0152248     0.0272251     0.0389361     0.0275144    0.00968227        3     
reco:linecluster:LineCluster                           0.142344      0.165191      0.185204      0.168025      0.0176119        3     
reco:trajcluster:TrajCluster                           0.0488099     0.315799      0.841227      0.0573592     0.371551         3     
reco:pandora:StandardPandora                            2.15336       2.47961       3.07785       2.20762      0.423601         3     
reco:pandoraTrack:LArPandoraTrackCreation             0.000659817   0.00501453     0.0101498     0.004234     0.00391337        3     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00200469     0.0101882     0.0167098     0.0118502    0.00611726        3     
reco:pandoracalo:Calorimetry                          0.000551249    0.0048606     0.0110596    0.00297099    0.00449326        3     
reco:pandorapid:Chi2ParticleID                        0.000117993   0.000529522   0.00124585    0.000224724   0.00050839        3     
reco:trkshowersplit:TrackShowerHits                   0.00027609    0.00725616     0.0196811    0.00181129    0.00880808        3     
reco:pmtrack:PMAlgTrackMaker                           0.0361075      1.34045       3.93004      0.0552025      1.83113         3     
reco:pmtrackcalo:Calorimetry                          0.000501144   0.00514893     0.0144086    0.000537056   0.00654758        3     
reco:pmtrackpid:Chi2ParticleID                         9.994e-05    0.000267553   0.000594005   0.000108714   0.000230864       3     
reco:pmtrajfit:PMAlgTrajFitter                         0.0380396      1.00575       2.87461      0.104597       1.32176         3     
reco:pmtrajfitcalo:Calorimetry                        0.000566222   0.00477407     0.0131674    0.000588547   0.00593502        3     
reco:pmtrajfitpid:Chi2ParticleID                      0.000110082   0.000199016   0.000373076   0.000113889   0.000123089       3     
reco:pmtracktc:PMAlgTrackMaker                         0.0130591       1.041        2.97936      0.130581       1.37147         3     
reco:pmtracktccalo:Calorimetry                        0.000390299   0.00416834     0.0115467    0.000567998   0.00521782        3     
reco:pmtracktcpid:Chi2ParticleID                      9.8436e-05    0.000236846   0.00049696    0.000115143   0.000184055       3     
reco:pmtrajfittc:PMAlgTrajFitter                       0.0377307     0.962718       2.79543      0.0549917      1.29594         3     
reco:pmtrajfittccalo:Calorimetry                      0.000405822    0.0049346     0.013888     0.000509981   0.00633115        3     
reco:pmtrajfittcpid:Chi2ParticleID                    9.5442e-05    0.000263198   0.000585287   0.000108864   0.000227817       3     
reco:blurredcluster:BlurredClustering                   0.05594      0.516462      0.976984      0.516462      0.460522         2     
reco:emtrkmichelid:EmTrackMichelId                     0.905939       1.24289       1.57985       1.24289      0.336955         2     
reco:emshower:EMShower                                0.000646962   0.00335149    0.00605601    0.00335149    0.00270453        2     
reco:cvnmap:CVNMapper                                 0.00018912     0.017459      0.0347288     0.017459      0.0172698        2     
reco:cvneva:CVNEvaluator                              0.000188455    0.588906       1.17762      0.588906      0.588718         2     
reco:energyrecnumu:EnergyReco                         0.00410434    0.00469721    0.00529007    0.00469721    0.000592868       2     
reco:energyrecnue:EnergyReco                          0.000264671   0.000532843   0.000801014   0.000532843   0.000268171       2     
reco:energyrecnc:EnergyReco                            0.0026364    0.00265817    0.00267994    0.00265817    2.17715e-05       2     
reco:energyrecnumurange:EnergyReco                    0.00223251    0.00240668    0.00258086    0.00240668    0.000174179       2     
reco:energyrecnumumcs:EnergyReco                      0.00273697     0.0038799    0.00502284     0.0038799    0.00114294        2     
reco:opdec:Deconvolution                               0.194992      0.218524      0.242055      0.218524      0.0235316        2     
reco:ophitspe:OpHitFinderDeco                          0.916669       1.05126       1.18585       1.05126       0.13459         2     
reco:opflash:OpFlashFinder                            0.00103072    0.00234516    0.00365959    0.00234516    0.00131444        2     
reco:rns:RandomNumberSaver                            1.9825e-05    0.000150587   0.000281348   0.000150587   0.000130761       2     
reco:anglereconue:NuAngularReco                       0.000550532   0.000779955   0.00100938    0.000779955   0.000229424       2     
reco:anglereconumu:NuAngularReco                      0.000195355   0.000230266   0.000265176   0.000230266   3.49105e-05       2     
reco:anglereconuepfps:NuAngularReco                   0.000856989   0.00143933    0.00202167    0.00143933    0.00058234        2     
reco:anglereconumupfps:NuAngularReco                  0.000235635   0.000284051   0.000332467   0.000284051   4.8416e-05        2     
[art]:TriggerResults:TriggerResultInserter            1.2388e-05    2.67835e-05   4.1179e-05    2.67835e-05   1.43955e-05       2     
end_path:out1:RootOutput                               2.71e-06      8.165e-06     1.362e-05     8.165e-06     5.455e-06        2     
end_path:out1:RootOutput(write)                        0.0692405     0.158399      0.247557      0.158399      0.0891585        2     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6365.81 MB
  Peak resident set size usage (VmHWM): 1583.06 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Nov-2024 08:17:10 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module BlurredClustering/blurredcluster run: 74496970 subRun: 1 event: 3703
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 2 entries while sim::OpDetDivRecs_opdigi__detsim. has 100 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-e5fa-1aa6-2685-8111.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496970_37_20231202T004235Z_gen_g4_detsim_hitreco__20240508T053127Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496970_37_20231202T004235Z_gen_g4_detsim_hitreco__20240508T053127Z_reco2_reco_2024-11-23T_071400Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work