Jobsub ID 108883.87@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108883.87@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 07:26:49
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_brug
Last heartbeat	2024-11-23 07:32:28
From worker node	Hostname	wn-choc-039.farm.nikhef.nl
	cpuinfo	Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 07:28:42
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6208838_246_20231122T053454Z_gen_g4_detsim_hitreco__20240503T064008Z_reco2.root
Jobscript	Exit code	0
	Real time	3m (211s)
	CPU time	1m (99s = 46%)
Outputting started	2024-11-23 07:32:14
Output files
Finished	2024-11-23 07:32:28
List job events Wrapper job log
Jobscript log (last 10,000 characters)

%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 08:32:13 CET run: 6208838 subRun: 1 event: 49204
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 08:32:13 CET run: 6208838 subRun: 1 event: 49204
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 08:32:13 CET run: 6208838 subRun: 1 event: 49204
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 08:32:13 CET run: 6208838 subRun: 1 event: 49204
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 08:32:13 CET run: 6208838 subRun: 1 event: 49204
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 08:32:13 CET run: 6208838 subRun: 1 event: 49204
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 08:32:13 CET run: 6208838 subRun: 1 event: 49204
Comparing two wires in the same plane: return failure
%MSG
23-Nov-2024 08:32:14 CET  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/9b/2e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6208838_246_20231122T053454Z_gen_g4_detsim_hitreco__20240503T064008Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                            0.00908533      5.32411       12.5341       4.37664       4.5307          4     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00903317     0.0111242     0.0170427    0.00921051    0.00341894        4     
reco:linecluster:LineCluster                           0.0810774     0.086761      0.0931522     0.0864072    0.00477978        4     
reco:trajcluster:TrajCluster                           0.0300098     0.0361628     0.0521688     0.0312362    0.00929495        4     
reco:pandora:StandardPandora                            1.42892       2.09935       3.07046       1.94901      0.631209         4     
reco:pandoraTrack:LArPandoraTrackCreation             0.000904722   0.00195364    0.00266735    0.00212125    0.000681403       4     
reco:pandoraShower:LArPandoraModularShowerCreation    0.00266995    0.00580184     0.0093819    0.00557776    0.00292939        4     
reco:pandoracalo:Calorimetry                          0.000642776    0.0015338    0.00222716    0.00163262    0.000644134       4     
reco:pandorapid:Chi2ParticleID                        0.000103304   0.000600919    0.0014465    0.000426935   0.000532295       4     
reco:trkshowersplit:TrackShowerHits                   0.000168466   0.000855245   0.00196558    0.000643466   0.000722872       4     
reco:pmtrack:PMAlgTrackMaker                          0.00499956     0.189384       0.7152       0.0186672     0.303778         4     
reco:pmtrackcalo:Calorimetry                          0.000108914   0.000571126   0.00165145    0.000262068   0.000635368       4     
reco:pmtrackpid:Chi2ParticleID                        4.5785e-05    9.77998e-05   0.000205843   6.97855e-05   6.52405e-05       4     
reco:pmtrajfit:PMAlgTrajFitter                        0.00144387     0.207771       0.57866      0.125491      0.235648         4     
reco:pmtrajfitcalo:Calorimetry                        0.000102122   0.000827149    0.0022552    0.000475638   0.000856498       4     
reco:pmtrajfitpid:Chi2ParticleID                      4.4385e-05    0.000116579   0.000221631   0.00010015    6.50132e-05       4     
reco:pmtracktc:PMAlgTrackMaker                        0.00148337     0.244233      0.889448      0.0430003     0.373911         4     
reco:pmtracktccalo:Calorimetry                        0.000108045   0.000471232   0.00132891    0.000223988   0.000497577       4     
reco:pmtracktcpid:Chi2ParticleID                      4.2758e-05    9.3742e-05    0.000188392   7.1909e-05    5.61797e-05       4     
reco:pmtrajfittc:PMAlgTrajFitter                      0.000290445    0.129069      0.405986      0.0549993     0.164061         4     
reco:pmtrajfittccalo:Calorimetry                      9.6302e-05    0.000569427   0.00168049    0.000250457   0.000644854       4     
reco:pmtrajfittcpid:Chi2ParticleID                    4.5472e-05    9.9017e-05    0.000197211   7.66925e-05   5.81624e-05       4     
reco:blurredcluster:BlurredClustering                  0.0750882     0.211241      0.409524       0.14911      0.143427         3     
reco:emtrkmichelid:EmTrackMichelId                     0.463365       1.78539       4.41972       0.47309       1.86276         3     
reco:emshower:EMShower                                0.00303472     0.0220398     0.0408234     0.0222612     0.0154279        3     
reco:cvnmap:CVNMapper                                 5.1305e-05    0.00425791     0.0125062    0.000216201   0.00583283        3     
reco:cvneva:CVNEvaluator                              2.4918e-05     0.429724       1.28893     0.000217781     0.60755         3     
reco:energyrecnumu:EnergyReco                         0.00362163    0.00596324     0.0092989    0.00496919    0.00242198        3     
reco:energyrecnue:EnergyReco                          0.000306219   0.00122793    0.00261367    0.00076389    0.000997524       3     
reco:energyrecnc:EnergyReco                           0.00259815    0.00279297    0.00311883    0.00266193    0.000231883       3     
reco:energyrecnumurange:EnergyReco                    0.000352825   0.00378631    0.00849215    0.00251395    0.00344251        3     
reco:energyrecnumumcs:EnergyReco                      0.00267419    0.00559042     0.0111346    0.00296252    0.00392206        3     
reco:opdec:Deconvolution                               0.123191      0.171499      0.252884      0.138423      0.0578827        3     
reco:ophitspe:OpHitFinderDeco                          0.621568      0.965037       1.59878      0.674759      0.448653         3     
reco:opflash:OpFlashFinder                            0.000512579   0.00180104    0.00345007    0.00144048    0.00122603        3     
reco:rns:RandomNumberSaver                            2.4613e-05    0.000113575   0.000290986   2.5125e-05    0.000125449       3     
reco:anglereconue:NuAngularReco                       0.000595103   0.000675413   0.000801264   0.000629871   9.01152e-05       3     
reco:anglereconumu:NuAngularReco                      0.000165817   0.000299053   0.00050011    0.000231233   0.000144655       3     
reco:anglereconuepfps:NuAngularReco                   0.000774361   0.00136851    0.00197228     0.0013589    0.000489097       3     
reco:anglereconumupfps:NuAngularReco                  0.000205573   0.000380598   0.000624044   0.000312177   0.000177559       3     
[art]:TriggerResults:TriggerResultInserter            1.2871e-05    2.52263e-05   4.9278e-05     1.353e-05    1.70092e-05       3     
end_path:out1:RootOutput                               3.106e-06    7.89833e-06   1.7268e-05     3.321e-06    6.62594e-06       3     
end_path:out1:RootOutput(write)                        0.0479125     0.128693      0.288248      0.0499192     0.112825         3     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6363.3 MB
  Peak resident set size usage (VmHWM): 1580.38 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 23-Nov-2024 08:32:14 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- ScheduleExecutionFailure BEGIN
    Path: ProcessingStopped.
    ---- BadAlloc BEGIN
      A bad_alloc exception was thrown while processing module BlurredClustering/blurredcluster run: 6208838 subRun: 1 event: 49204
      The job has probably exhausted the virtual memory available to the process.
    ---- BadAlloc END
    Exception going through path reco
  ---- ScheduleExecutionFailure END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 3 entries while sim::OpDetDivRecs_opdigi__detsim. has 200 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
RootOutput-9775-1997-d23d-dfb5.root
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6208838_246_20231122T053454Z_gen_g4_detsim_hitreco__20240503T064008Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6208838_246_20231122T053454Z_gen_g4_detsim_hitreco__20240503T064008Z_reco2_reco_2024-11-23T_072846Z.log
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work