justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.

Jobsub ID 350805.1@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID350805.1@justin-prod-sched01.dune.hep.ac.uk
Workflow ID5492
Stage ID1
User nameayankele@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-03-13 21:54:09
SiteNL_NIKHEF
EntryVIRGO_NL_NIKHEF_juk
Last heartbeat2025-03-14 04:51:24
From worker nodeHostnamewn-choc-034.farm.nikhef.nl
cpuinfoIntel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit129600 (36 hours)
GPU
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-03-14 04:49:51
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50624901_730_20231207T140535Z_gen_g4_detsim_hitreco__20240510T045130Z_reco2.root
JobscriptExit code0
Real time1m (76s)
CPU time1m (62s = 81%)
Max RSS bytes1582804992 (1509 MiB)
Outputting started2025-03-14 04:51:07
Output files
Finished2025-03-14 04:51:24
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

Input PFN = root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/73/9f/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50624901_730_20231207T140535Z_gen_g4_detsim_hitreco__20240510T045130Z_reco2.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so

MRB_PROJECT=larsoft
MRB_PROJECT_VERSION=v10_03_01
MRB_QUALS=e26:prof
MRB_TOP=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/ae8acaf9e82f18f5e2535e3ed17d6c420ab2a21b/preprocess_atmnu
MRB_SOURCE=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/ae8acaf9e82f18f5e2535e3ed17d6c420ab2a21b/preprocess_atmnu/srcs
MRB_BUILDDIR=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/ae8acaf9e82f18f5e2535e3ed17d6c420ab2a21b/preprocess_atmnu/build_slf7.x86_64
MRB_INSTALL=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/ae8acaf9e82f18f5e2535e3ed17d6c420ab2a21b/preprocess_atmnu/localProducts_larsoft_v10_03_01_e26_prof

PRODUCTS=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/ae8acaf9e82f18f5e2535e3ed17d6c420ab2a21b/preprocess_atmnu/localProducts_larsoft_v10_03_01_e26_prof:/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/ae8acaf9e82f18f5e2535e3ed17d6c420ab2a21b/preprocess_atmnu/localProducts_larsoft_v10_03_01_e26_prof/:/cvmfs/dune.opensciencegrid.org/products/dune:/cvmfs/larsoft.opensciencegrid.org/products:/cvmfs/larsoft.opensciencegrid.org/packages:/cvmfs/fermilab.opensciencegrid.org/products/common/db/
CETPKG_INSTALL=/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/ae8acaf9e82f18f5e2535e3ed17d6c420ab2a21b/preprocess_atmnu/localProducts_larsoft_v10_03_01_e26_prof

local product directory is /cvmfs/fifeuser3.opensciencegrid.org/sw/dune/ae8acaf9e82f18f5e2535e3ed17d6c420ab2a21b/preprocess_atmnu/localProducts_larsoft_v10_03_01_e26_prof
----------- this block should be empty ------------------
---------------------------------------------------------
=== Start last 100 lines of lar log file ===
Track...
TVector3 A 3D physics vector (x,y,z)=(-0.548339,0.019043,-0.836039) (rho,theta,phi)=(1.000000,146.724209,178.011008)
0.117278
Track ---> 1 43 14 : 22 111 111 111
Track...
TVector3 A 3D physics vector (x,y,z)=(0.463463,-0.865403,0.190472) (rho,theta,phi)=(1.000000,79.019651,-61.828923)
0.108036
Track...
TVector3 A 3D physics vector (x,y,z)=(0.216925,-0.930354,-0.295609) (rho,theta,phi)=(1.000000,107.194064,-76.875188)
0.138902
Track...
TVector3 A 3D physics vector (x,y,z)=(-0.219910,-0.913484,-0.342324) (rho,theta,phi)=(1.000000,110.018527,-103.535689)
0.115935
Track...
TVector3 A 3D physics vector (x,y,z)=(-0.354082,-0.780735,0.514857) (rho,theta,phi)=(1.000000,59.012088,-114.395424)
0.0740547
-1
 N Hits in Shower: 7711
0 PDG 22 111
Angle: 6.42407 6448.38
TVector3 A 3D physics vector (x,y,z)=(-5.649134,-13.130113,-1.276051) (rho,theta,phi)=(14.350641,95.101445,-113.279437)
TVector3 A 3D physics vector (x,y,z)=(-0.343066,-0.939277,0.008073) (rho,theta,phi)=(1.000000,89.537464,-110.064489)
Reco Shw Dir: 0.00807269 -1.92099
Reco vs Pri0  4.33336
 N Hits in Shower: 237
Shower ---> 1 900 6 : 22 111 111 111
1 PDG 22 111
Angle: 14.1879 153.776
TVector3 A 3D physics vector (x,y,z)=(-0.338682,-0.187059,-0.266181) (rho,theta,phi)=(0.469627,124.526901,-151.087451)
TVector3 A 3D physics vector (x,y,z)=(-0.529420,-0.492112,-0.691043) (rho,theta,phi)=(1.000000,133.712768,-137.091604)
 N Hits in Shower: 78
Shower ---> 1 43 14 : 22 111 111 111
2 PDG 22 111
Angle: 16.6534 76.9385
TVector3 A 3D physics vector (x,y,z)=(-0.063046,0.002811,-0.130649) (rho,theta,phi)=(0.145092,154.217553,177.447447)
TVector3 A 3D physics vector (x,y,z)=(-0.527535,0.275947,-0.803468) (rho,theta,phi)=(1.000000,143.462541,152.386447)
 N Hits in Shower: 36
3 PDG 22 111
Angle: 53.6714 22.1917
TVector3 A 3D physics vector (x,y,z)=(-5.649134,-13.130113,-1.276051) (rho,theta,phi)=(14.350641,95.101445,-113.279437)
TVector3 A 3D physics vector (x,y,z)=(0.463468,-0.865400,0.190475) (rho,theta,phi)=(1.000000,79.019491,-61.828551)
 N Hits in Shower: 24
4 PDG 2212 211
Angle: 102.922 54.228
TVector3 A 3D physics vector (x,y,z)=(0.186854,0.034177,0.276551) (rho,theta,phi)=(0.335504,34.483865,10.365316)
TVector3 A 3D physics vector (x,y,z)=(0.212364,-0.930054,-0.299835) (rho,theta,phi)=(1.000000,107.447674,-77.137860)
 N Hits in Shower: 19
5 PDG 2212 -1
Angle: 78.8254 55.9638
TVector3 A 3D physics vector (x,y,z)=(0.257140,-0.024889,-0.341561) (rho,theta,phi)=(0.428257,142.897709,-5.528628)
TVector3 A 3D physics vector (x,y,z)=(-0.220620,-0.913234,-0.342535) (rho,theta,phi)=(1.000000,110.031377,-103.581370)
 N Hits in Shower: 14
6 PDG 2212 -1
Angle: 5.94737 49.2585
TVector3 A 3D physics vector (x,y,z)=(-0.127196,-0.299839,0.243828) (rho,theta,phi)=(0.406859,53.180721,-112.987400)
TVector3 A 3D physics vector (x,y,z)=(-0.354086,-0.780736,0.514853) (rho,theta,phi)=(1.000000,59.012342,-114.395668)
-11000000
Loop Hits: 9020
End of Hit Loop 
Reco nuE: 18.3221
Reco lepE: 0.174677
Reco hadE: 18.1475
Reco Track Cont: 1
True Particle T: 0  0.160563
14-Mar-2025 05:51:06 CET  Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/73/9f/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50624901_730_20231207T140535Z_gen_g4_detsim_hitreco__20240510T045130Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.0133237     0.642245       3.22474      0.368961      0.653401        39     
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.00137637     0.017327      0.0378605     0.0173928    0.00914931       39     
reco:rns:RandomNumberSaver                    4.1085e-05    5.23376e-05   0.000362861   4.3619e-05    5.04355e-05      39     
[art]:TriggerResults:TriggerResultInserter    1.3514e-05    1.56891e-05   5.0865e-05    1.4861e-05    5.72133e-06      39     
end_path:recoEnergy:RecoEnergyS                0.193143      0.641132       3.20876      0.348931      0.652047        38     
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 6009.63 MB
  Peak resident set size usage (VmHWM): 1582.61 MB
====================================================================================================
%MSG-s ArtException:  PostEndJob 14-Mar-2025 05:51:07 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- EventProcessorFailure BEGIN
    EndPathExecutor: an exception occurred during current event processing
    ---- ScheduleExecutionFailure BEGIN
      Path: ProcessingStopped.
      ---- BadAlloc BEGIN
        A bad_alloc exception was thrown while processing module RecoEnergyS/recoEnergy run: 50624901 subRun: 1 event: 73039
        The job has probably exhausted the virtual memory available to the process.
      ---- BadAlloc END
      Exception going through path end_path
    ---- ScheduleExecutionFailure END
  ---- EventProcessorFailure END
---- EventProcessorFailure END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2025-08-14 16:36:50 UTC       justIN version: 01.03.02