justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.

Jobsub ID 231327.34@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID231327.34@justin-prod-sched02.dune.hep.ac.uk
Workflow ID7983
Stage ID1
User nameimawby@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit43200 (12 hours)
Submitted time2025-06-27 12:54:25
SiteUS_UChicago
EntryEngage_US_MWT2_iut2_gk02_condce_mcore
Last heartbeat2025-06-27 14:19:15
From worker nodeHostnameiut2-c337.iu.edu
cpuinfoAMD EPYC 7402 24-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
GPU
Inner Apptainer?True
Job statefinished
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-06-27 13:54:52
Input filesfardet-hd:nutau_dune10kt_1x2x6_1099_737_20230826T025216Z_gen_g4_detsim_hitreco__20240220T194500Z_reco2.root
JobscriptExit code0
Real time24m (1450s)
CPU time22m (1357s = 93%)
Max RSS bytes1452572672 (1385 MiB)
Outputting started2025-06-27 14:19:02
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/imawby/splitting_nutau_0/07983/1/001/HierarchyRecoPerformance_nutau_dune10kt_1x2x6_1099_737_20230826T025216Z_gen_g4_detsim_hitreco__20240220T194500Z_reco2.root
Finished2025-06-27 14:19:15
Saved logsjustin-logs:231327.34-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

Justin processors: 1
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/

MRB_PROJECT=larsoft
MRB_PROJECT_VERSION=v10_08_00
MRB_QUALS=e26:prof
MRB_TOP=/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/ba960f2d78c1c5ee2ebb9823e8434f1c8c704e19
MRB_SOURCE=/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/ba960f2d78c1c5ee2ebb9823e8434f1c8c704e19/srcs
MRB_BUILDDIR=/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/ba960f2d78c1c5ee2ebb9823e8434f1c8c704e19/build_slf7.x86_64
MRB_INSTALL=/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/ba960f2d78c1c5ee2ebb9823e8434f1c8c704e19/localProducts_larsoft_v10_08_00_e26_prof

PRODUCTS=/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/ba960f2d78c1c5ee2ebb9823e8434f1c8c704e19/localProducts_larsoft_v10_08_00_e26_prof:/cvmfs/dune.opensciencegrid.org/products/dune:/cvmfs/larsoft.opensciencegrid.org/products:/cvmfs/larsoft.opensciencegrid.org/packages:/cvmfs/fermilab.opensciencegrid.org/products/common/db/
CETPKG_INSTALL=/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/ba960f2d78c1c5ee2ebb9823e8434f1c8c704e19/localProducts_larsoft_v10_08_00_e26_prof

local product directory is /cvmfs/fifeuser1.opensciencegrid.org/sw/dune/ba960f2d78c1c5ee2ebb9823e8434f1c8c704e19/localProducts_larsoft_v10_08_00_e26_prof
----------- this block should be empty ------------------
---------------------------------------------------------
Input PFN = root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/ee/2f/nutau_dune10kt_1x2x6_1099_737_20230826T025216Z_gen_g4_detsim_hitreco__20240220T194500Z_reco2.root
lar exit code 0
=== Start last 100 lines of lar log file ===
YES RECO: 22 GEN: 3
YES RECO: 2212 GEN: 3
YES RECO: 2212 GEN: 3
Begin processing the 61st record. run: 1099 subRun: 1 event: 73761 at 27-Jun-2025 10:16:54 EDT
Begin processing the 62nd record. run: 1099 subRun: 1 event: 73762 at 27-Jun-2025 10:16:57 EDT
Begin processing the 63rd record. run: 1099 subRun: 1 event: 73763 at 27-Jun-2025 10:17:00 EDT
Begin processing the 64th record. run: 1099 subRun: 1 event: 73764 at 27-Jun-2025 10:17:03 EDT
Begin processing the 65th record. run: 1099 subRun: 1 event: 73765 at 27-Jun-2025 10:17:06 EDT
Begin processing the 66th record. run: 1099 subRun: 1 event: 73766 at 27-Jun-2025 10:17:09 EDT
Begin processing the 67th record. run: 1099 subRun: 1 event: 73767 at 27-Jun-2025 10:17:11 EDT
YES RECO: 2212 GEN: 3
YES RECO: 2212 GEN: 3
YES RECO: -211 GEN: 3
Begin processing the 68th record. run: 1099 subRun: 1 event: 73768 at 27-Jun-2025 10:17:14 EDT
Begin processing the 69th record. run: 1099 subRun: 1 event: 73769 at 27-Jun-2025 10:17:17 EDT
YES RECO: 211 GEN: 3
YES RECO: 2212 GEN: 4
Begin processing the 70th record. run: 1099 subRun: 1 event: 73770 at 27-Jun-2025 10:17:22 EDT
YES RECO: 22 GEN: 3
YES RECO: 22 GEN: 3
Begin processing the 71st record. run: 1099 subRun: 1 event: 73771 at 27-Jun-2025 10:17:25 EDT
Begin processing the 72nd record. run: 1099 subRun: 1 event: 73772 at 27-Jun-2025 10:17:28 EDT
YES RECO: -211 GEN: 3
YES RECO: 2212 GEN: 3
YES RECO: -11 GEN: 3
Begin processing the 73rd record. run: 1099 subRun: 1 event: 73773 at 27-Jun-2025 10:17:31 EDT
Begin processing the 74th record. run: 1099 subRun: 1 event: 73774 at 27-Jun-2025 10:17:34 EDT
Begin processing the 75th record. run: 1099 subRun: 1 event: 73775 at 27-Jun-2025 10:17:37 EDT
Begin processing the 76th record. run: 1099 subRun: 1 event: 73776 at 27-Jun-2025 10:17:41 EDT
Begin processing the 77th record. run: 1099 subRun: 1 event: 73777 at 27-Jun-2025 10:17:44 EDT
Begin processing the 78th record. run: 1099 subRun: 1 event: 73778 at 27-Jun-2025 10:17:47 EDT
Begin processing the 79th record. run: 1099 subRun: 1 event: 73779 at 27-Jun-2025 10:17:49 EDT
Begin processing the 80th record. run: 1099 subRun: 1 event: 73780 at 27-Jun-2025 10:17:53 EDT
YES RECO: 22 GEN: 3
YES RECO: 22 GEN: 3
YES RECO: 22 GEN: 3
YES RECO: 22 GEN: 3
Begin processing the 81st record. run: 1099 subRun: 1 event: 73781 at 27-Jun-2025 10:17:58 EDT
Begin processing the 82nd record. run: 1099 subRun: 1 event: 73782 at 27-Jun-2025 10:18:00 EDT
Begin processing the 83rd record. run: 1099 subRun: 1 event: 73783 at 27-Jun-2025 10:18:03 EDT
Begin processing the 84th record. run: 1099 subRun: 1 event: 73784 at 27-Jun-2025 10:18:06 EDT
Begin processing the 85th record. run: 1099 subRun: 1 event: 73785 at 27-Jun-2025 10:18:09 EDT
YES RECO: -13 GEN: 3
YES RECO: -11 GEN: 4
YES RECO: -11 GEN: 3
Begin processing the 86th record. run: 1099 subRun: 1 event: 73786 at 27-Jun-2025 10:18:12 EDT
Begin processing the 87th record. run: 1099 subRun: 1 event: 73787 at 27-Jun-2025 10:18:14 EDT
YES RECO: 2212 GEN: 3
YES RECO: -211 GEN: 3
Begin processing the 88th record. run: 1099 subRun: 1 event: 73788 at 27-Jun-2025 10:18:17 EDT
Begin processing the 89th record. run: 1099 subRun: 1 event: 73789 at 27-Jun-2025 10:18:20 EDT
Begin processing the 90th record. run: 1099 subRun: 1 event: 73790 at 27-Jun-2025 10:18:23 EDT
Begin processing the 91st record. run: 1099 subRun: 1 event: 73791 at 27-Jun-2025 10:18:26 EDT
YES RECO: 2212 GEN: 3
YES RECO: 22 GEN: 3
YES RECO: 22 GEN: 3
Begin processing the 92nd record. run: 1099 subRun: 1 event: 73792 at 27-Jun-2025 10:18:33 EDT
Begin processing the 93rd record. run: 1099 subRun: 1 event: 73793 at 27-Jun-2025 10:18:36 EDT
Begin processing the 94th record. run: 1099 subRun: 1 event: 73794 at 27-Jun-2025 10:18:39 EDT
PandoraContentApi::GetList(*this, m_inputHitListName, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: GetVolumeIdToHitListMap
    in file:     /exp/dune/app/users/imawby/dunesw_splitting/srcs/larpandoracontent/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 271
this->GetVolumeIdToHitListMap(volumeIdToHitListMap) return STATUS_CODE_NOT_INITIALIZED
    in function: Run
    in file:     /exp/dune/app/users/imawby/dunesw_splitting/srcs/larpandoracontent/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 165
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0002, LArDLMaster, STATUS_CODE_NOT_INITIALIZED
Begin processing the 95th record. run: 1099 subRun: 1 event: 73795 at 27-Jun-2025 10:18:39 EDT
YES RECO: 2212 GEN: 3
Begin processing the 96th record. run: 1099 subRun: 1 event: 73796 at 27-Jun-2025 10:18:42 EDT
Begin processing the 97th record. run: 1099 subRun: 1 event: 73797 at 27-Jun-2025 10:18:45 EDT
Begin processing the 98th record. run: 1099 subRun: 1 event: 73798 at 27-Jun-2025 10:18:48 EDT
Begin processing the 99th record. run: 1099 subRun: 1 event: 73799 at 27-Jun-2025 10:18:53 EDT
YES RECO: 2212 GEN: 3
YES RECO: 2212 GEN: 3
Begin processing the 100th record. run: 1099 subRun: 1 event: 73800 at 27-Jun-2025 10:18:59 EDT
27-Jun-2025 10:19:01 EDT  Closed output file "nutau_dune10kt_1x2x6_1099_737_20230826T025216Z_gen_g4_detsim_hitreco__20240220T194500Z_reco2_reco2.root"
27-Jun-2025 10:19:01 EDT  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-hd/ee/2f/nutau_dune10kt_1x2x6_1099_737_20230826T025216Z_gen_g4_detsim_hitreco__20240220T194500Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.072843       13.1994       672.549       2.86024       68.3917        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.0119372     0.0411562     0.474771      0.0300719     0.0552895       100    
reco:pandora:StandardPandora                   0.0464728      13.1449       672.414       2.80727       68.3812        100    
[art]:TriggerResults:TriggerResultInserter    1.1713e-05    2.51996e-05   5.7579e-05    2.31935e-05   1.01162e-05      100    
end_path:out1:RootOutput                       3.376e-06    5.22559e-06   1.9617e-05     4.278e-06    2.29934e-06      100    
end_path:out1:RootOutput(write)                0.0011212     0.0130807     0.104479     0.00506166     0.019862        100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2440.09 MB
  Peak resident set size usage (VmHWM): 1452.57 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
justIN time: 2025-08-14 21:53:21 UTC       justIN version: 01.03.02