justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.

Jobsub ID 233677.1@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID233677.1@justin-prod-sched02.dune.hep.ac.uk
Workflow ID8050
Stage ID1
User nameepennacc@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-07-02 07:10:33
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce513
Last heartbeat2025-07-02 10:40:28
From worker nodeHostnameb9p06p8345.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statefinished
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-07-02 07:51:47
Input filesfardet-vd:fardet-vd_mc_le_mc_2024a_list-5125_input.list_out1_standard_reco1_dunevd10kt_1x8x14_3view_30deg_hit-reconstructed_merged_skip000400_lim000025_mergeLar_20250228T102112.root
JobscriptExit code0
Real time2h (9964s)
CPU time1h (4641s = 46%)
Max RSS bytes1435103232 (1368 MiB)
Outputting started2025-07-02 10:37:53
Output filesfardet-vd:fardet-vd_mc_le_mc_2024a_list-5125_input.list_out1_standard_reco1_dunevd10kt_1x8x14_3view_30deg_hit-reconstructed_merged_skip000400_lim000025_mergeLar_20250228T102112_20250702T075154Z_reco2.root
Finished2025-07-02 10:40:28
Saved logsjustin-logs:233677.1-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

rd. run: 5125 subRun: 0 event: 42496 at 02-Jul-2025 12:37:37 CEST
PandoraContentApi::GetList(*this, m_inputHitListName, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: GetVolumeIdToHitListMap
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/AlmaLinux-9.4/build/larpandoracontent/v04_12_00-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 271
this->GetVolumeIdToHitListMap(volumeIdToHitListMap) return STATUS_CODE_NOT_INITIALIZED
    in function: Run
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/AlmaLinux-9.4/build/larpandoracontent/v04_12_00-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 165
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0003, LArDLMaster, STATUS_CODE_NOT_INITIALIZED
Begin processing the 2497th record. run: 5125 subRun: 0 event: 42497 at 02-Jul-2025 12:37:38 CEST
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  02-Jul-2025 12:37:41 CEST run: 5125 subRun: 0 event: 42497
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  02-Jul-2025 12:37:41 CEST run: 5125 subRun: 0 event: 42497
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  02-Jul-2025 12:37:41 CEST run: 5125 subRun: 0 event: 42497
Comparing two wires in the same plane: return failure
%MSG
Begin processing the 2498th record. run: 5125 subRun: 0 event: 42498 at 02-Jul-2025 12:37:41 CEST
Error: A CaloHitList is empty
ERR: Could not calculate zoom region - DlSNSignalAlgorithm unable to proceed
Error: A CaloHitList is empty
PandoraContentApi::GetList(*this, listname, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: Infer
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/AlmaLinux-9.4/build/larpandoracontent/v04_12_00-buildFW/src/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 194
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0004, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
DLVertexing: Input vertex list is empty! Can't perform pass 2
Begin processing the 2499th record. run: 5125 subRun: 0 event: 42499 at 02-Jul-2025 12:37:44 CEST
PandoraContentApi::GetList(*this, m_inputHitListName, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: GetVolumeIdToHitListMap
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/AlmaLinux-9.4/build/larpandoracontent/v04_12_00-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 271
this->GetVolumeIdToHitListMap(volumeIdToHitListMap) return STATUS_CODE_NOT_INITIALIZED
    in function: Run
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/AlmaLinux-9.4/build/larpandoracontent/v04_12_00-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 165
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0003, LArDLMaster, STATUS_CODE_NOT_INITIALIZED
Begin processing the 2500th record. run: 5125 subRun: 0 event: 42500 at 02-Jul-2025 12:37:44 CEST
PandoraContentApi::GetList(*this, m_inputHitListName, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: GetVolumeIdToHitListMap
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/AlmaLinux-9.4/build/larpandoracontent/v04_12_00-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 271
this->GetVolumeIdToHitListMap(volumeIdToHitListMap) return STATUS_CODE_NOT_INITIALIZED
    in function: Run
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/AlmaLinux-9.4/build/larpandoracontent/v04_12_00-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 165
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0003, LArDLMaster, STATUS_CODE_NOT_INITIALIZED
02-Jul-2025 12:37:45 CEST  Closed output file "fardet-vd_mc_le_mc_2024a_list-5125_input.list_out1_standard_reco1_dunevd10kt_1x8x14_3view_30deg_hit-reconstructed_merged_skip000400_lim000025_mergeLar_20250228T102112_20250702T075154Z_reco2.root"
02-Jul-2025 12:37:46 CEST  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/1c/2f/fardet-vd_mc_le_mc_2024a_list-5125_input.list_out1_standard_reco1_dunevd10kt_1x8x14_3view_30deg_hit-reconstructed_merged_skip000400_lim000025_mergeLar_20250228T102112.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.133301       2.29382       6.82084       2.85076       1.20692       2500    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000213571    0.012642      0.224294     0.000506995    0.0349838      2500    
reco:pandora:StandardPandora                           0.0139916      1.53859       4.92456       1.96726      0.873924       2500    
reco:pandoraTrack:LArPandoraTrackCreation             9.0001e-05    0.000311649   0.00467272    0.000273412   0.000178161     2500    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000145271   0.000636729   0.00953104    0.000626865   0.00045243      2500    
reco:pandoracalo:Calorimetry                          9.2021e-05    0.000256065    0.0134871    0.000240692   0.000287289     2500    
reco:pandorapid:Chi2ParticleID                        2.7491e-05    7.38482e-05   0.00122047    6.8715e-05    4.00721e-05     2500    
reco:linecluster:LineCluster                          0.000114461    0.355332      0.993299      0.432089      0.183169       2500    
reco:trajcluster:TrajCluster                          0.000149021   0.00138108     0.0130556    0.00139478    0.000820214     2500    
reco:pmtracktc:PMAlgTrackMaker                        0.00176923     0.049299      0.804366      0.0204011     0.0579066      2500    
reco:emtrkmichelid:EmTrackMichelId                    0.000132872    0.314379       1.38286      0.279341       0.24437       2500    
reco:solarflash:SolarOpFlash                           8.361e-05     0.0172123     0.348758     0.000525369    0.0455734      2500    
[art]:TriggerResults:TriggerResultInserter             9.25e-06     1.86652e-05   0.000345543   1.7015e-05    1.02905e-05     2500    
end_path:out1:RootOutput                               2.76e-06     6.04946e-06   4.6741e-05     4.89e-06     3.18122e-06     2500    
end_path:out1:RootOutput(write)                       0.000871488   0.00301027     0.0115851    0.00272831    0.00160057      2500    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2923.96 MB
  Peak resident set size usage (VmHWM): 1435.1 MB
====================================================================================================
Art has completed and will exit with status 0.
Moving on to metadata extractor 
============OUTPUT FILE:  fardet-vd_mc_le_mc_2024a_list-5125_input.list_out1_standard_reco1_dunevd10kt_1x8x14_3view_30deg_hit-reconstructed_merged_skip000400_lim000025_mergeLar_20250228T102112_20250702T075154Z_reco2.root
Extracted metadata
checking file====== fardet-vd:fardet-vd_mc_le_mc_2024a_list-5125_input.list_out1_standard_reco1_dunevd10kt_1x8x14_3view_30deg_hit-reconstructed_merged_skip000400_lim000025_mergeLar_20250228T102112.root
MD conversion to MetaCat OK
all-input-dids.txt
debugprod.log
fardet-vd_mc_le_mc_2024a_list-5125_input.list_out1_standard_reco1_dunevd10kt_1x8x14_3view_30deg_hit-reconstructed_merged_skip000400_lim000025_mergeLar_20250228T102112_20250702T075154Z_reco2.root
fardet-vd_mc_le_mc_2024a_list-5125_input.list_out1_standard_reco1_dunevd10kt_1x8x14_3view_30deg_hit-reconstructed_merged_skip000400_lim000025_mergeLar_20250228T102112_20250702T075154Z_reco2.root.ext.json
fardet-vd_mc_le_mc_2024a_list-5125_input.list_out1_standard_reco1_dunevd10kt_1x8x14_3view_30deg_hit-reconstructed_merged_skip000400_lim000025_mergeLar_20250228T102112_20250702T075154Z_reco2.root.json
jobscript.log
justin-processed-pfns.txt
old_md.json
reco2_hist.root
justIN time: 2025-08-14 21:46:23 UTC       justIN version: 01.03.02