justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 168206.9@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID168206.9@justin-prod-sched02.dune.hep.ac.uk
Workflow ID5840
Stage ID1
User nameamoor@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6291456000 (6000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-04-01 11:18:06
SiteUK_Brunel
EntryCMSHTPC_T2_UK_London_Brunel_dc2_22
Last heartbeat2025-04-01 11:21:45
From worker nodeHostnamewn-a2-26-01.brunel.ac.uk
cpuinfoIntel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6291456000 (6000 MiB)
Wall seconds limit171000 (47 hours)
GPU
Inner Apptainer?True
Job statefinished
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-04-01 11:19:22
Input filesusertests:000013_reco_data_2025-03-31T_142801Z_reco_data_2025-03-31T_155252Z_reco_data_2025-04-01T_083044Z_reco_data_2025-04-01T_101343Z.root
JobscriptExit code0
Real time1m (76s)
CPU time0m (56s = 73%)
Max RSS bytes1170915328 (1116 MiB)
Outputting started2025-04-01 11:20:39
Output filesusertests:000013_reco_data_2025-03-31T_142801Z_reco_data_2025-03-31T_155252Z_reco_data_2025-04-01T_083044Z_reco_data_2025-04-01T_101343Z_reco_data_2025-04-01T_111925Z.root
Finished2025-04-01 11:21:45
Saved logsjustin-logs:168206.9-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

AFM reco2 jobscript.
Input PFN = root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/usertests/34/f4/000013_reco_data_2025-03-31T_142801Z_reco_data_2025-03-31T_155252Z_reco_data_2025-04-01T_083044Z_reco_data_2025-04-01T_101343Z.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 50 lines of lar log file ===
Begin processing the 19th record. run: 20000031 subRun: 0 event: 19 at 01-Apr-2025 12:20:35 BST
PandoraContentApi::GetList(*this, m_inputHitListName, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: GetVolumeIdToHitListMap
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_14_01-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 271
this->GetVolumeIdToHitListMap(volumeIdToHitListMap) return STATUS_CODE_NOT_INITIALIZED
    in function: Run
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_14_01-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 165
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0003, LArDLMaster, STATUS_CODE_NOT_INITIALIZED
Begin processing the 20th record. run: 20000031 subRun: 0 event: 20 at 01-Apr-2025 12:20:35 BST
PandoraContentApi::GetList(*this, m_inputHitListName, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: GetVolumeIdToHitListMap
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_14_01-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 271
this->GetVolumeIdToHitListMap(volumeIdToHitListMap) return STATUS_CODE_NOT_INITIALIZED
    in function: Run
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/ALMA9/build/larpandoracontent/v04_14_01-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 165
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0003, LArDLMaster, STATUS_CODE_NOT_INITIALIZED
01-Apr-2025 12:20:36 BST  Closed output file "000013_reco_data_2025-03-31T_142801Z_reco_data_2025-03-31T_155252Z_reco_data_2025-04-01T_083044Z_reco_data_2025-04-01T_101343Z_reco_data_2025-04-01T_111925Z.root"
01-Apr-2025 12:20:36 BST  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/usertests/34/f4/000013_reco_data_2025-03-31T_142801Z_reco_data_2025-03-31T_155252Z_reco_data_2025-04-01T_083044Z_reco_data_2025-04-01T_101343Z.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                            0.00809978     0.0116329     0.0508265    0.00856133    0.00921708       20     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00457956    0.00562527    0.00916975    0.00470228    0.00175592       20     
reco:pandora:StandardPandora                          0.000883266    0.0026556     0.0324498    0.000990321   0.00684484       20     
reco:pandoraTrack:LArPandoraTrackCreation             0.00012088    0.000214051    0.0016215    0.000134137   0.00032433       20     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000163651   0.000260607   0.00184455    0.000170381   0.000363619      20     
reco:pandoracalo:Calorimetry                          0.000237552    0.000314     0.00140305    0.000249339   0.000250829      20     
reco:pandorapid:Chi2ParticleID                        3.5589e-05    5.78779e-05   0.000457917   3.67965e-05   9.17803e-05      20     
reco:trkshowersplit:TrackShowerHits                   0.000198629   0.00022998    0.000600661   0.000208028   8.57135e-05      20     
reco:rns:RandomNumberSaver                            1.4514e-05    2.47345e-05   0.000180253   1.5705e-05    3.5737e-05       20     
[art]:TriggerResults:TriggerResultInserter             9.633e-06    1.19739e-05   4.0385e-05    1.02225e-05   6.56946e-06      20     
end_path:out1:RootOutput                               2.688e-06    3.6542e-06    1.5529e-05     2.875e-06    2.78356e-06      20     
end_path:out1:RootOutput(write)                       0.00156706     0.0019576    0.00592783    0.00173189    0.000919866      20     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2246.47 MB
  Peak resident set size usage (VmHWM): 1170.92 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 50 lines of lar log file ===
lar exit code 0
extractor_prod.py exit code 0
pdjson2metadata exit code 0
Metadata extraction succeeds
===Metadata JSON===
{
    "metadata": {
        "DUNE.campaign": "justIN.w5840s1",
        "art.file_format_era": "ART_2011a",
        "art.file_format_version": 15,
        "art.first_event": 1,
        "art.last_event": 20,
        "art.process_name": "Reco2",
        "art.run_type": "physics",
        "core.application": "art.reco",
        "core.application.family": "art",
        "core.application.name": "reco",
        "core.application.version": "v10_04_06d00",
        "core.data_stream": "out1",
        "core.data_tier": "full-reconstructed",
        "core.end_time": 1743506436.0,
        "core.event_count": 20,
        "core.file_format": "artroot",
        "core.file_type": "mc",
        "core.first_event_number": 1,
        "core.last_event_number": 20,
        "core.run_type": "physics",
        "core.runs": [
            20000031
        ],
        "core.runs_subruns": [
            2000003100000
        ],
        "core.start_time": 1743506417.0
    },
    "parents": [
        {
            "did": "usertests:000013_reco_data_2025-03-31T_142801Z_reco_data_2025-03-31T_155252Z_reco_data_2025-04-01T_083044Z_reco_data_2025-04-01T_101343Z.root"
        }
    ]
}
===================
.:
total 141308
-rw-r--r--. 1 pildune000 dunepil 144621156 Apr  1 12:20 000013_reco_data_2025-03-31T_142801Z_reco_data_2025-03-31T_155252Z_reco_data_2025-04-01T_083044Z_reco_data_2025-04-01T_101343Z_reco_data_2025-04-01T_111925Z.root
-rw-r--r--. 1 pildune000 dunepil     48145 Apr  1 12:20 000013_reco_data_2025-03-31T_142801Z_reco_data_2025-03-31T_155252Z_reco_data_2025-04-01T_083044Z_reco_data_2025-04-01T_101343Z_reco_2025-04-01T_111925Z.log
-rw-r--r--. 1 pildune000 dunepil      7358 Apr  1 12:20 jobscript.log
-rw-r--r--. 1 pildune000 dunepil      1168 Apr  1 12:20 000013_reco_data_2025-03-31T_142801Z_reco_data_2025-03-31T_155252Z_reco_data_2025-04-01T_083044Z_reco_data_2025-04-01T_101343Z_reco_data_2025-04-01T_111925Z.root.json
-rw-r--r--. 1 pildune000 dunepil      1042 Apr  1 12:20 000013_reco_data_2025-03-31T_142801Z_reco_data_2025-03-31T_155252Z_reco_data_2025-04-01T_083044Z_reco_data_2025-04-01T_101343Z_reco_data_2025-04-01T_111925Z.root.ext.json
-rw-r--r--. 1 pildune000 dunepil       519 Apr  1 12:20 reco2_hist.root
-rw-r--r--. 1 pildune000 dunepil       284 Apr  1 12:19 all-input-dids.txt
-rw-r--r--. 1 pildune000 dunepil       205 Apr  1 12:20 justin-processed-pfns.txt
-rw-r--r--. 1 pildune000 dunepil         0 Apr  1 12:19 debugprod.log
justIN time: 2025-04-03 08:14:27 UTC       justIN version: 01.03.00