justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 257583.14@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID257583.14@justin-prod-sched01.dune.hep.ac.uk
Workflow ID3262
Stage ID1
User nameamoor@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes5242880000 (5000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-09-16 15:23:58
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2024-09-16 20:02:40
From worker nodeHostnamedunegli-3944254-0-fnpc19132.fnal.gov
cpuinfoAMD EPYC 7502 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes5242880000 (5000 MiB)
Wall seconds limit172800 (48 hours)
Inner Apptainer?True
Job statefinished
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-09-16 15:26:04
Input filesusertests:000066_reco_data_2024-08-16T_161646Z_reco_data_2024-08-19T_112044Z_reco_data_2024-08-20T_114144Z_reco_data_2024-08-22T_091719Z.root
JobscriptExit code0
Real time4h (16550s)
CPU time4h (16360s = 98%)
Outputting started2024-09-16 20:01:57
Output filesusertests:000066_reco_data_2024-08-16T_161646Z_reco_data_2024-08-19T_112044Z_reco_data_2024-08-20T_114144Z_reco_data_2024-08-22T_091719Z_reco_data_2024-09-16T_152610Z.root
Finished2024-09-16 20:02:40
Saved logsjustin-logs:257583.14-justin-prod-sched01.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

AFM reco2 jobscript.
Input PFN = root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/ec/65/000066_reco_data_2024-08-16T_161646Z_reco_data_2024-08-19T_112044Z_reco_data_2024-08-20T_114144Z_reco_data_2024-08-22T_091719Z.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_4_3b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libXrdPosixPreload.so
=== Start last 50 lines of lar log file ===
PandoraContentApi::GetList(*this, m_inputHitListName, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: GetVolumeIdToHitListMap
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s120a-e20/label1/swarm/label2/SLF7/build/larpandoracontent/v04_04_02-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 271
this->GetVolumeIdToHitListMap(volumeIdToHitListMap) return STATUS_CODE_NOT_INITIALIZED
    in function: Run
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s120a-e20/label1/swarm/label2/SLF7/build/larpandoracontent/v04_04_02-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 165
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s120a-e20/label1/swarm/label2/SLF7/build/pandora/v03_16_00h/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0003, LArDLMaster, STATUS_CODE_NOT_INITIALIZED
Begin processing the 17th record. run: 20000031 subRun: 0 event: 961 at 16-Sep-2024 20:01:47 UTC
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  16-Sep-2024 20:01:47 UTC run: 20000031 subRun: 0 event: 961
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  16-Sep-2024 20:01:47 UTC run: 20000031 subRun: 0 event: 961
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
Begin processing the 18th record. run: 20000031 subRun: 0 event: 984 at 16-Sep-2024 20:01:47 UTC
16-Sep-2024 20:01:48 UTC  Closed output file "000066_reco_data_2024-08-16T_161646Z_reco_data_2024-08-19T_112044Z_reco_data_2024-08-20T_114144Z_reco_data_2024-08-22T_091719Z_reco_data_2024-09-16T_152610Z.root"
16-Sep-2024 20:01:48 UTC  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/ec/65/000066_reco_data_2024-08-16T_161646Z_reco_data_2024-08-19T_112044Z_reco_data_2024-08-20T_114144Z_reco_data_2024-08-22T_091719Z.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.012015       915.513       16210.2      0.883368       3709.91        18     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000601325   0.00417705     0.0206501    0.00135799    0.00574081       18     
reco:linecluster:LineCluster                          0.00160864      30.7495       489.796      0.0559604      112.191        18     
reco:trajcluster:TrajCluster                          0.00100005      31.969        450.471      0.247702       104.716        18     
reco:pandora:StandardPandora                          0.000978538     846.766       15171.8      0.220033       3474.35        18     
reco:pandoraTrack:LArPandoraTrackCreation             0.000134231     1.23074       19.4205      0.0331956      4.42679        18     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000207332    0.166383      0.831182      0.0446855     0.221382        18     
reco:pandoracalo:Calorimetry                          0.000103691    0.467239       7.02001      0.0310668      1.59682        18     
reco:pandorapid:Chi2ParticleID                        3.9811e-05    0.00150119      0.01194     0.000494444   0.00270281       18     
reco:trkshowersplit:TrackShowerHits                   0.000175062    0.462837       6.48076      0.0119325      1.49342        18     
reco:ophit:OpHitFinder                                0.00176165      3.52905       63.2477      0.0155636      14.4839        18     
reco:opflash:OpFlashFinder                            0.000152871   0.000765912   0.00274038    0.000661096   0.000594245      18     
reco:rns:RandomNumberSaver                             1.746e-05    3.77408e-05   0.000236052   2.4075e-05    4.85559e-05      18     
[art]:TriggerResults:TriggerResultInserter             1.034e-05    1.54257e-05   4.6921e-05    1.36305e-05   7.78017e-06      18     
end_path:out1:RootOutput                               3.64e-06     6.07844e-06   2.3901e-05     4.365e-06    4.62369e-06      18     
end_path:out1:RootOutput(write)                       0.00379854     0.165201       1.51495      0.0682529     0.342179        18     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3035.19 MB
  Peak resident set size usage (VmHWM): 1972.31 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 50 lines of lar log file ===
lar exit code 0
extractor_prod.py exit code 0
pdjson2metadata exit code 0
Metadata extraction succeeds
===Metadata JSON===
{
    "metadata": {
        "DUNE.campaign": "justIN.w3262s1",
        "art.file_format_era": "ART_2011a",
        "art.file_format_version": 15,
        "art.first_event": 107,
        "art.last_event": 984,
        "art.process_name": "Reco2",
        "art.run_type": "physics",
        "core.application": "art.reco",
        "core.application.family": "art",
        "core.application.name": "reco",
        "core.application.version": "v09_75_03d00",
        "core.data_stream": "out1",
        "core.data_tier": "full-reconstructed",
        "core.end_time": 1726516908.0,
        "core.event_count": 18,
        "core.file_format": "artroot",
        "core.file_type": "mc",
        "core.first_event_number": 107,
        "core.last_event_number": 984,
        "core.run_type": "physics",
        "core.runs": [
            20000031
        ],
        "core.runs_subruns": [
            2000003100000
        ],
        "core.start_time": 1726500413.0
    },
    "parents": [
        {
            "did": "usertests:000066_reco_data_2024-08-16T_161646Z_reco_data_2024-08-19T_112044Z_reco_data_2024-08-20T_114144Z_reco_data_2024-08-22T_091719Z.root"
        }
    ]
}
===================
.:
total 1195360
-rw-r--r-- 1 dunegli fnalgrid 1223661189 Sep 16 20:01 000066_reco_data_2024-08-16T_161646Z_reco_data_2024-08-19T_112044Z_reco_data_2024-08-20T_114144Z_reco_data_2024-08-22T_091719Z_reco_data_2024-09-16T_152610Z.root
-rw-r--r-- 1 dunegli fnalgrid     346843 Sep 16 20:01 000066_reco_data_2024-08-16T_161646Z_reco_data_2024-08-19T_112044Z_reco_data_2024-08-20T_114144Z_reco_data_2024-08-22T_091719Z_reco_2024-09-16T_152610Z.log
-rw-r--r-- 1 dunegli fnalgrid       7560 Sep 16 20:01 reco2_hist.root
-rw-r--r-- 1 dunegli fnalgrid       7384 Sep 16 20:01 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid       1174 Sep 16 20:01 000066_reco_data_2024-08-16T_161646Z_reco_data_2024-08-19T_112044Z_reco_data_2024-08-20T_114144Z_reco_data_2024-08-22T_091719Z_reco_data_2024-09-16T_152610Z.root.json
-rw-r--r-- 1 dunegli fnalgrid       1049 Sep 16 20:01 000066_reco_data_2024-08-16T_161646Z_reco_data_2024-08-19T_112044Z_reco_data_2024-08-20T_114144Z_reco_data_2024-08-22T_091719Z_reco_data_2024-09-16T_152610Z.root.ext.json
-rw-r--r-- 1 dunegli fnalgrid        284 Sep 16 15:26 all-input-dids.txt
-rw-r--r-- 1 dunegli fnalgrid        218 Sep 16 20:01 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid          0 Sep 16 15:26 debugprod.log
justIN time: 2024-09-29 13:27:15 UTC       justIN version: 01.01.08