justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 257502.2@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID257502.2@justin-prod-sched01.dune.hep.ac.uk
Workflow ID3262
Stage ID1
User nameamoor@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes5242880000 (5000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-09-16 13:42:58
SiteUS_FNAL-T1
EntryCMSHTPC_T1_US_FNAL_condce_opp1_whole
Last heartbeat2024-09-17 12:17:31
From worker nodeHostnamedunegli-34037-0-cmswn2301.fnal.gov
cpuinfoIntel(R) Xeon(R) CPU E5-2670 v3 @ 2.30GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes5242880000 (5000 MiB)
Wall seconds limit171000 (47 hours)
Inner Apptainer?True
Job statefinished
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-09-16 13:54:29
Input filesusertests:000046_reco_data_2024-08-16T_161627Z_reco_data_2024-08-19T_112402Z_reco_data_2024-08-20T_111359Z_reco_data_2024-08-22T_093129Z.root
JobscriptExit code0
Real time22h (80534s)
CPU time22h (80146s = 99%)
Outputting started2024-09-17 12:16:46
Output filesusertests:000046_reco_data_2024-08-16T_161627Z_reco_data_2024-08-19T_112402Z_reco_data_2024-08-20T_111359Z_reco_data_2024-08-22T_093129Z_reco_data_2024-09-16T_135446Z.root
Finished2024-09-17 12:17:31
Saved logsjustin-logs:257502.2-justin-prod-sched01.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

AFM reco2 jobscript.
Input PFN = root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/57/66/000046_reco_data_2024-08-16T_161627Z_reco_data_2024-08-19T_112402Z_reco_data_2024-08-20T_111359Z_reco_data_2024-08-22T_093129Z.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_4_3b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libXrdPosixPreload.so
=== Start last 50 lines of lar log file ===
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s120a-e20/label1/swarm/label2/SLF7/build/larpandoracontent/v04_04_02-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 271
this->GetVolumeIdToHitListMap(volumeIdToHitListMap) return STATUS_CODE_NOT_INITIALIZED
    in function: Run
    in file:     /scratch/workspace/build-larsoft/BUILDTYPE/prof/QUAL/s120a-e20/label1/swarm/label2/SLF7/build/larpandoracontent/v04_04_02-buildFW/src/larpandoracontent/LArControlFlow/MasterAlgorithm.cc line#: 165
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s120a-e20/label1/swarm/label2/SLF7/build/pandora/v03_16_00h/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0003, LArDLMaster, STATUS_CODE_NOT_INITIALIZED
Begin processing the 16th record. run: 20000031 subRun: 0 event: 970 at 17-Sep-2024 12:16:38 UTC
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  17-Sep-2024 12:16:38 UTC run: 20000031 subRun: 0 event: 970
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  17-Sep-2024 12:16:38 UTC run: 20000031 subRun: 0 event: 970
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  17-Sep-2024 12:16:38 UTC run: 20000031 subRun: 0 event: 970
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
17-Sep-2024 12:16:39 UTC  Closed output file "000046_reco_data_2024-08-16T_161627Z_reco_data_2024-08-19T_112402Z_reco_data_2024-08-20T_111359Z_reco_data_2024-08-22T_093129Z_reco_data_2024-09-16T_135446Z.root"
17-Sep-2024 12:16:39 UTC  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/57/66/000046_reco_data_2024-08-16T_161627Z_reco_data_2024-08-19T_112402Z_reco_data_2024-08-20T_111359Z_reco_data_2024-08-22T_093129Z.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.0132947      5024.04       80362.7       0.62363       19452.3        16     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000536269   0.00943735      0.11501     0.00120171     0.0275303       16     
reco:linecluster:LineCluster                          0.00204303      113.739       1818.76      0.0420796      440.235        16     
reco:trajcluster:TrajCluster                          0.00106546      114.013       1813.39      0.238813       438.778        16     
reco:pandora:StandardPandora                          0.000897758     4788.59        76612       0.102577       18544.7        16     
reco:pandoraTrack:LArPandoraTrackCreation             9.0717e-05      1.47586       22.4554      0.012797       5.41778        16     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000123044    0.107313      0.435551      0.0475451     0.128831        16     
reco:pandoracalo:Calorimetry                           7.372e-05     0.628842       9.42287      0.0126718      2.2712         16     
reco:pandorapid:Chi2ParticleID                        2.8895e-05    0.00124458     0.0115711    0.000253864   0.00271597       16     
reco:trkshowersplit:TrackShowerHits                   7.0561e-05     0.670744       10.4637     0.00583468      2.52867        16     
reco:ophit:OpHitFinder                                0.00398132      4.68349       74.4425      0.0116953      18.0119        16     
reco:opflash:OpFlashFinder                             8.992e-05    0.000603527   0.00254291    0.000428879   0.000581101      16     
reco:rns:RandomNumberSaver                            1.0724e-05    3.42244e-05   0.000221176   1.7288e-05    4.92166e-05      16     
[art]:TriggerResults:TriggerResultInserter             7.111e-06    1.27161e-05   5.8671e-05    9.3655e-06    1.19233e-05      16     
end_path:out1:RootOutput                               1.355e-06    5.59506e-06   2.3571e-05    3.3735e-06    5.34409e-06      16     
end_path:out1:RootOutput(write)                       0.00274394      0.1191        1.25279      0.0279088     0.295078        16     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3409.61 MB
  Peak resident set size usage (VmHWM): 2311.76 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 50 lines of lar log file ===
lar exit code 0
extractor_prod.py exit code 0
pdjson2metadata exit code 0
Metadata extraction succeeds
===Metadata JSON===
{
    "metadata": {
        "DUNE.campaign": "justIN.w3262s1",
        "art.file_format_era": "ART_2011a",
        "art.file_format_version": 15,
        "art.first_event": 2,
        "art.last_event": 970,
        "art.process_name": "Reco2",
        "art.run_type": "physics",
        "core.application": "art.reco",
        "core.application.family": "art",
        "core.application.name": "reco",
        "core.application.version": "v09_75_03d00",
        "core.data_stream": "out1",
        "core.data_tier": "full-reconstructed",
        "core.end_time": 1726575399.0,
        "core.event_count": 16,
        "core.file_format": "artroot",
        "core.file_type": "mc",
        "core.first_event_number": 2,
        "core.last_event_number": 970,
        "core.run_type": "physics",
        "core.runs": [
            20000031
        ],
        "core.runs_subruns": [
            2000003100000
        ],
        "core.start_time": 1726494944.0
    },
    "parents": [
        {
            "did": "usertests:000046_reco_data_2024-08-16T_161627Z_reco_data_2024-08-19T_112402Z_reco_data_2024-08-20T_111359Z_reco_data_2024-08-22T_093129Z.root"
        }
    ]
}
===================
.:
total 1251876
-rw-r--r-- 1 dunegli fnalgrid 1281553236 Sep 17 12:16 000046_reco_data_2024-08-16T_161627Z_reco_data_2024-08-19T_112402Z_reco_data_2024-08-20T_111359Z_reco_data_2024-08-22T_093129Z_reco_data_2024-09-16T_135446Z.root
-rw-r--r-- 1 dunegli fnalgrid     331214 Sep 17 12:16 000046_reco_data_2024-08-16T_161627Z_reco_data_2024-08-19T_112402Z_reco_data_2024-08-20T_111359Z_reco_data_2024-08-22T_093129Z_reco_2024-09-16T_135446Z.log
-rw-r--r-- 1 dunegli fnalgrid       7560 Sep 17 12:16 reco2_hist.root
-rw-r--r-- 1 dunegli fnalgrid       7390 Sep 17 12:16 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid       1170 Sep 17 12:16 000046_reco_data_2024-08-16T_161627Z_reco_data_2024-08-19T_112402Z_reco_data_2024-08-20T_111359Z_reco_data_2024-08-22T_093129Z_reco_data_2024-09-16T_135446Z.root.json
-rw-r--r-- 1 dunegli fnalgrid       1045 Sep 17 12:16 000046_reco_data_2024-08-16T_161627Z_reco_data_2024-08-19T_112402Z_reco_data_2024-08-20T_111359Z_reco_data_2024-08-22T_093129Z_reco_data_2024-09-16T_135446Z.root.ext.json
-rw-r--r-- 1 dunegli fnalgrid        284 Sep 16 13:54 all-input-dids.txt
-rw-r--r-- 1 dunegli fnalgrid        218 Sep 17 12:16 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid          0 Sep 16 13:55 debugprod.log
justIN time: 2024-09-29 13:21:55 UTC       justIN version: 01.01.08