justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 257499.90@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID257499.90@justin-prod-sched01.dune.hep.ac.uk
Workflow ID3262
Stage ID1
User nameamoor@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes5242880000 (5000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-09-16 13:39:28
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2024-09-16 17:06:35
From worker nodeHostnamedunegli-3944240-0-fnpc9041.fnal.gov
cpuinfoIntel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes5242880000 (5000 MiB)
Wall seconds limit172800 (48 hours)
Inner Apptainer?True
Job statefinished
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-09-16 13:48:04
Input filesusertests:000247_reco_data_2024-08-16T_162014Z_reco_data_2024-08-19T_111726Z_reco_data_2024-08-20T_120716Z_reco_data_2024-08-22T_092409Z.root
JobscriptExit code0
Real time3h (11881s)
CPU time3h (11666s = 98%)
Outputting started2024-09-16 17:06:06
Output filesusertests:000247_reco_data_2024-08-16T_162014Z_reco_data_2024-08-19T_111726Z_reco_data_2024-08-20T_120716Z_reco_data_2024-08-22T_092409Z_reco_data_2024-09-16T_134826Z.root
Finished2024-09-16 17:06:35
Saved logsjustin-logs:257499.90-justin-prod-sched01.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

AFM reco2 jobscript.
Input PFN = root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/a3/94/000247_reco_data_2024-08-16T_162014Z_reco_data_2024-08-19T_111726Z_reco_data_2024-08-20T_120716Z_reco_data_2024-08-22T_092409Z.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_4_3b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libXrdPosixPreload.so
=== Start last 50 lines of lar log file ===
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  16-Sep-2024 17:05:59 UTC run: 20000031 subRun: 0 event: 937
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  16-Sep-2024 17:05:59 UTC run: 20000031 subRun: 0 event: 937
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  16-Sep-2024 17:05:59 UTC run: 20000031 subRun: 0 event: 937
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  16-Sep-2024 17:05:59 UTC run: 20000031 subRun: 0 event: 937
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  16-Sep-2024 17:05:59 UTC run: 20000031 subRun: 0 event: 937
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  16-Sep-2024 17:05:59 UTC run: 20000031 subRun: 0 event: 937
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
16-Sep-2024 17:06:00 UTC  Closed output file "000247_reco_data_2024-08-16T_162014Z_reco_data_2024-08-19T_111726Z_reco_data_2024-08-20T_120716Z_reco_data_2024-08-22T_092409Z_reco_data_2024-09-16T_134826Z.root"
16-Sep-2024 17:06:00 UTC  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/a3/94/000247_reco_data_2024-08-16T_162014Z_reco_data_2024-08-19T_111726Z_reco_data_2024-08-20T_120716Z_reco_data_2024-08-22T_092409Z.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                            0.00984729      899.899       10969.7       1.68832       2911.14        13     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000550996    0.0036443     0.0179273    0.00119737    0.00590498       13     
reco:linecluster:LineCluster                          0.00141094      71.5348       909.138      0.0948869      241.813        13     
reco:trajcluster:TrajCluster                          0.000861417     84.8639       1043.71      0.700614       276.88         13     
reco:pandora:StandardPandora                          0.000687956     735.507       8924.07      0.438915       2368.95        13     
reco:pandoraTrack:LArPandoraTrackCreation             0.000114617     1.69056       16.8376      0.0863854      4.41187        13     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000163156    0.118124      0.427288      0.0965153     0.124735        13     
reco:pandoracalo:Calorimetry                          8.9726e-05     0.669899       6.61499      0.0584239      1.72953        13     
reco:pandorapid:Chi2ParticleID                        3.6678e-05    0.00176151    0.00774691    0.000814089   0.00214947       13     
reco:trkshowersplit:TrackShowerHits                   7.6678e-05     0.609498       6.63217      0.0280842      1.74482        13     
reco:ophit:OpHitFinder                                0.00260938      4.72568       61.2335      0.0126457      16.3124        13     
reco:opflash:OpFlashFinder                            0.000107724   0.000700314   0.00231046    0.000571931   0.000539875      13     
reco:rns:RandomNumberSaver                            1.5033e-05    3.29538e-05   0.000147544    2.186e-05    3.46792e-05      13     
[art]:TriggerResults:TriggerResultInserter             9.449e-06    1.36602e-05    4.284e-05    1.1481e-05    8.53329e-06      13     
end_path:out1:RootOutput                               2.731e-06    5.40731e-06   1.6611e-05     4.291e-06    3.58684e-06      13     
end_path:out1:RootOutput(write)                       0.00272586     0.172782       1.05466      0.0632987     0.271437        13     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3000.97 MB
  Peak resident set size usage (VmHWM): 1898.58 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 50 lines of lar log file ===
lar exit code 0
extractor_prod.py exit code 0
pdjson2metadata exit code 0
Metadata extraction succeeds
===Metadata JSON===
{
    "metadata": {
        "DUNE.campaign": "justIN.w3262s1",
        "art.file_format_era": "ART_2011a",
        "art.file_format_version": 15,
        "art.first_event": 77,
        "art.last_event": 937,
        "art.process_name": "Reco2",
        "art.run_type": "physics",
        "core.application": "art.reco",
        "core.application.family": "art",
        "core.application.name": "reco",
        "core.application.version": "v09_75_03d00",
        "core.data_stream": "out1",
        "core.data_tier": "full-reconstructed",
        "core.end_time": 1726506360.0,
        "core.event_count": 13,
        "core.file_format": "artroot",
        "core.file_type": "mc",
        "core.first_event_number": 77,
        "core.last_event_number": 937,
        "core.run_type": "physics",
        "core.runs": [
            20000031
        ],
        "core.runs_subruns": [
            2000003100000
        ],
        "core.start_time": 1726494667.0
    },
    "parents": [
        {
            "did": "usertests:000247_reco_data_2024-08-16T_162014Z_reco_data_2024-08-19T_111726Z_reco_data_2024-08-20T_120716Z_reco_data_2024-08-22T_092409Z.root"
        }
    ]
}
===================
.:
total 1324224
-rw-r--r-- 1 dunegli fnalgrid 1354207760 Sep 16 17:06 000247_reco_data_2024-08-16T_162014Z_reco_data_2024-08-19T_111726Z_reco_data_2024-08-20T_120716Z_reco_data_2024-08-22T_092409Z_reco_data_2024-09-16T_134826Z.root
-rw-r--r-- 1 dunegli fnalgrid     429238 Sep 16 17:06 000247_reco_data_2024-08-16T_162014Z_reco_data_2024-08-19T_111726Z_reco_data_2024-08-20T_120716Z_reco_data_2024-08-22T_092409Z_reco_2024-09-16T_134826Z.log
-rw-r--r-- 1 dunegli fnalgrid       7560 Sep 16 17:06 reco2_hist.root
-rw-r--r-- 1 dunegli fnalgrid       7140 Sep 16 17:06 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid       1172 Sep 16 17:06 000247_reco_data_2024-08-16T_162014Z_reco_data_2024-08-19T_111726Z_reco_data_2024-08-20T_120716Z_reco_data_2024-08-22T_092409Z_reco_data_2024-09-16T_134826Z.root.json
-rw-r--r-- 1 dunegli fnalgrid       1047 Sep 16 17:06 000247_reco_data_2024-08-16T_162014Z_reco_data_2024-08-19T_111726Z_reco_data_2024-08-20T_120716Z_reco_data_2024-08-22T_092409Z_reco_data_2024-09-16T_134826Z.root.ext.json
-rw-r--r-- 1 dunegli fnalgrid        284 Sep 16 13:48 all-input-dids.txt
-rw-r--r-- 1 dunegli fnalgrid        218 Sep 16 17:06 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid          0 Sep 16 13:49 debugprod.log
justIN time: 2024-09-29 13:19:31 UTC       justIN version: 01.01.08