justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 72765.147@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID72765.147@justin-prod-sched02.dune.hep.ac.uk
Workflow ID3262
Stage ID1
User nameamoor@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes5242880000 (5000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-09-16 13:51:43
SiteUS_UChicago
EntryEngage_US_MWT2_uct2_condce_mcore
Last heartbeat2024-09-16 17:00:43
From worker nodeHostnamemwt2-c039.campuscluster.illinois.edu
cpuinfoAMD EPYC 7443 24-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes5242880000 (5000 MiB)
Wall seconds limit86400 (24 hours)
Inner Apptainer?True
Job statefinished
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-09-16 14:20:02
Input filesusertests:000968_reco_data_2024-08-16T_163119Z_reco_data_2024-08-19T_110342Z_reco_data_2024-08-20T_125951Z_reco_data_2024-08-22T_093306Z.root
JobscriptExit code0
Real time2h (9607s)
CPU time2h (9564s = 99%)
Outputting started2024-09-16 17:00:11
Output filesusertests:000968_reco_data_2024-08-16T_163119Z_reco_data_2024-08-19T_110342Z_reco_data_2024-08-20T_125951Z_reco_data_2024-08-22T_093306Z_reco_data_2024-09-16T_142007Z.root
Finished2024-09-16 17:00:43
Saved logsjustin-logs:72765.147-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

AFM reco2 jobscript.
Input PFN = root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/08/d9/000968_reco_data_2024-08-16T_163119Z_reco_data_2024-08-19T_110342Z_reco_data_2024-08-20T_125951Z_reco_data_2024-08-22T_093306Z.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_4_3b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libXrdPosixPreload.so
=== Start last 50 lines of lar log file ===
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  16-Sep-2024 12:00:03 CDT run: 20000031 subRun: 0 event: 626
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  16-Sep-2024 12:00:03 CDT run: 20000031 subRun: 0 event: 626
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  16-Sep-2024 12:00:03 CDT run: 20000031 subRun: 0 event: 626
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  16-Sep-2024 12:00:03 CDT run: 20000031 subRun: 0 event: 626
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  16-Sep-2024 12:00:03 CDT run: 20000031 subRun: 0 event: 626
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
Begin processing the 11th record. run: 20000031 subRun: 0 event: 726 at 16-Sep-2024 12:00:04 CDT
16-Sep-2024 12:00:05 CDT  Closed output file "000968_reco_data_2024-08-16T_163119Z_reco_data_2024-08-19T_110342Z_reco_data_2024-08-20T_125951Z_reco_data_2024-08-22T_093306Z_reco_data_2024-09-16T_142007Z.root"
16-Sep-2024 12:00:05 CDT  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/08/d9/000968_reco_data_2024-08-16T_163119Z_reco_data_2024-08-19T_110342Z_reco_data_2024-08-20T_125951Z_reco_data_2024-08-22T_093306Z.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.0823199      867.106       9507.75      0.238768       2732.42        11     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00467947    0.00775118     0.0168917    0.00565304    0.00435652       11     
reco:linecluster:LineCluster                           0.027194       216.106       2373.49      0.0727121      682.226        11     
reco:trajcluster:TrajCluster                           0.0136221      204.17        2238.02      0.0412947      643.163        11     
reco:pandora:StandardPandora                          0.00079863      444.542       4873.75      0.017837       1400.64        11     
reco:pandoraTrack:LArPandoraTrackCreation             9.5831e-05     0.734015       7.3097      0.00264845      2.08878        11     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000142408    0.0434776     0.187017     0.00652098     0.0590288       11     
reco:pandoracalo:Calorimetry                          8.9619e-05     0.419367       4.24551     0.00151031      1.21309        11     
reco:pandorapid:Chi2ParticleID                        2.8363e-05    0.00083177    0.00458997    0.000145273    0.0013488       11     
reco:trkshowersplit:TrackShowerHits                   0.000113874    0.928899       9.92734     0.000685268     2.84655        11     
reco:ophit:OpHitFinder                                 0.0266824     0.0497604     0.0868613     0.0416631     0.0199826       11     
reco:opflash:OpFlashFinder                            0.000133722   0.000609083   0.00227826    0.000350519   0.000621054      11     
reco:rns:RandomNumberSaver                            1.7343e-05    3.67218e-05    0.0001702    2.0859e-05    4.25715e-05      11     
[art]:TriggerResults:TriggerResultInserter             8.315e-06    1.29907e-05   3.4935e-05     9.739e-06    7.62013e-06      11     
end_path:out1:RootOutput                               3.857e-06    5.88291e-06   1.7443e-05     4.388e-06    3.75967e-06      11     
end_path:out1:RootOutput(write)                       0.00229971      0.10227       0.72608     0.00840163      0.20675        11     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3484.16 MB
  Peak resident set size usage (VmHWM): 2340.5 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 50 lines of lar log file ===
lar exit code 0
extractor_prod.py exit code 0
pdjson2metadata exit code 0
Metadata extraction succeeds
===Metadata JSON===
{
    "metadata": {
        "DUNE.campaign": "justIN.w3262s1",
        "art.file_format_era": "ART_2011a",
        "art.file_format_version": 15,
        "art.first_event": 169,
        "art.last_event": 726,
        "art.process_name": "Reco2",
        "art.run_type": "physics",
        "core.application": "art.reco",
        "core.application.family": "art",
        "core.application.name": "reco",
        "core.application.version": "v09_75_03d00",
        "core.data_stream": "out1",
        "core.data_tier": "full-reconstructed",
        "core.end_time": 1726506005.0,
        "core.event_count": 11,
        "core.file_format": "artroot",
        "core.file_type": "mc",
        "core.first_event_number": 169,
        "core.last_event_number": 726,
        "core.run_type": "physics",
        "core.runs": [
            20000031
        ],
        "core.runs_subruns": [
            2000003100000
        ],
        "core.start_time": 1726496438.0
    },
    "parents": [
        {
            "did": "usertests:000968_reco_data_2024-08-16T_163119Z_reco_data_2024-08-19T_110342Z_reco_data_2024-08-20T_125951Z_reco_data_2024-08-22T_093306Z.root"
        }
    ]
}
===================
.:
total 1379924
-rw-r--r--. 1 dune osgvo 1412770003 Sep 16 12:00 000968_reco_data_2024-08-16T_163119Z_reco_data_2024-08-19T_110342Z_reco_data_2024-08-20T_125951Z_reco_data_2024-08-22T_093306Z_reco_data_2024-09-16T_142007Z.root
-rw-r--r--. 1 dune osgvo     230336 Sep 16 12:00 000968_reco_data_2024-08-16T_163119Z_reco_data_2024-08-19T_110342Z_reco_data_2024-08-20T_125951Z_reco_data_2024-08-22T_093306Z_reco_2024-09-16T_142007Z.log
-rw-r--r--. 1 dune osgvo       7560 Sep 16 12:00 reco2_hist.root
-rw-r--r--. 1 dune osgvo       7081 Sep 16 12:00 jobscript.log
-rw-r--r--. 1 dune osgvo       1174 Sep 16 12:00 000968_reco_data_2024-08-16T_163119Z_reco_data_2024-08-19T_110342Z_reco_data_2024-08-20T_125951Z_reco_data_2024-08-22T_093306Z_reco_data_2024-09-16T_142007Z.root.json
-rw-r--r--. 1 dune osgvo       1049 Sep 16 12:00 000968_reco_data_2024-08-16T_163119Z_reco_data_2024-08-19T_110342Z_reco_data_2024-08-20T_125951Z_reco_data_2024-08-22T_093306Z_reco_data_2024-09-16T_142007Z.root.ext.json
-rw-r--r--. 1 dune osgvo        284 Sep 16 09:20 all-input-dids.txt
-rw-r--r--. 1 dune osgvo        218 Sep 16 12:00 justin-processed-pfns.txt
-rw-r--r--. 1 dune osgvo          0 Sep 16 09:20 debugprod.log
justIN time: 2024-09-29 13:32:51 UTC       justIN version: 01.01.08