Jobsub ID 257508.117@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	257508.117@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	3262
Stage ID	1
User name	amoor@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	5242880000 (5000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-09-16 13:59:43
Site	US_FNAL-T1
Entry	CMSHTPC_T1_US_FNAL_condce_opp1_whole
Last heartbeat	2024-09-16 14:20:00
From worker node	Hostname	dunegli-34037-0-cmswn2301.fnal.gov
	cpuinfo	Intel(R) Xeon(R) CPU E5-2670 v3 @ 2.30GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	5242880000 (5000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	aborted
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-09-16 14:13:41
Input files	usertests:000155_reco_data_2024-08-16T_161841Z_reco_data_2024-08-19T_110811Z_reco_data_2024-08-20T_110318Z_reco_data_2024-08-22T_092437Z.root
Outputting started	2024-09-16 14:17:15
Output files
Finished	2024-09-16 14:20:00
List job events Wrapper job log

Jobscript log (last 10,000 characters)

AFM reco2 jobscript.
Input PFN = root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/b8/ec/000155_reco_data_2024-08-16T_161841Z_reco_data_2024-08-19T_110811Z_reco_data_2024-08-20T_110318Z_reco_data_2024-08-22T_092437Z.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_4_3b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libXrdPosixPreload.so
=== Start last 50 lines of lar log file ===
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  16-Sep-2024 14:17:08 UTC run: 20000031 subRun: 0 event: 907
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  16-Sep-2024 14:17:08 UTC run: 20000031 subRun: 0 event: 907
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  16-Sep-2024 14:17:08 UTC run: 20000031 subRun: 0 event: 907
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  16-Sep-2024 14:17:08 UTC run: 20000031 subRun: 0 event: 907
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  16-Sep-2024 14:17:08 UTC run: 20000031 subRun: 0 event: 907
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  16-Sep-2024 14:17:08 UTC run: 20000031 subRun: 0 event: 907
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
16-Sep-2024 14:17:08 UTC  Closed output file "000155_reco_data_2024-08-16T_161841Z_reco_data_2024-08-19T_110811Z_reco_data_2024-08-20T_110318Z_reco_data_2024-08-22T_092437Z_reco_data_2024-09-16T_141345Z.root"
16-Sep-2024 14:17:08 UTC  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/b8/ec/000155_reco_data_2024-08-16T_161841Z_reco_data_2024-08-19T_110811Z_reco_data_2024-08-20T_110318Z_reco_data_2024-08-22T_092437Z.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.0164112      8.19531       88.5972      0.897977       21.6199        17     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00059912    0.00210906    0.00861427    0.00157585    0.00183324       17     
reco:linecluster:LineCluster                          0.00186451     0.272798       2.63802      0.0573792     0.623296        17     
reco:trajcluster:TrajCluster                          0.000910381     3.04679       31.9053      0.325461       7.93487        17     
reco:pandora:StandardPandora                          0.000696582     4.37793       51.3576      0.231747       12.4513        17     
reco:pandoraTrack:LArPandoraTrackCreation             0.000124894    0.157814       1.29391      0.0145602     0.324192        17     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000180505    0.111217      0.332021      0.0762488     0.109752        17     
reco:pandoracalo:Calorimetry                          0.000101347    0.0778805     0.527099      0.0241977     0.133764        17     
reco:pandorapid:Chi2ParticleID                        4.1044e-05    0.000721597   0.00340812    0.000215571   0.000950186      17     
reco:trkshowersplit:TrackShowerHits                   0.00016174     0.0498719     0.424711     0.00933216     0.104477        17     
reco:ophit:OpHitFinder                                0.00380753     0.0276658     0.0714486     0.0210606     0.0205413       17     
reco:opflash:OpFlashFinder                            0.000128864   0.00060232    0.00211105    0.000552041   0.000449514      17     
reco:rns:RandomNumberSaver                            1.6367e-05    3.12185e-05   0.000165926   2.1138e-05    3.40486e-05      17     
[art]:TriggerResults:TriggerResultInserter             9.975e-06    1.39465e-05   4.1001e-05    1.1349e-05    7.1224e-06       17     
end_path:out1:RootOutput                               3.028e-06    4.85676e-06   1.9009e-05     3.422e-06    3.68841e-06      17     
end_path:out1:RootOutput(write)                       0.00300901     0.0692246      0.29101      0.0464655     0.079362        17     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2201.92 MB
  Peak resident set size usage (VmHWM): 1100.57 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 50 lines of lar log file ===
lar exit code 0
extractor_prod.py exit code 0
pdjson2metadata exit code 0
Metadata extraction succeeds
===Metadata JSON===
{
    "metadata": {
        "DUNE.campaign": "justIN.w3262s1",
        "art.file_format_era": "ART_2011a",
        "art.file_format_version": 15,
        "art.first_event": 1,
        "art.last_event": 907,
        "art.process_name": "Reco2",
        "art.run_type": "physics",
        "core.application": "art.reco",
        "core.application.family": "art",
        "core.application.name": "reco",
        "core.application.version": "v09_75_03d00",
        "core.data_stream": "out1",
        "core.data_tier": "full-reconstructed",
        "core.end_time": 1726496228.0,
        "core.event_count": 17,
        "core.file_format": "artroot",
        "core.file_type": "mc",
        "core.first_event_number": 1,
        "core.last_event_number": 907,
        "core.run_type": "physics",
        "core.runs": [
            20000031
        ],
        "core.runs_subruns": [
            2000003100000
        ],
        "core.start_time": 1726496115.0
    },
    "parents": [
        {
            "did": "usertests:000155_reco_data_2024-08-16T_161841Z_reco_data_2024-08-19T_110811Z_reco_data_2024-08-20T_110318Z_reco_data_2024-08-22T_092437Z.root"
        }
    ]
}
===================
.:
total 315472
-rw-r--r-- 1 dunegli fnalgrid 322855748 Sep 16 14:17 000155_reco_data_2024-08-16T_161841Z_reco_data_2024-08-19T_110811Z_reco_data_2024-08-20T_110318Z_reco_data_2024-08-22T_092437Z_reco_data_2024-09-16T_141345Z.root
-rw-r--r-- 1 dunegli fnalgrid    149845 Sep 16 14:17 000155_reco_data_2024-08-16T_161841Z_reco_data_2024-08-19T_110811Z_reco_data_2024-08-20T_110318Z_reco_data_2024-08-22T_092437Z_reco_2024-09-16T_141345Z.log
-rw-r--r-- 1 dunegli fnalgrid      7560 Sep 16 14:17 reco2_hist.root
-rw-r--r-- 1 dunegli fnalgrid      7138 Sep 16 14:17 jobscript.log
-rw-r--r-- 1 dunegli fnalgrid      1170 Sep 16 14:17 000155_reco_data_2024-08-16T_161841Z_reco_data_2024-08-19T_110811Z_reco_data_2024-08-20T_110318Z_reco_data_2024-08-22T_092437Z_reco_data_2024-09-16T_141345Z.root.json
-rw-r--r-- 1 dunegli fnalgrid      1044 Sep 16 14:17 000155_reco_data_2024-08-16T_161841Z_reco_data_2024-08-19T_110811Z_reco_data_2024-08-20T_110318Z_reco_data_2024-08-22T_092437Z_reco_data_2024-09-16T_141345Z.root.ext.json
-rw-r--r-- 1 dunegli fnalgrid       284 Sep 16 14:13 all-input-dids.txt
-rw-r--r-- 1 dunegli fnalgrid       218 Sep 16 14:17 justin-processed-pfns.txt
-rw-r--r-- 1 dunegli fnalgrid         0 Sep 16 14:14 debugprod.log