justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 75222.0@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID75222.0@justin-prod-sched02.dune.hep.ac.uk
Workflow ID3316
Stage ID1
User nameamoor@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-09-20 09:52:10
SiteUS_NotreDame
EntryCMSHTPC_T3_US_NotreDame_deepthought
Last heartbeat2024-09-20 10:00:33
From worker nodeHostnamed12chas581.crc.nd.edu
cpuinfoIntel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
Inner Apptainer?True
Job stateaborted
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-09-20 09:55:23
Input filesusertests:000343_reco_data_2024-08-16T_162142Z_reco_data_2024-08-19T_110647Z_reco_data_2024-08-20T_095436Z_reco_data_2024-08-22T_091530Z_reco_data_2024-09-16T_135837Z.root
Outputting started2024-09-20 10:00:19
Output files
Finished2024-09-20 10:00:33
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

dunesw/v09_75_03d00/fcl:/cvmfs/dune.opensciencegrid.org/products/dune/protoduneana/v09_75_03d00/job:/cvmfs/larsoft.opensciencegrid.org/products/geant4reweight/v01_18_01/fcl:/cvmfs/dune.opensciencegrid.org/products/dune/duneprototypes/v09_75_03d00/fcl:/cvmfs/dune.opensciencegrid.org/products/dune/dunesim/v09_75_03d00/fcl:/cvmfs/dune.opensciencegrid.org/products/dune/dunecalib/v09_75_03d00/fcl:/cvmfs/dune.opensciencegrid.org/products/dune/dunedataprep/v09_75_03d00/fcl:/cvmfs/dune.opensciencegrid.org/products/dune/duneana/v09_75_03d00/fcl:/cvmfs/dune.opensciencegrid.org/products/dune/dunereco/v09_75_03d00/fcl:/cvmfs/dune.opensciencegrid.org/products/dune/duneopdet/v09_75_03d00/fcl:/cvmfs/dune.opensciencegrid.org/products/dune/dunecore/v09_75_03d00/fcl:/cvmfs/larsoft.opensciencegrid.org/products/nusystematics/v01_03_09/fcl:/cvmfs/larsoft.opensciencegrid.org/products/systematicstools/v01_03_01/fcl:.:./job:/cvmfs/larsoft.opensciencegrid.org/products/larwirecell/v09_14_01/fcl:/cvmfs/larsoft.opensciencegrid.org/products/larsimdnn/v09_05_02/job:/cvmfs/larsoft.opensciencegrid.org/products/larrecodnn/v09_21_03/job:/cvmfs/larsoft.opensciencegrid.org/products/larpandora/v09_21_02/job:/cvmfs/larsoft.opensciencegrid.org/products/larexamples/v09_08_02/job:/cvmfs/larsoft.opensciencegrid.org/products/lareventdisplay/v09_10_02/job:/cvmfs/larsoft.opensciencegrid.org/products/nuevdb/v1_08_01/fcl:/cvmfs/larsoft.opensciencegrid.org/products/larana/v09_14_02/job:/cvmfs/larsoft.opensciencegrid.org/products/larreco/v09_20_02/job:/cvmfs/larsoft.opensciencegrid.org/products/larsim/v09_36_02/job:/cvmfs/larsoft.opensciencegrid.org/products/nugen/v1_19_04/fcl:/cvmfs/larsoft.opensciencegrid.org/products/larg4/v09_16_01/fcl:/cvmfs/larsoft.opensciencegrid.org/products/nurandom/v1_10_01/fcl:/cvmfs/larsoft.opensciencegrid.org/products/nug4/v1_15_01/fcl:/cvmfs/larsoft.opensciencegrid.org/products/larevt/v09_09_01/job:/cvmfs/larsoft.opensciencegrid.org/products/lardata/v09_15_01/job:/cvmfs/larsoft.opensciencegrid.org/products/lardataalg/v09_15_01/job:/cvmfs/larsoft.opensciencegrid.org/products/larcore/v09_10_00/job:/cvmfs/larsoft.opensciencegrid.org/products/larcorealg/v09_12_00/job:/cvmfs/larsoft.opensciencegrid.org/products/artg4tk/v11_00_01/fcl:/cvmfs/dune.opensciencegrid.org/products/dune/dunepdlegacy/v1_00_03/fcl
=== Start last 50 lines of lar log file ===
Warning: there was no shower found for shower-like PFParticle with ID 238
Warning: there was no shower found for shower-like PFParticle with ID 239
Warning: there was no shower found for shower-like PFParticle with ID 240
Warning: there was no shower found for shower-like PFParticle with ID 241
Warning: there was no shower found for shower-like PFParticle with ID 242
Warning: there was no shower found for shower-like PFParticle with ID 243
Warning: there was no shower found for shower-like PFParticle with ID 244
Warning: there was no shower found for shower-like PFParticle with ID 245
Warning: there was no shower found for shower-like PFParticle with ID 246
Warning: there was no shower found for shower-like PFParticle with ID 247
Warning: there was no shower found for shower-like PFParticle with ID 248
Warning: there was no shower found for shower-like PFParticle with ID 249
Warning: there was no shower found for shower-like PFParticle with ID 250
Warning: there was no shower found for shower-like PFParticle with ID 251
Warning: there was no shower found for shower-like PFParticle with ID 252
Warning: there was no shower found for shower-like PFParticle with ID 253
Warning: there was no shower found for shower-like PFParticle with ID 254
Warning: there was no shower found for shower-like PFParticle with ID 255
Warning: there was no shower found for shower-like PFParticle with ID 256
20-Sep-2024 06:00:10 EDT  Closed output file "000343_reco_data_2024-08-16T_162142Z_reco_data_2024-08-19T_110647Z_reco_data_2024-08-20T_095436Z_reco_data_2024-08-22T_091530Z_reco_data_2024-09-16T_135837Z_reco_data_2024-09-20T_095605Z.root"
20-Sep-2024 06:00:10 EDT  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/43/b2/000343_reco_data_2024-08-16T_162142Z_reco_data_2024-08-19T_110647Z_reco_data_2024-08-20T_095436Z_reco_data_2024-08-22T_091530Z_reco_data_2024-09-16T_135837Z.root"

==================================================================================================================================
TimeTracker printout (sec)                          Min           Avg           Max         Median          RMS         nEvts   
==================================================================================================================================
Full event                                        5.12107       11.8577       18.5942       11.8577       6.73658         2     
----------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                           0.0121135     0.0122937     0.0124739     0.0122937    0.000180217       2     
end_path:out1:RootOutput                        1.0088e-05    3.4205e-05    5.8322e-05    3.4205e-05    2.4117e-05        2     
end_path:analysistree:AnalysisTree               0.981333       5.63077       10.2802       5.63077       4.64943         2     
end_path:pmtrackeff:NeutrinoTrackingEff         0.000456627   0.000474767   0.000492907   0.000474767    1.814e-05        2     
end_path:pmtracktceff:NeutrinoTrackingEff       3.4905e-05    8.64885e-05   0.000138072   8.64885e-05   5.15835e-05       2     
end_path:pandoraeff:NeutrinoTrackingEff         2.8211e-05    7.36405e-05   0.00011907    7.36405e-05   4.54295e-05       2     
end_path:pmtrajfiteff:NeutrinoTrackingEff       2.8677e-05    7.88425e-05   0.000129008   7.88425e-05   5.01655e-05       2     
end_path:pmtrajfittceff:NeutrinoTrackingEff     2.7498e-05    7.06605e-05   0.000113823   7.06605e-05   4.31625e-05       2     
end_path:pandorashweff:NeutrinoShowerEff        0.000370478   0.000562019   0.00075356    0.000562019   0.000191541       2     
end_path:emshowershweff:NeutrinoShowerEff       0.000199221   0.000283577   0.000367934   0.000283577   8.43565e-05       2     
end_path:trajclustershweff:NeutrinoShowerEff    0.00014018    0.000246352   0.000352525   0.000246352   0.000106172       2     
end_path:runcheckereco:CheckRecoEnergy          2.0232e-05    0.000136612   0.000252993   0.000136612   0.000116381       2     
end_path:flashmatchAna:FlashMatchAna             0.117026      0.336735      0.556444      0.336735      0.219709         2     
end_path:out1:RootOutput(write)                   4.00786       5.87516       7.74247       5.87516       1.8673          2     
==================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2839.41 MB
  Peak resident set size usage (VmHWM): 1682.08 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 50 lines of lar log file ===
lar exit code 0
extractor_prod.py exit code 0
pdjson2metadata exit code 0
Metadata extraction succeeds
===Metadata JSON===
{
    "metadata": {
        "DUNE.campaign": "justIN.w3316s1",
        "art.file_format_era": "ART_2011a",
        "art.file_format_version": 15,
        "art.first_event": 4,
        "art.last_event": 128,
        "art.process_name": "Mergeana",
        "art.run_type": "physics",
        "core.application": "art.reco",
        "core.application.family": "art",
        "core.application.name": "reco",
        "core.application.version": "v09_75_03d00",
        "core.data_stream": "out1",
        "core.data_tier": "full-reconstructed",
        "core.end_time": 1726826409.0,
        "core.event_count": 2,
        "core.file_format": "artroot",
        "core.file_type": "mc",
        "core.first_event_number": 4,
        "core.last_event_number": 128,
        "core.run_type": "physics",
        "core.runs": [
            20000031
        ],
        "core.runs_subruns": [
            2000003100000
        ],
        "core.start_time": 1726826385.0
    },
    "parents": [
        {
            "did": "usertests:000343_reco_data_2024-08-16T_162142Z_reco_data_2024-08-19T_110647Z_reco_data_2024-08-20T_095436Z_reco_data_2024-08-22T_091530Z_reco_data_2024-09-16T_135837Z.root"
        }
    ]
}
===================
.:
total 86696
-rw-r--r-- 1 dune osgusers 86693729 Sep 20 06:00 000343_reco_data_2024-08-16T_162142Z_reco_data_2024-08-19T_110647Z_reco_data_2024-08-20T_095436Z_reco_data_2024-08-22T_091530Z_reco_data_2024-09-16T_135837Z_reco_data_2024-09-20T_095605Z.root
-rw-r--r-- 1 dune osgusers  2020774 Sep 20 06:00 ana_hist.root
-rw-r--r-- 1 dune osgusers    28644 Sep 20 06:00 000343_reco_data_2024-08-16T_162142Z_reco_data_2024-08-19T_110647Z_reco_data_2024-08-20T_095436Z_reco_data_2024-08-22T_091530Z_reco_data_2024-09-16T_135837Z_reco_2024-09-20T_095605Z.log
-rw-r--r-- 1 dune osgusers     9451 Sep 20 06:00 jobscript.log
-rw-r--r-- 1 dune osgusers     1202 Sep 20 06:00 000343_reco_data_2024-08-16T_162142Z_reco_data_2024-08-19T_110647Z_reco_data_2024-08-20T_095436Z_reco_data_2024-08-22T_091530Z_reco_data_2024-09-16T_135837Z_reco_data_2024-09-20T_095605Z.root.json
-rw-r--r-- 1 dune osgusers     1105 Sep 20 06:00 000343_reco_data_2024-08-16T_162142Z_reco_data_2024-08-19T_110647Z_reco_data_2024-08-20T_095436Z_reco_data_2024-08-22T_091530Z_reco_data_2024-09-16T_135837Z_reco_data_2024-09-20T_095605Z.root.ext.json
-rw-r--r-- 1 dune osgusers      344 Sep 20 05:55 all-input-dids.txt
-rw-r--r-- 1 dune osgusers      248 Sep 20 06:00 justin-processed-pfns.txt
-rw-r--r-- 1 dune osgusers        0 Sep 20 05:56 debugprod.log
justIN time: 2024-11-17 04:02:17 UTC       justIN version: 01.01.09