justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.

Jobsub ID 370993.1@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID370993.1@justin-prod-sched01.dune.hep.ac.uk
Workflow ID6304
Stage ID1
User nameamoor@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes5242880000 (5000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-04-18 06:49:20
SiteUS_UConn-HPC
EntryGLUEX_US_UConn-HPC_osgce
Last heartbeat2025-04-18 13:11:59
From worker nodeHostnamecn430
cpuinfoAMD EPYC 7452 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes5242880000 (5000 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-04-18 06:51:02
Input filesusertests:000200_reco_data_2025-04-02T_082441Z_reco_data_2025-04-02T_180416Z_reco_data_2025-04-04T_130336Z_reco_data_2025-04-07T_111210Z_reco_data_2025-04-08T_131316Z.root
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-04-18 07:02:14
Saved logsjustin-logs:370993.1-justin-prod-sched01.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

AFM ana jobscript.
Input PFN = root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/usertests/36/4d/000200_reco_data_2025-04-02T_082441Z_reco_data_2025-04-02T_180416Z_reco_data_2025-04-04T_130336Z_reco_data_2025-04-07T_111210Z_reco_data_2025-04-08T_131316Z.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/7659c97c6f688ef0efcd196837c575648fc28548
/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/7659c97c6f688ef0efcd196837c575648fc28548:.:./job:/cvmfs/dune.opensciencegrid.org/products/dune/dunesw/v10_04_06d00/fcl:/cvmfs/dune.opensciencegrid.org/products/dune/protoduneana/v10_04_06d00/job:/cvmfs/larsoft.opensciencegrid.org/products/geant4reweight/v01_20_08/fcl:/cvmfs/dune.opensciencegrid.org/products/dune/duneprototypes/v10_04_06d00/fcl:/cvmfs/dune.opensciencegrid.org/products/dune/dunesim/v10_04_06d00/fcl:/cvmfs/dune.opensciencegrid.org/products/dune/dunetrigger/v10_04_06d00/fcl:/cvmfs/dune.opensciencegrid.org/products/dune/dunecalib/v10_04_06d00/fcl:/cvmfs/dune.opensciencegrid.org/products/dune/dunedataprep/v10_04_06d00/fcl:/cvmfs/dune.opensciencegrid.org/products/dune/duneana/v10_04_06d00/fcl:/cvmfs/dune.opensciencegrid.org/products/dune/dunereco/v10_04_06d00/fcl:/cvmfs/dune.opensciencegrid.org/products/dune/duneopdet/v10_04_06d00/fcl:/cvmfs/dune.opensciencegrid.org/products/dune/dunecore/v10_04_06d00/fcl:/cvmfs/larsoft.opensciencegrid.org/products/nusystematics/v1_05_04/fcl:/cvmfs/larsoft.opensciencegrid.org/products/systematicstools/v01_04_04/fcl:.:./job:/cvmfs/larsoft.opensciencegrid.org/products/larwirecell/v10_01_06/fcl:/cvmfs/larsoft.opensciencegrid.org/products/larsimdnn/v10_00_08/job:/cvmfs/larsoft.opensciencegrid.org/products/larrecodnn/v10_01_06/job:/cvmfs/larsoft.opensciencegrid.org/products/larpandora/v10_00_15/job:/cvmfs/larsoft.opensciencegrid.org/products/larexamples/v10_00_09/job:/cvmfs/larsoft.opensciencegrid.org/products/lareventdisplay/v10_00_11/job:/cvmfs/larsoft.opensciencegrid.org/products/nuevdb/v1_09_08/fcl:/cvmfs/larsoft.opensciencegrid.org/products/larana/v10_00_12/job:/cvmfs/larsoft.opensciencegrid.org/products/larreco/v10_01_08/job:/cvmfs/larsoft.opensciencegrid.org/products/larsim/v10_02_02/job:/cvmfs/larsoft.opensciencegrid.org/products/nugen/v1_21_04/fcl:/cvmfs/larsoft.opensciencegrid.org/products/larg4/v10_00_06/fcl:/cvmfs/larsoft.opensciencegrid.org/products/nurandom/v1_11_05/fcl:/cvmfs/larsoft.opensciencegrid.org/products/nug4/v1_16_06/fcl:/cvmfs/larsoft.opensciencegrid.org/products/larevt/v10_00_06/job:/cvmfs/larsoft.opensciencegrid.org/products/lardata/v10_00_06/job:/cvmfs/larsoft.opensciencegrid.org/products/lardataalg/v10_00_04/job:/cvmfs/larsoft.opensciencegrid.org/products/larcore/v10_00_02/job:/cvmfs/larsoft.opensciencegrid.org/products/larcorealg/v10_00_02/job:/cvmfs/larsoft.opensciencegrid.org/products/artg4tk/v12_00_03/fcl:/cvmfs/dune.opensciencegrid.org/products/dune/dunepdlegacy/v1_01_04/fcl
=== Start last 200 lines of lar log file ===
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/dune.opensciencegrid.org/products/dune/dunecore/v10_04_06d00/gdml/dune10kt_v6_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Error: Unsupported GDML Tag Used :gdml_simple_extension. Please Check Geometry/Schema.
Error: Unsupported GDML Tag Used :extension. Please Check Geometry/Schema.
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 9704
Info in <TGeoManager::CloseGeometry>: 844009 nodes/ 12339 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
registering to  primaryGeneratorActionsMap_
registering to   eventActionsMap_
registering to   trackingActionsMap_
registering to  steppingActionsMap_
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Active Boundaries: 
	x: -746.028 to 746.028
	y: -607.829 to 607.829
	z: -0.87625 to 5808.87
18-Apr-2025 02:51:19 EDT  Initiating request to open input file "root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/usertests/36/4d/000200_reco_data_2025-04-02T_082441Z_reco_data_2025-04-02T_180416Z_reco_data_2025-04-04T_130336Z_reco_data_2025-04-07T_111210Z_reco_data_2025-04-08T_131316Z.root"
%MSG-s ArtException:  NeutrinoShowerEff:trajclustershweff@Construction  18-Apr-2025 03:02:11 EDT ModuleConstruction
cet::exception caught in art
---- FileOpenError BEGIN
  ---- FatalRootError BEGIN
    Fatal Root Error: TNetXNGFile::Open
    [FATAL] Socket timeout
    ROOT severity: 3000
  ---- FatalRootError END
  
  RootInputFileSequence::initFile(): Input file root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/usertests/36/4d/000200_reco_data_2025-04-02T_082441Z_reco_data_2025-04-02T_180416Z_reco_data_2025-04-04T_130336Z_reco_data_2025-04-07T_111210Z_reco_data_2025-04-08T_131316Z.root was not found or could not be opened.
---- FileOpenError END
%MSG
Art has completed and will exit with status 20.
=== End last 200 lines of lar log file ===
lar exit code 0
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_04_06d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_04_06d00/bin/extractor_prod.py", line 373, in main
    mddict = expSpecificMetadata.getmetadata()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_04_06d00/bin/extractor_prod.py", line 344, in getmetadata
    jobt = self.get_job(proc)
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_04_06d00/bin/extractor_prod.py", line 69, in get_job
    raise RuntimeError('sam_metadata_dumper returned nonzero exit status {}.'.format(rc))
RuntimeError: sam_metadata_dumper returned nonzero exit status 1.
extractor_prod.py exit code 1
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
pdjson2metadata exit code 1
.:
total 36
-rw-r--r-- 1 osgusers domain users 18761 Apr 18 03:02 Mergeana-20250418T070211_ana_hist.root
-rw-r--r-- 1 osgusers domain users  6837 Apr 18 03:02 jobscript.log
-rw-r--r-- 1 osgusers domain users  2672 Apr 18 03:02 000200_reco_data_2025-04-02T_082441Z_reco_data_2025-04-02T_180416Z_reco_data_2025-04-04T_130336Z_reco_data_2025-04-07T_111210Z_reco_data_2025-04-08T_131316Z_reco_2025-04-18T_065105Z.log
-rw-r--r-- 1 osgusers domain users   344 Apr 18 02:51 all-input-dids.txt
-rw-r--r-- 1 osgusers domain users     0 Apr 18 03:02 000200_reco_data_2025-04-02T_082441Z_reco_data_2025-04-02T_180416Z_reco_data_2025-04-04T_130336Z_reco_data_2025-04-07T_111210Z_reco_data_2025-04-08T_131316Z_reco_data_2025-04-18T_065105Z.root.ext.json
-rw-r--r-- 1 osgusers domain users     0 Apr 18 03:02 000200_reco_data_2025-04-02T_082441Z_reco_data_2025-04-02T_180416Z_reco_data_2025-04-04T_130336Z_reco_data_2025-04-07T_111210Z_reco_data_2025-04-08T_131316Z_reco_data_2025-04-18T_065105Z.root.json
-rw-r--r-- 1 osgusers domain users     0 Apr 18 02:51 debugprod.log
justIN time: 2025-08-21 21:38:37 UTC       justIN version: 01.03.02