justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 390976.81@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID390976.81@justin-prod-sched01.dune.hep.ac.uk
Workflow ID6613
Stage ID1
User namehiguera@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-05-02 22:27:13
SiteUK_Liverpool
EntryDUNE_UK_Liverpool_hepgrid5
Last heartbeat2025-05-02 22:53:46
From worker nodeHostnamer24-n06.ph.liv.ac.uk
cpuinfoIntel(R) Xeon(R) CPU E5620 @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit259200 (72 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-05-02 22:52:37
Input fileshd-protodune:np04hd_raw_run029425_0070_dataflow1_datawriter_0_20241005T230613.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-05-02 22:53:46
Saved logsjustin-logs:390976.81-justin-prod-sched01.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

cvmfs/dune.opensciencegrid.org/products/dune -q e26:prof
DUNEEXAMPLES_FQ_DIR=/cvmfs/dune.opensciencegrid.org/products/dune/duneexamples/v10_05_00d00/slf7.x86_64.e26.prof
DUNEDATAPREP_VERSION=v10_05_00d00
DUNECORE_INC=/cvmfs/dune.opensciencegrid.org/products/dune/dunecore/v10_05_00d00/include
DUNEPDLEGACY_INC=/cvmfs/dune.opensciencegrid.org/products/dune/dunepdlegacy/v1_01_04/include
SETUP_DUNEANA=duneana v10_05_00d00 -f Linux64bit+3.10-2.17 -z /cvmfs/dune.opensciencegrid.org/products/dune -q e26:prof
DUNEPROTOTYPES_LIB=/cvmfs/dune.opensciencegrid.org/products/dune/duneprototypes/v10_05_00d00/slf7.x86_64.e26.prof/lib
DUNEDETDATAFORMATS_VERSION=v4_4_5
DUNEUTIL_VERSION=v10_05_00d00
DUNESW_LIB=/cvmfs/dune.opensciencegrid.org/products/dune/dunesw/v10_05_00d00/slf7.x86_64.e26.prof/lib
SETUP_PROTODUNEANA=protoduneana v10_05_00d00 -f Linux64bit+3.10-2.17 -z /cvmfs/dune.opensciencegrid.org/products/dune -q e26:prof
SETUP_DUNESIM=dunesim v10_05_00d00 -f Linux64bit+3.10-2.17 -z /cvmfs/dune.opensciencegrid.org/products/dune -q e26:prof
PROTODUNEANA_FQ_DIR=/cvmfs/dune.opensciencegrid.org/products/dune/protoduneana/v10_05_00d00/slf7.x86_64.e26.prof
DUNESIM_LIB=/cvmfs/dune.opensciencegrid.org/products/dune/dunesim/v10_05_00d00/slf7.x86_64.e26.prof/lib
SETUP_DUNEOPDET=duneopdet v10_05_00d00 -f Linux64bit+3.10-2.17 -z /cvmfs/dune.opensciencegrid.org/products/dune -q e26:prof
DUNESIM_FQ_DIR=/cvmfs/dune.opensciencegrid.org/products/dune/dunesim/v10_05_00d00/slf7.x86_64.e26.prof
SETUP_DUNEPDLEGACY=dunepdlegacy v1_01_04 -f Linux64bit+3.10-2.17 -z /cvmfs/dune.opensciencegrid.org/products/dune -q e26:prof
DUNECALIB_VERSION=v10_05_00d00
SETUP_DUNEDAQDATAFORMATS=dunedaqdataformats v4_4_5 -f NULL -z /cvmfs/dune.opensciencegrid.org/products/dune
DUNEEXAMPLES_DIR=/cvmfs/dune.opensciencegrid.org/products/dune/duneexamples/v10_05_00d00
DUNECORE_LIB=/cvmfs/dune.opensciencegrid.org/products/dune/dunecore/v10_05_00d00/slf7.x86_64.e26.prof/lib
DUNEPDLEGACY_LIB=/cvmfs/dune.opensciencegrid.org/products/dune/dunepdlegacy/v1_01_04/slf7.x86_64.e26.prof/lib
SETUP_DUNETRIGGER=dunetrigger v10_05_00d00 -f Linux64bit+3.10-2.17 -z /cvmfs/dune.opensciencegrid.org/products/dune -q e26:prof
DUNEANA_LIB=/cvmfs/dune.opensciencegrid.org/products/dune/duneana/v10_05_00d00/slf7.x86_64.e26.prof/lib
SETUP_DUNECORE=dunecore v10_05_00d00 -f Linux64bit+3.10-2.17 -z /cvmfs/dune.opensciencegrid.org/products/dune -q e26:prof
SETUP_DUNEANAOBJ=duneanaobj v03_05_00 -f Linux64bit+3.10-2.17 -z /cvmfs/dune.opensciencegrid.org/products/dune -q e26:prof
DUNETRIGGER_LIB=/cvmfs/dune.opensciencegrid.org/products/dune/dunetrigger/v10_05_00d00/slf7.x86_64.e26.prof/lib
DUNESIM_VERSION=v10_05_00d00
DUNECALIB_INC=/cvmfs/dune.opensciencegrid.org/products/dune/dunecalib/v10_05_00d00/include
DUNEOPDET_LIB=/cvmfs/dune.opensciencegrid.org/products/dune/duneopdet/v10_05_00d00/slf7.x86_64.e26.prof/lib
DUNEUTIL_FQ_DIR=/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v10_05_00d00/slf7.x86_64.e26.prof
DUNEANAOBJ_LIB=/cvmfs/dune.opensciencegrid.org/products/dune/duneanaobj/v03_05_00/slf7.x86_64.e26.prof/lib
DUNEPROTOTYPES_FQ_DIR=/cvmfs/dune.opensciencegrid.org/products/dune/duneprototypes/v10_05_00d00/slf7.x86_64.e26.prof
PROTODUNEANA_LIB=/cvmfs/dune.opensciencegrid.org/products/dune/protoduneana/v10_05_00d00/slf7.x86_64.e26.prof/lib
DUNEPROTOTYPES_INC=/cvmfs/dune.opensciencegrid.org/products/dune/duneprototypes/v10_05_00d00/include
DUNEEXAMPLES_LIB=/cvmfs/dune.opensciencegrid.org/products/dune/duneexamples/v10_05_00d00/slf7.x86_64.e26.prof/lib
SETUP_DUNECALIB=dunecalib v10_05_00d00 -f Linux64bit+3.10-2.17 -z /cvmfs/dune.opensciencegrid.org/products/dune -q e26:prof
DUNERECO_VERSION=v10_05_00d00
DUNEDETDATAFORMATS_INC=/cvmfs/dune.opensciencegrid.org/products/dune/dunedetdataformats/v4_4_5/include
SETUP_DUNESW=dunesw v10_05_00d00 -f Linux64bit+3.10-2.17 -z /cvmfs/dune.opensciencegrid.org/products/dune -q e26:prof
PROTODUNEANA_INC=/cvmfs/dune.opensciencegrid.org/products/dune/protoduneana/v10_05_00d00/include
DUNEDATAPREP_LIB=/cvmfs/dune.opensciencegrid.org/products/dune/dunedataprep/v10_05_00d00/slf7.x86_64.e26.prof/lib
DUNERECO_DIR=/cvmfs/dune.opensciencegrid.org/products/dune/dunereco/v10_05_00d00
Installing yaml
Collecting pyyaml
  Downloading PyYAML-6.0.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (737 kB)
Installing collected packages: pyyaml
Successfully installed pyyaml-6.0.2
WARNING: You are using pip version 21.2.4; however, version 25.1.1 is available.
You should consider upgrading via the '/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python3 -m pip install --upgrade pip' command.
Done
Will use justin-get-file
input file: np04hd_raw_run029425_0070_dataflow1_datawriter_0_20241005T230613.hdf5
jobsub_id: 390976.81@justin-prod-sched01.dune.hep.ac.uk
Getting run subrun from hd-protodune:np04hd_raw_run029425_0070_dataflow1_datawriter_0_20241005T230613.hdf5
29425 2942500001
Found sps data for run 29425: /cvmfs/fifeuser3.opensciencegrid.org/sw/dune/fa49d1d4aaf0d5ceb75154614adc48877f735a0c/spillrun029425.csv
nevents: -1
input flag: -i root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/hd-protodune/56/61/np04hd_raw_run029425_0070_dataflow1_datawriter_0_20241005T230613.hdf5 --parent hd-protodune:np04hd_raw_run029425_0070_dataflow1_datawriter_0_20241005T230613.hdf5
reco
Running reco
['lar', '-c', 'run_np04data_processor_run29425.fcl', 'root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/hd-protodune/56/61/np04hd_raw_run029425_0070_dataflow1_datawriter_0_20241005T230613.hdf5', '-n', '-1', '--nskip', '0', '-o', 'proc_bsmtrigger_protodunehd_1_390976_81_1746226366_reco.root']
Stdout:
DAPHNE Channel Map: Building DAPHNE channel map from file DAPHNE_test5_ChannelMap_v1.txt
PD2HD Channel Map: Building TPC wiremap from file PD2HDChannelMap_WIBEth_electronics_v1.txt
Art has completed and will exit with status 1.

Stderr:
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  crtreco
  crttag
  emtrkmichelid
  filterspilloff
  filterspillon
  nhitsfilter
  opdec
  opflash
  ophitspe
  opslicer
  pandoraGnocchiCalo
  pandoraGnocchiCalonosce
  pdhddaphne
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/dune.opensciencegrid.org/products/dune/dunecore/v10_05_00d00/gdml/protodunehd_v6_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 1148
Info in <TGeoManager::CloseGeometry>: 25610 nodes/ 5537 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
SysError in <TFile::TFile>: file /cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash/ProtoDUNE/HD/SpaceCharge/Sim_NoFluidFlow/v2/SCE_PDHD_Sim_NoFluidFlow_E500_v2.root can not be opened for reading Input/output error
%MSG-s ArtException:  Early 02-May-2025 23:53:29 BST JobSetup
cet::exception caught in art
---- OtherArt BEGIN
  ServiceCreation
  ---- SpaceChargeProtoDUNEhd BEGIN
    Could not find the space charge effect file '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash/ProtoDUNE/HD/SpaceCharge/Sim_NoFluidFlow/v2/SCE_PDHD_Sim_NoFluidFlow_E500_v2.root'!
  ---- SpaceChargeProtoDUNEhd END
  cet::exception caught during construction of service type spacecharge::SpaceChargeServiceProtoDUNEhd
---- OtherArt END
%MSG

Error in processing reco
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  crtreco
  crttag
  emtrkmichelid
  filterspilloff
  filterspillon
  nhitsfilter
  opdec
  opflash
  ophitspe
  opslicer
  pandoraGnocchiCalo
  pandoraGnocchiCalonosce
  pdhddaphne
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/dune.opensciencegrid.org/products/dune/dunecore/v10_05_00d00/gdml/protodunehd_v6_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 1148
Info in <TGeoManager::CloseGeometry>: 25610 nodes/ 5537 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
SysError in <TFile::TFile>: file /cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash/ProtoDUNE/HD/SpaceCharge/Sim_NoFluidFlow/v2/SCE_PDHD_Sim_NoFluidFlow_E500_v2.root can not be opened for reading Input/output error
%MSG-s ArtException:  Early 02-May-2025 23:53:29 BST JobSetup
cet::exception caught in art
---- OtherArt BEGIN
  ServiceCreation
  ---- SpaceChargeProtoDUNEhd BEGIN
    Could not find the space charge effect file '/cvmfs/dune.osgstorage.org/pnfs/fnal.gov/usr/dune/persistent/stash/ProtoDUNE/HD/SpaceCharge/Sim_NoFluidFlow/v2/SCE_PDHD_Sim_NoFluidFlow_E500_v2.root'!
  ---- SpaceChargeProtoDUNEhd END
  cet::exception caught during construction of service type spacecharge::SpaceChargeServiceProtoDUNEhd
---- OtherArt END
%MSG

Error running. Exiting with 1
justIN time: 2025-05-22 22:36:28 UTC       justIN version: 01.03.01