justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 306539.189@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID306539.189@justin-prod-sched01.dune.hep.ac.uk
Workflow ID4348
Stage ID1
User nameamoor@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes5242880000 (5000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-12-03 09:06:43
SiteUK_Brunel
EntryCMSHTPC_T2_UK_London_Brunel_dc2_22
Last heartbeat2024-12-03 09:26:27
From worker nodeHostnamewn-b8-29-00.brunel.ac.uk
cpuinfoIntel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes5242880000 (5000 MiB)
Wall seconds limit171000 (47 hours)
Inner Apptainer?True
Job statejobscript_error
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-12-03 09:25:50
Input filesusertests:000984_reco_data_2024-11-14T_093620Z_reco_data_2024-11-14T_140516Z_reco_data_2024-11-27T_151929Z_reco_data_2024-12-02T_122736Z_reco_data_2024-12-02T_155227Z.root
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Outputting started 
Output files
Finished2024-12-03 09:26:27
Saved logsjustin-logs:306539.189-justin-prod-sched01.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

AFM ana jobscript.
Input PFN = root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/usertests/4e/8e/000984_reco_data_2024-11-14T_093620Z_reco_data_2024-11-14T_140516Z_reco_data_2024-11-27T_151929Z_reco_data_2024-12-02T_122736Z_reco_data_2024-12-02T_155227Z.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_4_3b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libXrdPosixPreload.so
/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/36cd515f81496cd78de9c544b1945390cb3498ef
/cvmfs/fifeuser1.opensciencegrid.org/sw/dune/36cd515f81496cd78de9c544b1945390cb3498ef:.:./job:/cvmfs/dune.opensciencegrid.org/products/dune/dunesw/v09_75_03d00/fcl:/cvmfs/dune.opensciencegrid.org/products/dune/protoduneana/v09_75_03d00/job:/cvmfs/larsoft.opensciencegrid.org/products/geant4reweight/v01_18_01/fcl:/cvmfs/dune.opensciencegrid.org/products/dune/duneprototypes/v09_75_03d00/fcl:/cvmfs/dune.opensciencegrid.org/products/dune/dunesim/v09_75_03d00/fcl:/cvmfs/dune.opensciencegrid.org/products/dune/dunecalib/v09_75_03d00/fcl:/cvmfs/dune.opensciencegrid.org/products/dune/dunedataprep/v09_75_03d00/fcl:/cvmfs/dune.opensciencegrid.org/products/dune/duneana/v09_75_03d00/fcl:/cvmfs/dune.opensciencegrid.org/products/dune/dunereco/v09_75_03d00/fcl:/cvmfs/dune.opensciencegrid.org/products/dune/duneopdet/v09_75_03d00/fcl:/cvmfs/dune.opensciencegrid.org/products/dune/dunecore/v09_75_03d00/fcl:/cvmfs/larsoft.opensciencegrid.org/products/nusystematics/v01_03_09/fcl:/cvmfs/larsoft.opensciencegrid.org/products/systematicstools/v01_03_01/fcl:.:./job:/cvmfs/larsoft.opensciencegrid.org/products/larwirecell/v09_14_01/fcl:/cvmfs/larsoft.opensciencegrid.org/products/larsimdnn/v09_05_02/job:/cvmfs/larsoft.opensciencegrid.org/products/larrecodnn/v09_21_03/job:/cvmfs/larsoft.opensciencegrid.org/products/larpandora/v09_21_02/job:/cvmfs/larsoft.opensciencegrid.org/products/larexamples/v09_08_02/job:/cvmfs/larsoft.opensciencegrid.org/products/lareventdisplay/v09_10_02/job:/cvmfs/larsoft.opensciencegrid.org/products/nuevdb/v1_08_01/fcl:/cvmfs/larsoft.opensciencegrid.org/products/larana/v09_14_02/job:/cvmfs/larsoft.opensciencegrid.org/products/larreco/v09_20_02/job:/cvmfs/larsoft.opensciencegrid.org/products/larsim/v09_36_02/job:/cvmfs/larsoft.opensciencegrid.org/products/nugen/v1_19_04/fcl:/cvmfs/larsoft.opensciencegrid.org/products/larg4/v09_16_01/fcl:/cvmfs/larsoft.opensciencegrid.org/products/nurandom/v1_10_01/fcl:/cvmfs/larsoft.opensciencegrid.org/products/nug4/v1_15_01/fcl:/cvmfs/larsoft.opensciencegrid.org/products/larevt/v09_09_01/job:/cvmfs/larsoft.opensciencegrid.org/products/lardata/v09_15_01/job:/cvmfs/larsoft.opensciencegrid.org/products/lardataalg/v09_15_01/job:/cvmfs/larsoft.opensciencegrid.org/products/larcore/v09_10_00/job:/cvmfs/larsoft.opensciencegrid.org/products/larcorealg/v09_12_00/job:/cvmfs/larsoft.opensciencegrid.org/products/artg4tk/v11_00_01/fcl:/cvmfs/dune.opensciencegrid.org/products/dune/dunepdlegacy/v1_00_03/fcl
=== Start last 200 lines of lar log file ===

RooFit v3.60 -- Developed by Wouter Verkerke and David Kirkby 
                Copyright (C) 2000-2013 NIKHEF, University of California & Stanford University
                All rights reserved, please read http://roofit.sourceforge.net/license.txt

Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/dune.opensciencegrid.org/products/dune/dunecore/v09_75_03d00/gdml/dune10kt_v4_1x2x6.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 1149
Info in <TGeoManager::CloseGeometry>: 67545 nodes/ 3831 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
registering to  primaryGeneratorActionsMap_
registering to   eventActionsMap_
registering to   trackingActionsMap_
registering to  steppingActionsMap_
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Active Boundaries: 
	x: -363.376 to 363.376
	y: -607.829 to 607.829
	z: -0.87625 to 1393.46
03-Dec-2024 09:26:07 GMT  Initiating request to open input file "root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/usertests/4e/8e/000984_reco_data_2024-11-14T_093620Z_reco_data_2024-11-14T_140516Z_reco_data_2024-11-27T_151929Z_reco_data_2024-12-02T_122736Z_reco_data_2024-12-02T_155227Z.root"
Plugin version SecClnt v5.4.3 is incompatible with secztn v5.6.8 (must be <= 5.4.x) in sec.protocol libXrdSecztn-5.so
%MSG-s ArtException:  NeutrinoShowerEff:trajclustershweff@Construction  03-Dec-2024 09:26:08 GMT ModuleConstruction
cet::exception caught in art
---- FileOpenError BEGIN
  RootInputFileSequence::initFile(): Input file root://cephc02.gla.scotgrid.ac.uk:1094//cephfs/dune/RSE/usertests/4e/8e/000984_reco_data_2024-11-14T_093620Z_reco_data_2024-11-14T_140516Z_reco_data_2024-11-27T_151929Z_reco_data_2024-12-02T_122736Z_reco_data_2024-12-02T_155227Z.root was not found or could not be opened.
---- FileOpenError END
%MSG
Art has completed and will exit with status 20.
=== End last 200 lines of lar log file ===
lar exit code 0
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_75_03d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_75_03d00/bin/extractor_prod.py", line 373, in main
    mddict = expSpecificMetadata.getmetadata()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_75_03d00/bin/extractor_prod.py", line 344, in getmetadata
    jobt = self.get_job(proc)
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_75_03d00/bin/extractor_prod.py", line 69, in get_job
    raise RuntimeError('sam_metadata_dumper returned nonzero exit status {}.'.format(rc))
RuntimeError: sam_metadata_dumper returned nonzero exit status 1.
extractor_prod.py exit code 1
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
pdjson2metadata exit code 1
.:
total 36
-rw-r--r--. 1 pildune000 dunepil 18761 Dec  3 09:26 -_20241203T092608_ana_hist.root
-rw-r--r--. 1 pildune000 dunepil  6790 Dec  3 09:26 jobscript.log
-rw-r--r--. 1 pildune000 dunepil  2710 Dec  3 09:26 000984_reco_data_2024-11-14T_093620Z_reco_data_2024-11-14T_140516Z_reco_data_2024-11-27T_151929Z_reco_data_2024-12-02T_122736Z_reco_data_2024-12-02T_155227Z_reco_2024-12-03T_092553Z.log
-rw-r--r--. 1 pildune000 dunepil   344 Dec  3 09:25 all-input-dids.txt
-rw-r--r--. 1 pildune000 dunepil     0 Dec  3 09:26 000984_reco_data_2024-11-14T_093620Z_reco_data_2024-11-14T_140516Z_reco_data_2024-11-27T_151929Z_reco_data_2024-12-02T_122736Z_reco_data_2024-12-02T_155227Z_reco_data_2024-12-03T_092553Z.root.ext.json
-rw-r--r--. 1 pildune000 dunepil     0 Dec  3 09:26 000984_reco_data_2024-11-14T_093620Z_reco_data_2024-11-14T_140516Z_reco_data_2024-11-27T_151929Z_reco_data_2024-12-02T_122736Z_reco_data_2024-12-02T_155227Z_reco_data_2024-12-03T_092553Z.root.json
-rw-r--r--. 1 pildune000 dunepil     0 Dec  3 09:26 debugprod.log
justIN time: 2024-12-04 09:05:02 UTC       justIN version: 01.01.09