justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 303584.77@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID303584.77@justin-prod-sched01.dune.hep.ac.uk
Workflow ID4274
Stage ID1
User nameepennacc@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes4193255424 (3999 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-11-27 20:03:02
SiteNL_SURFsara
EntryDUNE_SurfSARA_arc02
Last heartbeat2024-11-27 21:44:38
From worker nodeHostnamewn-la-04.gina.surfsara.nl
cpuinfoAMD EPYC 9754 128-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4193255424 (3999 MiB)
Wall seconds limit129600 (36 hours)
Inner Apptainer?True
Job statejobscript_error
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-11-27 21:41:45
Input filesmonte-carlo-004274-000240
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Outputting started 
Output files
Finished2024-11-27 21:44:38
Saved logsjustin-logs:303584.77-justin-prod-sched01.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

ctorService::Ctr:  Early 27-Nov-2024 22:43:40 CET JobSetup
Volume: volCryostat, stepLimit: 0.4
%MSG
registering to  primaryGeneratorActionsMap_
registering to   eventActionsMap_
registering to   trackingActionsMap_
registering to  steppingActionsMap_
%MSG-i ParticleListActionService:  Early 27-Nov-2024 22:43:40 CET JobSetup
The full tracking information will not be stored for particles resulting from the following processes: 
{ "conv" "LowEnConversion" "Pair" "compt" "Compt" "Brem" "phot" "Photo" "Ion" "annihil" }

%MSG
%MSG-i ParticleListActionService:  Early 27-Nov-2024 22:43:40 CET JobSetup
Trajectory sparsification enabled with SparsifyMargin : 0.015

%MSG
%MSG-i PhotonVisibilityService:  Early 27-Nov-2024 22:43:40 CET JobSetup
PhotonVisbilityService initializing
%MSG
%MSG-i SignalShapingServiceDUNE:  Early 27-Nov-2024 22:43:40 CET JobSetup
Getting Filter from .fcl file
%MSG
%MSG-i SignalShapingServiceDUNE:  Early 27-Nov-2024 22:43:40 CET JobSetup
 using the field response provided from a .root file 
%MSG
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
%MSG-i RANDOM:  PDFastSimPAR:PDFastSimXe@Construction 27-Nov-2024 22:43:40 CET  ModuleConstruction
Instantiated HepJamesRandom engine "PDFastSimXe:0:photon" with seed 0.
%MSG
%MSG-i NuRandomService:  PDFastSimPAR:PDFastSimXe@Construction  27-Nov-2024 22:43:40 CET ModuleConstruction
Seeding HepJamesRandom engine "PDFastSimXe:photon" with seed 0.
%MSG
%MSG-i RANDOM:  PDFastSimPAR:PDFastSimXe@Construction 27-Nov-2024 22:43:40 CET  ModuleConstruction
Instantiated HepJamesRandom engine "PDFastSimXe:0:scinttime" with seed 0.
%MSG
%MSG-i NuRandomService:  PDFastSimPAR:PDFastSimXe@Construction  27-Nov-2024 22:43:40 CET ModuleConstruction
Seeding HepJamesRandom engine "PDFastSimXe:scinttime" with seed 0.
%MSG
%MSG-i ScintTimeXeDoping:  PDFastSimPAR:PDFastSimXe@Construction  27-Nov-2024 22:43:40 CET ModuleConstruction
Configured:
  XeConcentration: 10 ppm
  ArTripletTime:   1600 ns
  Xe150nmTime:     5600 ns
  TauAX:           7320 ns
  TauXX:           13550 ns
Calculated:
  TauTA singlet:   5.80952 ns
  TauTA triplet:   502.23 ns
  TauTX:           1091.01 ns
  MaxTime Singlet: 7542.5 ns
  MaxTime Triplet: 8209.4 ns

%MSG
%MSG-i PDFastSimPAR:  PDFastSimPAR:PDFastSimXe@Construction  27-Nov-2024 22:43:40 CET ModuleConstruction
Initializing PDFastSimPAR.
%MSG
%MSG-i SemiAnalyticalModel:  PDFastSimPAR:PDFastSimXe@Construction  27-Nov-2024 22:43:40 CET ModuleConstruction
Initializing Semi-analytical model.
%MSG
%MSG-i SemiAnalyticalModel:  PDFastSimPAR:PDFastSimXe@Construction  27-Nov-2024 22:43:40 CET ModuleConstruction
Using VUV visibility parameterization
%MSG
%MSG-i SemiAnalyticalModel:  PDFastSimPAR:PDFastSimXe@Construction  27-Nov-2024 22:43:40 CET ModuleConstruction
Setting absorption length to: 8000
%MSG
%MSG-i SemiAnalyticalModel:  PDFastSimPAR:PDFastSimXe@Construction  27-Nov-2024 22:43:40 CET ModuleConstruction
Semi-analytical model initialized.
%MSG
%MSG-i PropagationTimeModel:  PDFastSimPAR:PDFastSimXe@Construction  27-Nov-2024 22:43:40 CET ModuleConstruction
Initializing Photon propagation time model.
%MSG
%MSG-i PropagationTimeModel:  PDFastSimPAR:PDFastSimXe@Construction  27-Nov-2024 22:43:40 CET ModuleConstruction
Using geometric VUV time propagation
%MSG
%MSG-i PropagationTimeModel:  PDFastSimPAR:PDFastSimXe@Construction  27-Nov-2024 22:43:40 CET ModuleConstruction
Photon propagation time model initalized.
%MSG
%MSG-i PDFastSimPAR:  PDFastSimPAR:PDFastSimXe@Construction  27-Nov-2024 22:43:40 CET ModuleConstruction
Using Lite Photons
%MSG
%MSG-i PDFastSimPAR:  PDFastSimPAR:PDFastSimXe@Construction  27-Nov-2024 22:43:40 CET ModuleConstruction
PDFastSimPAR Initialization finish.
Simulate using semi-analytic model for number of hits.
%MSG
%MSG-i RANDOM:  PDFastSimPVS:PDFastSimXeExternal@Construction  27-Nov-2024 22:43:40 CET ModuleConstruction
Instantiated HepJamesRandom engine "PDFastSimXeExternal:0:photon" with seed 0.
%MSG
%MSG-i NuRandomService:  PDFastSimPVS:PDFastSimXeExternal@Construction  27-Nov-2024 22:43:40 CET ModuleConstruction
Seeding HepJamesRandom engine "PDFastSimXeExternal:photon" with seed 0.
%MSG
%MSG-i RANDOM:  PDFastSimPVS:PDFastSimXeExternal@Construction  27-Nov-2024 22:43:40 CET ModuleConstruction
Instantiated HepJamesRandom engine "PDFastSimXeExternal:0:scinttime" with seed 0.
%MSG
%MSG-i NuRandomService:  PDFastSimPVS:PDFastSimXeExternal@Construction  27-Nov-2024 22:43:40 CET ModuleConstruction
Seeding HepJamesRandom engine "PDFastSimXeExternal:scinttime" with seed 0.
%MSG
%MSG-i ScintTimeXeDoping:  PDFastSimPVS:PDFastSimXeExternal@Construction  27-Nov-2024 22:43:40 CET ModuleConstruction
Configured:
  XeConcentration: 10 ppm
  ArTripletTime:   1600 ns
  Xe150nmTime:     5600 ns
  TauAX:           7320 ns
  TauXX:           13550 ns
Calculated:
  TauTA singlet:   5.80952 ns
  TauTA triplet:   502.23 ns
  TauTX:           1091.01 ns
  MaxTime Singlet: 7542.5 ns
  MaxTime Triplet: 8209.4 ns

%MSG
%MSG-i PDFastSimPVS:  PDFastSimPVS:PDFastSimXeExternal@Construction  27-Nov-2024 22:43:40 CET ModuleConstruction
Initializing PDFastSimPVS.
%MSG
%MSG-i PropagationTimeModel:  PDFastSimPVS:PDFastSimXeExternal@Construction  27-Nov-2024 22:43:40 CET ModuleConstruction
Initializing Photon propagation time model.
%MSG
%MSG-i PropagationTimeModel:  PDFastSimPVS:PDFastSimXeExternal@Construction  27-Nov-2024 22:43:40 CET ModuleConstruction
Using VUV timing parameterization
%MSG
%MSG-i PropagationTimeModel:  PDFastSimPVS:PDFastSimXeExternal@Construction  27-Nov-2024 22:43:40 CET ModuleConstruction
Generating VUV parameters
%MSG
%MSG-i PropagationTimeModel:  PDFastSimPVS:PDFastSimXeExternal@Construction  27-Nov-2024 22:44:11 CET ModuleConstruction
Photon propagation time model initalized.
%MSG
%MSG-i PDFastSimPVS:  PDFastSimPVS:PDFastSimXeExternal@Construction  27-Nov-2024 22:44:11 CET ModuleConstruction
Use Lite Photon.
%MSG
%MSG-i PDFastSimPVS:  PDFastSimPVS:PDFastSimXeExternal@Construction  27-Nov-2024 22:44:11 CET ModuleConstruction
PDFastSimPVS Initialization finish
%MSG
%MSG-s ArtException:  TriggerResultInserter:TriggerResults@Construction  27-Nov-2024 22:44:11 CET ModuleConstruction
cet::exception caught in art
---- CatalogServiceError BEGIN
  Input file not found: prodmarley_nue_cc_flat_dunevd10kt_1x8x14_3view_30deg_20241127T214149Z_gen_000240_supernova_g4stage1.root.
---- CatalogServiceError END
%MSG
Art has completed and will exit with status 24.
ls: cannot access *_20241127T214149Z_*: No such file or directory
ERROR: lar (geant4 step2) exit code: 31024 
output files size: 
../justin-jobscript: line 151: return: can only `return' from a function or sourced script
============detsim=========================

 *** Break *** segmentation violation
../justin-jobscript: line 159:  1356 Segmentation fault      lar -c standard_detsim_dunevd10kt_1x8x14_3view_30deg.fcl ${g4stage2_name}.root -o ${detsim_name}.root -n -1
============reco =========================
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  cvneva
  cvnmap
  energyrecnc
  energyrecnue
  energyrecnumu
  linecluster
  opflash10ppm
  opflashArOnly
  opflashXeOnly
  ophitArOnly
  ophitXeOnly
  pandora
  pandoraShower
  pandoraTrack
  pandoracalo
  pandorapid
  pmtracktc
  rns
  trajcluster
  wclsdatanfsp
%MSG-s ArtException:  Early 27-Nov-2024 22:44:37 CET JobSetup
cet::exception caught in art
---- Configuration BEGIN
  Unable to load requested library /cvmfs/dune.opensciencegrid.org/products/dune/dunedataprep/v09_91_04d00/slf7.x86_64.e26.prof/lib/libdunedataprep_DataPrep_Service_DuneDeconvolutionService_service.so
  libRIO.so.6.28: cannot open shared object file: No such file or directory
---- Configuration END
%MSG
Art has completed and will exit with status 65.
ls: cannot access *_20241127T214149Z_*: No such file or directory
ERROR: lar (reco) exit code: 41065 
output files size: 
../justin-jobscript: line 187: return: can only `return' from a function or sourced script
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_91_04d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_91_04d00/bin/extractor_prod.py", line 373, in main
    mddict = expSpecificMetadata.getmetadata()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_91_04d00/bin/extractor_prod.py", line 343, in getmetadata
    proc = self.extract_metadata_to_pipe()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_91_04d00/bin/extractor_prod.py", line 45, in extract_metadata_to_pipe
    proc = Popen(["sam_metadata_dumper", local], stdout=PIPE,
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/lib/python3.9/subprocess.py", line 1754, in _execute_child
    self.pid = _posixsubprocess.fork_exec(
BlockingIOError: [Errno 11] Resource temporarily unavailable
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
ERROR: metadata writing  42001 
output files size: 
prodmarley_nue_cc_flat_dunevd10kt_1x8x14_3view_30deg_20241127T214149Z_gen_000240_supernova_g4stage1_g4stage2_detsim_reco.root.ext.json 0
prodmarley_nue_cc_flat_dunevd10kt_1x8x14_3view_30deg_20241127T214149Z_gen_000240_supernova_g4stage1_g4stage2_detsim_reco.root.json 0
../justin-jobscript: line 226: return: can only `return' from a function or sourced script
Exiting with error
../justin-jobscript: line 234: return: can only `return' from a function or sourced script
justIN time: 2024-12-04 19:34:05 UTC       justIN version: 01.01.09