justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 207689.156@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID207689.156@justin-prod-sched02.dune.hep.ac.uk
Workflow ID7106
Stage ID1
User namecalcuttj@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes8387559424 (7999 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-05-22 08:26:59
SiteUS_FNAL-FermiGrid
EntryFNAL_GPGrid_ce03_mcore_op_duneonly
Last heartbeat2025-05-22 10:40:01
From worker nodeHostnamedunegli-5200839-0-fnpc8004.fnal.gov
cpuinfoIntel(R) Xeon(R) CPU E5-2670 v3 @ 2.30GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes8387559424 (7999 MiB)
Wall seconds limit172800 (48 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-05-22 09:37:19
Input filesmonte-carlo-007106-006311
JobscriptExit code247
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-05-22 10:40:01
Saved logsjustin-logs:207689.156-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

================================
Full event                                      13.5929       16.4269       21.1259       15.6685       2.69197         5     
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000158734   0.000306759   0.000709963   0.000202348   0.000208346       5     
simulate:IonAndScint:IonAndScint                1.93049       2.24748       2.69815       2.15959      0.291281         5     
simulate:IonAndScintExternal:IonAndScint      0.00150547    0.00344319    0.00631967    0.00306306     0.0015909        5     
[art]:TriggerResults:TriggerResultInserter    1.4825e-05    2.78004e-05   5.9929e-05    2.0575e-05    1.6305e-05        5     
end_path:out1:RootOutput                       3.258e-06    6.4734e-06    1.7202e-05     3.998e-06    5.38401e-06       5     
end_path:out1:RootOutput(write)                 11.6603       14.1751       18.6605       13.5021       2.47773         5     
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 5464.7 MB
  Peak resident set size usage (VmHWM): 4455.56 MB
  Details saved in: 'mem.db'
====================================================================================================

{'art_out': 'placeholder_1_207689_156_1747906651_stage1_stage2.root'}
stage3
Running stage3
['lar', '-c', 'run_wirecell_perfect_apa1.fcl', 'placeholder_1_207689_156_1747906651_stage1_stage2.root', '-n', '-1', '-e', '207689:156:1', '-o', 'placeholder_1_207689_156_1747906651_stage1_stage2_stage3.root']
Stdout:
Begin processing the 1st record. run: 207689 subRun: 156 event: 1 at 22-May-2025 09:48:48 UTC
Larwirecell::SimDepoSource got 998903 associated depos from InputTag: label = 'IonAndScint', instance = 'priorSCE'
Begin processing the 2nd record. run: 207689 subRun: 156 event: 2 at 22-May-2025 09:59:03 UTC
Larwirecell::SimDepoSource got 1210136 associated depos from InputTag: label = 'IonAndScint', instance = 'priorSCE'
Begin processing the 3rd record. run: 207689 subRun: 156 event: 3 at 22-May-2025 10:09:57 UTC
Larwirecell::SimDepoSource got 1098646 associated depos from InputTag: label = 'IonAndScint', instance = 'priorSCE'
Begin processing the 4th record. run: 207689 subRun: 156 event: 4 at 22-May-2025 10:20:43 UTC
Larwirecell::SimDepoSource got 985350 associated depos from InputTag: label = 'IonAndScint', instance = 'priorSCE'
Begin processing the 5th record. run: 207689 subRun: 156 event: 5 at 22-May-2025 10:30:40 UTC
Larwirecell::SimDepoSource got 879363 associated depos from InputTag: label = 'IonAndScint', instance = 'priorSCE'

Stderr:
The following paths have not been enabled for execution and will be ignored:
  analyze
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  IonAndScint
  IonAndScintExternal
  cosmicgenerator
  largeant
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/dune.opensciencegrid.org/products/dune/dunecore/v10_01_03d00/gdml/protodunehd_v6_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 1148
Info in <TGeoManager::CloseGeometry>: 25608 nodes/ 5535 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
registering to  primaryGeneratorActionsMap_
registering to   eventActionsMap_
registering to   trackingActionsMap_
registering to  steppingActionsMap_
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsSimDepoSource:"
Outputer: "wclsSimChannelSink:postdrift"
Outputer: "wclsFrameSaver:simdigits"
wclsFrameSaver: promising to produce raw::RawDigits named "daq"
22-May-2025 09:48:47 UTC  Initiating request to open input file "placeholder_1_207689_156_1747906651_stage1_stage2.root"
22-May-2025 09:48:48 UTC  Opened input file "placeholder_1_207689_156_1747906651_stage1_stage2.root"
SimDepoSource got 998903 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSource: ready with 998903 depos spanning: [-3301.79, 2992.82]us
Retagger: tagging trace set: daq with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "daq"
22-May-2025 09:58:37 UTC  Opened output file with pattern "placeholder_1_207689_156_1747906651_stage1_stage2_stage3.root"
SimDepoSource got 1210136 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSource: ready with 1210136 depos spanning: [-3322.06, 3076]us
Retagger: tagging trace set: daq with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "daq"
SimDepoSource got 1098646 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSource: ready with 1098646 depos spanning: [-3263.34, 3066.93]us
Retagger: tagging trace set: daq with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "daq"
SimDepoSource got 985350 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSource: ready with 985350 depos spanning: [-3283.17, 3116.04]us
Retagger: tagging trace set: daq with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "daq"
SimDepoSource got 879363 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSource: ready with 879363 depos spanning: [-3278.01, 3075.46]us

Error in processing stage3
The following paths have not been enabled for execution and will be ignored:
  analyze
The following module labels are either not assigned to any path,
or they have been assigned to ignored path(s):
  IonAndScint
  IonAndScintExternal
  cosmicgenerator
  largeant
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/dune.opensciencegrid.org/products/dune/dunecore/v10_01_03d00/gdml/protodunehd_v6_refactored.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 1148
Info in <TGeoManager::CloseGeometry>: 25608 nodes/ 5535 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
registering to  primaryGeneratorActionsMap_
registering to   eventActionsMap_
registering to   trackingActionsMap_
registering to  steppingActionsMap_
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).
Inputer: "wclsSimDepoSource:"
Outputer: "wclsSimChannelSink:postdrift"
Outputer: "wclsFrameSaver:simdigits"
wclsFrameSaver: promising to produce raw::RawDigits named "daq"
22-May-2025 09:48:47 UTC  Initiating request to open input file "placeholder_1_207689_156_1747906651_stage1_stage2.root"
22-May-2025 09:48:48 UTC  Opened input file "placeholder_1_207689_156_1747906651_stage1_stage2.root"
SimDepoSource got 998903 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSource: ready with 998903 depos spanning: [-3301.79, 2992.82]us
Retagger: tagging trace set: daq with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "daq"
22-May-2025 09:58:37 UTC  Opened output file with pattern "placeholder_1_207689_156_1747906651_stage1_stage2_stage3.root"
SimDepoSource got 1210136 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSource: ready with 1210136 depos spanning: [-3322.06, 3076]us
Retagger: tagging trace set: daq with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "daq"
SimDepoSource got 1098646 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSource: ready with 1098646 depos spanning: [-3263.34, 3066.93]us
Retagger: tagging trace set: daq with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "daq"
SimDepoSource got 985350 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSource: ready with 985350 depos spanning: [-3283.17, 3116.04]us
Retagger: tagging trace set: daq with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "daq"
SimDepoSource got 879363 depos from art tag "InputTag: label = 'IonAndScint', instance = ''" returns: okay
SimDepoSource: ready with 879363 depos spanning: [-3278.01, 3075.46]us

Error running. Exiting with 247
justIN time: 2025-05-22 22:12:00 UTC       justIN version: 01.03.01