21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 416643.1@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 416643.1@justin-prod-sched01.dune.hep.ac.uk | |
| Workflow ID | 7715 | |
| Stage ID | 1 | |
| User name | gabrielb@fnal.gov | |
| HTCondor Group | group_dune | |
| Requested | Processors | 1 |
| GPU | Yes | |
| RSS bytes | 25165824000 (24000 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-06-18 13:45:40 | |
| Site | UK_QMUL | |
| Entry | DUNE_UK_London_QMUL_arcce03_gpu | |
| Last heartbeat | 2025-06-18 14:28:19 | |
| From worker node | Hostname | cn298.htc.esc.qmul |
| cpuinfo | AMD EPYC 7402 24-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 25165824000 (24000 MiB) | |
| Wall seconds limit | 171000 (47 hours) | |
| GPU | NVIDIA A100-PCIE-40GB 570.133.20 8.0 92.00.25.00.08 40442MiB | |
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-06-18 13:51:14 | |
| Input files | ||
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-06-18 14:28:19 | |
| Saved logs | justin-logs:416643.1-justin-prod-sched01.dune.hep.ac.uk.logs.tgz | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
%MSG
%MSG-i endOfEventAction: larg4Main:largeant@BeginModule 18-Jun-2025 15:19:16 BST run: 1 subRun: 0 event: 8 ParticleListAction.cc:830
Found 755 particles
%MSG
%MSG-i NuRandomService: IonAndScint:IonAndScint@BeginModule 18-Jun-2025 15:19:17 BST run: 1 subRun: 0 event: 8
Random seed for this event, engine 'IonAndScint.ISCalcAlg': 232773772
%MSG
IonAndScint Module Producer
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveInner
Skip SimEnergyDeposit in: LArG4DetectorServicevolTPCActiveOuter
%MSG-i NuRandomService: IonAndScint:IonAndScintExternal@BeginModule 18-Jun-2025 15:19:18 BST run: 1 subRun: 0 event: 8
Random seed for this event, engine 'IonAndScintExternal.ISCalcAlg': 433758197
%MSG
IonAndScint Module Producer
Skip SimEnergyDeposit in:
Skip SimEnergyDeposit in: LArG4DetectorServicevolTPCActiveInner
Skip SimEnergyDeposit in: priorSCE
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveOuter
%MSG-i NuRandomService: PDFastSimPAR:PDFastSim@BeginModule 18-Jun-2025 15:19:18 BST run: 1 subRun: 0 event: 8
Random seed for this event, engine 'PDFastSim.photon': 437318323
%MSG
%MSG-i NuRandomService: PDFastSimPAR:PDFastSim@BeginModule 18-Jun-2025 15:19:18 BST run: 1 subRun: 0 event: 8
Random seed for this event, engine 'PDFastSim.scinttime': 839944283
%MSG
%MSG-i NuRandomService: PDFastSimPVS:PDFastSimExternal@BeginModule 18-Jun-2025 15:21:06 BST run: 1 subRun: 0 event: 8
Random seed for this event, engine 'PDFastSimExternal.photon': 834358848
%MSG
%MSG-i NuRandomService: PDFastSimPVS:PDFastSimExternal@BeginModule 18-Jun-2025 15:21:06 BST run: 1 subRun: 0 event: 8
Random seed for this event, engine 'PDFastSimExternal.scinttime': 726127455
%MSG
Begin processing the 9th record. run: 1 subRun: 0 event: 9 at 18-Jun-2025 15:21:53 BST
%MSG-w ParticleInventory: ProcessEvent 18-Jun-2025 15:21:53 BST run: 1 subRun: 0 event: 9
Rebuild failed to get the MCParticles. This is expected when running on a generation or simulation step.
%MSG
%MSG-w ParticleInventory: ProcessEvent 18-Jun-2025 15:21:53 BST run: 1 subRun: 0 event: 9
Rebuild failed to get the MCParticles. This is expected when running on a generation or simulation step.
%MSG
%MSG-w ParticleInventory: ProcessEvent 18-Jun-2025 15:21:53 BST run: 1 subRun: 0 event: 9
Rebuild failed to get the MCParticles. This is expected when running on a generation or simulation step.
%MSG
%MSG-w PhotonBackTrackerService: ProcessEvent 18-Jun-2025 15:21:53 BST run: 1 subRun: 0 event: 9
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-i NuRandomService: larg4Main:largeant@BeginModule 18-Jun-2025 15:21:53 BST run: 1 subRun: 0 event: 9
Random seed for this event, engine 'largeant': 371637314
%MSG
%MSG-i generatePrimaries: larg4Main:largeant@BeginModule 18-Jun-2025 15:21:53 BST run: 1 subRun: 0 event: 9 MCTruthEventAction.cc:112
Generating 829 particles
%MSG
%MSG-w beginOfEventAction::keepableGenerators: larg4Main:largeant@BeginModule 18-Jun-2025 15:21:54 BST run: 1 subRun: 0 event: 9
storeTrajectories set to true and a non-empty keepGenTrajectories list provided in configuration file, but none of the generators in this list are present in the event! Double check list or don't provide keepGenTrajectories in the configuration to keep all trajectories from all generator labels. This may be expected for generators that have a nonzero probability of producing no particles (e.g. some radiologicals)
%MSG
%MSG-i endOfEventAction: larg4Main:largeant@BeginModule 18-Jun-2025 15:22:03 BST run: 1 subRun: 0 event: 9 ParticleListAction.cc:817
MCTruth Handles Size: 1
%MSG
%MSG-i endOfEventAction: larg4Main:largeant@BeginModule 18-Jun-2025 15:22:04 BST run: 1 subRun: 0 event: 9 ParticleListAction.cc:827
mclistHandle Size: 1
%MSG
%MSG-i endOfEventAction: larg4Main:largeant@BeginModule 18-Jun-2025 15:22:04 BST run: 1 subRun: 0 event: 9 ParticleListAction.cc:830
Found 829 particles
%MSG
%MSG-i NuRandomService: IonAndScint:IonAndScint@BeginModule 18-Jun-2025 15:22:05 BST run: 1 subRun: 0 event: 9
Random seed for this event, engine 'IonAndScint.ISCalcAlg': 681629466
%MSG
IonAndScint Module Producer
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveInner
Skip SimEnergyDeposit in: LArG4DetectorServicevolTPCActiveOuter
%MSG-i NuRandomService: IonAndScint:IonAndScintExternal@BeginModule 18-Jun-2025 15:22:06 BST run: 1 subRun: 0 event: 9
Random seed for this event, engine 'IonAndScintExternal.ISCalcAlg': 718656094
%MSG
IonAndScint Module Producer
Skip SimEnergyDeposit in:
Skip SimEnergyDeposit in: LArG4DetectorServicevolTPCActiveInner
Skip SimEnergyDeposit in: priorSCE
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveOuter
%MSG-i NuRandomService: PDFastSimPAR:PDFastSim@BeginModule 18-Jun-2025 15:22:06 BST run: 1 subRun: 0 event: 9
Random seed for this event, engine 'PDFastSim.photon': 294463505
%MSG
%MSG-i NuRandomService: PDFastSimPAR:PDFastSim@BeginModule 18-Jun-2025 15:22:06 BST run: 1 subRun: 0 event: 9
Random seed for this event, engine 'PDFastSim.scinttime': 418912612
%MSG
%MSG-i NuRandomService: PDFastSimPVS:PDFastSimExternal@BeginModule 18-Jun-2025 15:25:00 BST run: 1 subRun: 0 event: 9
Random seed for this event, engine 'PDFastSimExternal.photon': 191630795
%MSG
%MSG-i NuRandomService: PDFastSimPVS:PDFastSimExternal@BeginModule 18-Jun-2025 15:25:00 BST run: 1 subRun: 0 event: 9
Random seed for this event, engine 'PDFastSimExternal.scinttime': 622611951
%MSG
IonAndScint endJob.
IonAndScint endJob.
18-Jun-2025 15:25:10 BST Closed input file "data_gen.root"
================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
================================================================================================================================
Full event 143.517 166.638 192.418 158.768 15.2288 9
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000542378 0.00304671 0.00535142 0.00276035 0.00129728 9
simulate:rns:RandomNumberSaver 0.000212405 0.009797 0.0124658 0.011827 0.00368033 9
simulate:largeant:larg4Main 9.49892 11.814 13.7372 11.9967 1.44706 9
simulate:IonAndScint:IonAndScint 0.558184 0.655988 0.868491 0.627757 0.0857478 9
simulate:IonAndScintExternal:IonAndScint 0.0318944 0.04365 0.0626261 0.0416749 0.00949359 9
simulate:PDFastSim:PDFastSimPAR 92.1729 116.284 173.809 108.481 22.2412 9
simulate:PDFastSimExternal:PDFastSimPVS 4.45189 5.89106 10.7341 5.55017 1.76936 9
[art]:TriggerResults:TriggerResultInserter 1.4226e-05 2.32553e-05 8.2293e-05 1.5779e-05 2.09036e-05 9
end_path:out1:RootOutput 3.506e-06 9.74911e-06 2.6149e-05 8.446e-06 6.00499e-06 9
end_path:out1:RootOutput(write) 32.0573 35.9132 41.0558 34.4936 3.08106 8
================================================================================================================================
%MSG-i NuRandomService: RootOutput:out1@EndJob 18-Jun-2025 15:25:10 BST ModuleEndJob
Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
algorithm version: EventTimestamp_v1
Configured value Last value ModuleLabel.InstanceName
(per event) 681629466 IonAndScint.ISCalcAlg
(per event) 718656094 IonAndScintExternal.ISCalcAlg
(per event) 294463505 PDFastSim.photon
(per event) 418912612 PDFastSim.scinttime
(per event) 191630795 PDFastSimExternal.photon
(per event) 622611951 PDFastSimExternal.scinttime
(per event) 371637314 largeant
%MSG
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 13109.1 MB
Peak resident set size usage (VmHWM): 12091.3 MB
====================================================================================================
TrigReport ---------- Event summary -------------
TrigReport Events total = 9 passed = 9 failed = 0
TrigReport ---------- Modules in End-path ----------
TrigReport Run Success Error Name
TrigReport 9 9 0 out1
TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 1572.345143 Real = 1582.956955
MemReport ---------- Memory summary [base-10 MB] ------
MemReport VmPeak = 13109.1 VmHWM = 12091.3
%MSG-s ArtException: PostEndJob 18-Jun-2025 15:25:49 BST ModuleEndJob
---- EventProcessorFailure BEGIN
EventProcessor: an exception occurred during current event processing
---- FatalRootError BEGIN
Fatal Root Error: TBufferFile::WriteByteCount
bytecount too large (more than 1073741822)
ROOT severity: 3000
---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
Fatal Root Error: TTree::SetEntries
Tree branches have different numbers of entries, eg EventAuxiliary has 8 entries while simb::MCTruths_cosmicgenerator__SinglesGen. has 10 entries.
ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
lar exit code 1