justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 206498.12@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID206498.12@justin-prod-sched02.dune.hep.ac.uk
Workflow ID7046
Stage ID1
User nameykermaid@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes6815744000 (6500 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-05-19 13:55:11
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce513
Last heartbeat2025-05-21 02:49:40
From worker nodeHostnameb9p02p1188.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes6815744000 (6500 MiB)
Wall seconds limit343800 (95 hours)
GPU
Inner Apptainer?True
Job statejobscript_error
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-05-19 13:56:26
Input filesmonte-carlo-007046-000063
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-05-21 02:49:40
Saved logsjustin-logs:206498.12-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

ges...
Begin processing the 4th record. run: 1 subRun: 0 event: 4 at 20-May-2025 09:15:09 CEST
%MSG-w PhotonBackTrackerService:  ProcessEvent 20-May-2025 09:15:14 CEST  run: 1 subRun: 0 event: 4
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-i NuRandomService:  IonAndScint:IonAndScint@BeginModule  20-May-2025 09:15:14 CEST run: 1 subRun: 0 event: 4
Random seed for this event, engine 'IonAndScint.ISCalcAlg': 333012076
%MSG
IonAndScint Module Producer
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolCryostat
Skip SimEnergyDeposit in: LArG4DetectorServicevolBeamPlIINi
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActive
%MSG-i NuRandomService:  PDFastSimANN:PDFastSimAr@BeginModule  20-May-2025 09:15:17 CEST run: 1 subRun: 0 event: 4
Random seed for this event, engine 'PDFastSimAr.photon': 526619916
%MSG
%MSG-i NuRandomService:  PDFastSimANN:PDFastSimAr@BeginModule  20-May-2025 09:15:17 CEST run: 1 subRun: 0 event: 4
Random seed for this event, engine 'PDFastSimAr.scinttime': 261520183
%MSG
PDFastSimANN Module Producer...
PDFastSimANN module produced 2801431 images...
Begin processing the 5th record. run: 1 subRun: 0 event: 5 at 20-May-2025 14:43:49 CEST
%MSG-w PhotonBackTrackerService:  ProcessEvent 20-May-2025 14:43:55 CEST  run: 1 subRun: 0 event: 5
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-i NuRandomService:  IonAndScint:IonAndScint@BeginModule  20-May-2025 14:43:55 CEST run: 1 subRun: 0 event: 5
Random seed for this event, engine 'IonAndScint.ISCalcAlg': 594888930
%MSG
IonAndScint Module Producer
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolCryostat
Skip SimEnergyDeposit in: LArG4DetectorServicevolBeamPlIINi
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActive
%MSG-i NuRandomService:  PDFastSimANN:PDFastSimAr@BeginModule  20-May-2025 14:43:59 CEST run: 1 subRun: 0 event: 5
Random seed for this event, engine 'PDFastSimAr.photon': 545346912
%MSG
%MSG-i NuRandomService:  PDFastSimANN:PDFastSimAr@BeginModule  20-May-2025 14:43:59 CEST run: 1 subRun: 0 event: 5
Random seed for this event, engine 'PDFastSimAr.scinttime': 74742502
%MSG
PDFastSimANN Module Producer...
PDFastSimANN module produced 3280153 images...
Begin processing the 6th record. run: 1 subRun: 0 event: 6 at 20-May-2025 21:12:55 CEST
%MSG-w PhotonBackTrackerService:  ProcessEvent 20-May-2025 21:13:00 CEST  run: 1 subRun: 0 event: 6
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-i NuRandomService:  IonAndScint:IonAndScint@BeginModule  20-May-2025 21:13:00 CEST run: 1 subRun: 0 event: 6
Random seed for this event, engine 'IonAndScint.ISCalcAlg': 551946388
%MSG
IonAndScint Module Producer
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolCryostat
Skip SimEnergyDeposit in: LArG4DetectorServicevolBeamPlIINi
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActive
%MSG-i NuRandomService:  PDFastSimANN:PDFastSimAr@BeginModule  20-May-2025 21:13:04 CEST run: 1 subRun: 0 event: 6
Random seed for this event, engine 'PDFastSimAr.photon': 541269086
%MSG
%MSG-i NuRandomService:  PDFastSimANN:PDFastSimAr@BeginModule  20-May-2025 21:13:04 CEST run: 1 subRun: 0 event: 6
Random seed for this event, engine 'PDFastSimAr.scinttime': 537282406
%MSG
PDFastSimANN Module Producer...
PDFastSimANN module produced 3774387 images...
IonAndScint endJob.
PDFastSimANN endJob.
Close TF session.
21-May-2025 04:48:35 CEST  Closed input file "prod_cosmics_protodunevd_1_206498_12_1747662996_gen_g4_stage1.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                      16971.4       22019.5       27301.6       21678.1       3465.55         6     
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.00087962    0.00353608    0.00904272    0.00248047    0.00288159        6     
simulate:rns:RandomNumberSaver                 3.803e-05    9.73092e-05   0.000362373   4.56205e-05   0.000118581       6     
simulate:IonAndScint:IonAndScint                2.12223       3.24966       4.06333       3.2448       0.684235         6     
simulate:PDFastSimAr:PDFastSimANN               16922.5       21971.3       27297.5       21621.3       3475.24         6     
[art]:TriggerResults:TriggerResultInserter     2.317e-05    6.60277e-05   0.000264173   2.5941e-05    8.86659e-05       6     
end_path:out1:RootOutput                        6.1e-06     3.19903e-05   0.000153372    7.15e-06     5.43129e-05       6     
end_path:out1:RootOutput(write)                 46.745        53.8767       64.1775       52.3878       5.72677         5     
================================================================================================================================
%MSG-i NuRandomService:  RootOutput:out1@EndJob 21-May-2025 04:48:35 CEST  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1
   Configured value          Last value   ModuleLabel.InstanceName
        (per event)           551946388   IonAndScint.ISCalcAlg
        (per event)           541269086   PDFastSimAr.photon
        (per event)           537282406   PDFastSimAr.scinttime

%MSG

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 16197.3 MB
  Peak resident set size usage (VmHWM): 14063.2 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 6 passed = 6 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport          6          6          0 out1

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 500579.887793 Real = 132223.908165

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 16197.3 VmHWM = 14063.2

%MSG-s ArtException:  PostEndJob 21-May-2025 04:49:19 CEST ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- FatalRootError BEGIN
    Fatal Root Error: TBufferFile::WriteByteCount
    bytecount too large (more than 1073741822)
    ROOT severity: 3000
  ---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 5 entries while sim::ParticleAncestryMap_largeant__G4Stage1. has 10 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.

Stderr:
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser4.opensciencegrid.org/sw/dune/70eefeb7ad07920f18e97df5bcaae46af9f21703/protodunevd_v5_ggd_May7PDfix_CRT.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16460 nodes/ 2469 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).

Error in processing g4_stage2
Info in <TGeoManager::Import>: Reading geometry from file: /cvmfs/fifeuser4.opensciencegrid.org/sw/dune/70eefeb7ad07920f18e97df5bcaae46af9f21703/protodunevd_v5_ggd_May7PDfix_CRT.gdml
Info in <TGeoManager::TGeoManager>: Geometry GDMLImport, Geometry imported from GDML created
Info in <TGeoManager::SetTopVolume>: Top volume is volWorld. Master volume is volWorld
Info in <TGeoNavigator::BuildCache>: --- Maximum geometry depth set to 100
Info in <TGeoManager::CheckGeometry>: Fixing runtime shapes...
Info in <TGeoManager::CheckGeometry>: ...Nothing to fix
Info in <TGeoManager::CloseGeometry>: Counting nodes...
Info in <TGeoManager::Voxelize>: Voxelizing...
Info in <TGeoManager::CloseGeometry>: Building cache...
Info in <TGeoManager::CountLevels>: max level = 5, max placements = 292
Info in <TGeoManager::CloseGeometry>: 16460 nodes/ 2469 volume UID's in Geometry imported from GDML
Info in <TGeoManager::CloseGeometry>: ----------------modeler ready----------------
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak).
Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak).

Error running. Exiting with 1
justIN time: 2025-05-23 04:50:27 UTC       justIN version: 01.03.01