justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 289742.116@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID289742.116@justin-prod-sched01.dune.hep.ac.uk
Workflow ID4103
Stage ID1
User nameamoor@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes8388608000 (8000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-11-14 13:39:01
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce508
Last heartbeat2024-11-14 16:18:15
From worker nodeHostnameb9p20p9398.cern.ch
cpuinfoIntel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes8388608000 (8000 MiB)
Wall seconds limit171000 (47 hours)
Inner Apptainer?True
Job statejobscript_error
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-11-14 14:00:58
Input filesusertests:000338_reco_data_2024-11-14T_092915Z.root
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Outputting started 
Output files
Finished2024-11-14 16:18:15
Saved logsjustin-logs:289742.116-justin-prod-sched01.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

AFM g4 jobscript.
Input PFN = root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/usertests/c0/76/000338_reco_data_2024-11-14T_092915Z.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_4_3b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
	Brem : 191667
	conv : 31460
	Ion : 84396
%MSG
%MSG-i endOfEventAction:  larg4Main:largeant@BeginModule  14-Nov-2024 16:01:22 CET run: 20000031 subRun: 0 event: 795 ParticleListAction.cc:701
MCTruth Handles Size: 1
%MSG
%MSG-i endOfEventAction:  larg4Main:largeant@BeginModule  14-Nov-2024 16:01:22 CET run: 20000031 subRun: 0 event: 795 ParticleListAction.cc:708
mclistHandle Size: 1
%MSG
%MSG-i endOfEventAction:  larg4Main:largeant@BeginModule  14-Nov-2024 16:01:22 CET run: 20000031 subRun: 0 event: 795 ParticleListAction.cc:711
Found 1 particles
%MSG
%MSG-i NuRandomService:  IonAndScint:IonAndScint@BeginModule  14-Nov-2024 16:01:22 CET run: 20000031 subRun: 0 event: 795
Random seed for this event, engine 'IonAndScint.ISCalcAlg': 56695298
%MSG
IonAndScint Module Producer
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCPlaneUInner
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveInner
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCInner
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCPlaneVInner
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCPlaneZInner
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveOuter
%MSG-i NuRandomService:  SimDriftElectrons:elecDrift@BeginModule  14-Nov-2024 16:01:27 CET run: 20000031 subRun: 0 event: 795
Random seed for this event, engine 'elecDrift': 373412406
%MSG
%MSG-i NuRandomService:  PDFastSimPAR:PDFastSim@BeginModule  14-Nov-2024 16:02:16 CET run: 20000031 subRun: 0 event: 795
Random seed for this event, engine 'PDFastSim.photon': 244322826
%MSG
%MSG-i NuRandomService:  PDFastSimPAR:PDFastSim@BeginModule  14-Nov-2024 16:02:16 CET run: 20000031 subRun: 0 event: 795
Random seed for this event, engine 'PDFastSim.scinttime': 284352464
%MSG
IonAndScint endJob.
14-Nov-2024 17:17:13 CET  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/usertests/c0/76/000338_reco_data_2024-11-14T_092915Z.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.0100399       10.05        4598.79      0.409669       164.068        795    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000117076   0.000593508    0.0216453    0.000201942   0.00248138       795    
simulate:rns:RandomNumberSaver                2.5281e-05    4.80504e-05   0.000275519   4.6228e-05    1.44166e-05      795    
simulate:largeant:larg4Main                   0.00804294     0.783715       55.7356      0.370551       2.87617        795    
simulate:IonAndScint:IonAndScint              0.000100842   0.00888544      4.77682     0.000157267    0.169734        795    
simulate:elecDrift:SimDriftElectrons          4.4066e-05     0.0935095      49.2731     7.5577e-05      1.75146        795    
simulate:PDFastSim:PDFastSimPAR               6.5602e-05      9.02834       4489.01     0.000115658     160.116        795    
[art]:TriggerResults:TriggerResultInserter     9.701e-06    1.53805e-05   5.3801e-05    1.4734e-05    5.05593e-06      795    
end_path:out1:RootOutput                       2.877e-06    4.40427e-06   3.1476e-05     3.892e-06    2.08422e-06      795    
end_path:out1:RootOutput(write)               0.000777317    0.134262       10.6637     0.00641145     0.589174        794    
================================================================================================================================
%MSG-i NuRandomService:  RootOutput:out1@EndJob 14-Nov-2024 17:17:13 CET  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1
   Configured value          Last value   ModuleLabel.InstanceName
        (per event)            56695298   IonAndScint.ISCalcAlg
        (per event)           244322826   PDFastSim.photon
        (per event)           284352464   PDFastSim.scinttime
        (per event)           373412406   elecDrift
        (per event)           568976875   largeant

%MSG

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 10151 MB
  Peak resident set size usage (VmHWM): 7417.32 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 795 passed = 795 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport        795        795          0 out1

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 7914.960636 Real = 8035.874989

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 10151 VmHWM = 7417.32

%MSG-s ArtException:  PostEndJob 14-Nov-2024 17:17:45 CET ModuleEndJob
---- EventProcessorFailure BEGIN
  EventProcessor: an exception occurred during current event processing
  ---- FatalRootError BEGIN
    Fatal Root Error: TBufferFile::WriteByteCount
    bytecount too large (more than 1073741822)
    ROOT severity: 3000
  ---- FatalRootError END
---- EventProcessorFailure END
---- FatalRootError BEGIN
  Fatal Root Error: TTree::SetEntries
  Tree branches have different numbers of entries, eg EventAuxiliary has 794 entries while art::TriggerResults_TriggerResults__MUSUNGen. has 1000 entries.
  ROOT severity: 2000
---- FatalRootError END
%MSG
Art has completed and will exit with status 1.
=== End last 100 lines of lar log file ===
lar exit code 0
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_75_03d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_75_03d00/bin/extractor_prod.py", line 373, in main
    mddict = expSpecificMetadata.getmetadata()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_75_03d00/bin/extractor_prod.py", line 344, in getmetadata
    jobt = self.get_job(proc)
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_75_03d00/bin/extractor_prod.py", line 69, in get_job
    raise RuntimeError('sam_metadata_dumper returned nonzero exit status {}.'.format(rc))
RuntimeError: sam_metadata_dumper returned nonzero exit status 1.
extractor_prod.py exit code 1
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
pdjson2metadata exit code 1
.:
total 1869056
-rw-r--r--. 1 duneprd np-comp 1911021260 Nov 14 17:17 RootOutput-9f07-3281-13f3-9e54.root
-rw-r--r--. 1 duneprd np-comp    2873551 Nov 14 17:17 000338_reco_data_2024-11-14T_092915Z_reco_2024-11-14T_140103Z.log
-rw-r--r--. 1 duneprd np-comp       7488 Nov 14 17:17 jobscript.log
-rw-r--r--. 1 duneprd np-comp        519 Nov 14 17:17 g4_hist.root
-rw-r--r--. 1 duneprd np-comp        104 Nov 14 15:00 all-input-dids.txt
-rw-r--r--. 1 duneprd np-comp          0 Nov 14 17:17 000338_reco_data_2024-11-14T_092915Z_reco_data_2024-11-14T_140103Z.root.ext.json
-rw-r--r--. 1 duneprd np-comp          0 Nov 14 17:17 000338_reco_data_2024-11-14T_092915Z_reco_data_2024-11-14T_140103Z.root.json
justIN time: 2024-11-24 19:05:01 UTC       justIN version: 01.01.09