justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 245173.0@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID245173.0@justin-prod-sched01.dune.hep.ac.uk
Workflow ID2869
Stage ID1
User nameamoor@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-08-19 23:44:49
SiteUS_Colorado
EntryCMSHTPC_T3_US_Colorado_heposg01-colorado
Last heartbeat2024-08-20 02:00:41
From worker nodeHostnamelnxfarm338.colorado.edu
cpuinfoIntel(R) Core(TM) i9-7920X CPU @ 2.90GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
Inner Apptainer?True
Job statejobscript_error
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-08-19 23:47:15
Input filesusertests:000073_reco_data_2024-08-16T_161654Z.root
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Outputting started 
Output files
Finished2024-08-20 02:00:41
Saved logsjustin-logs:245173.0-justin-prod-sched01.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

AFM g4 jobscript.
Input PFN = root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/8d/be/000073_reco_data_2024-08-16T_161654Z.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_4_3b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libXrdPosixPreload.so
../justin-jobscript: line 72:  1280 Aborted                 (core dumped) lar -c $FCL_FILE $events_option -o $outFile "$pfn" > ${fname}_reco_${now}.log 2>&1
=== Start last 100 lines of lar log file ===
Generating 1 particles
%MSG
%MSG-i ParticleListActionService:  larg4Main:largeant@BeginModule  19-Aug-2024 18:02:59 MDT run: 20000031 subRun: 0 event: 331
Not Stored Process summary:
	compt : 2079370
	annihil : 99854
	Pair : 76
	phot : 666920
	Brem : 612433
	conv : 99768
	Ion : 262723
%MSG
%MSG-i endOfEventAction:  larg4Main:largeant@BeginModule  19-Aug-2024 18:02:59 MDT run: 20000031 subRun: 0 event: 331 ParticleListAction.cc:701
MCTruth Handles Size: 1
%MSG
%MSG-i endOfEventAction:  larg4Main:largeant@BeginModule  19-Aug-2024 18:02:59 MDT run: 20000031 subRun: 0 event: 331 ParticleListAction.cc:708
mclistHandle Size: 1
%MSG
%MSG-i endOfEventAction:  larg4Main:largeant@BeginModule  19-Aug-2024 18:02:59 MDT run: 20000031 subRun: 0 event: 331 ParticleListAction.cc:711
Found 1 particles
%MSG
%MSG-i NuRandomService:  IonAndScint:IonAndScint@BeginModule  19-Aug-2024 18:03:00 MDT run: 20000031 subRun: 0 event: 331
Random seed for this event, engine 'IonAndScint.ISCalcAlg': 135540678
%MSG
IonAndScint Module Producer
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCPlaneUInner
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveInner
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCInner
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCPlaneVInner
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCPlaneZInner
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveOuter
%MSG-i NuRandomService:  SimDriftElectrons:elecDrift@BeginModule  19-Aug-2024 18:03:10 MDT run: 20000031 subRun: 0 event: 331
Random seed for this event, engine 'elecDrift': 317736782
%MSG
%MSG-i NuRandomService:  PDFastSimPAR:PDFastSim@BeginModule  19-Aug-2024 18:04:18 MDT run: 20000031 subRun: 0 event: 331
Random seed for this event, engine 'PDFastSim.photon': 312321013
%MSG
%MSG-i NuRandomService:  PDFastSimPAR:PDFastSim@BeginModule  19-Aug-2024 18:04:18 MDT run: 20000031 subRun: 0 event: 331
Random seed for this event, engine 'PDFastSim.scinttime': 38795196
%MSG
IonAndScint endJob.
19-Aug-2024 20:00:18 MDT  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/8d/be/000073_reco_data_2024-08-16T_161654Z.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                    0.00388454      23.7429       7116.62      0.288428       390.49         331    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        7.1102e-05    0.000625748    0.0455959    0.000147846   0.00369246       331    
simulate:rns:RandomNumberSaver                 1.482e-05    3.02687e-05   0.000312929   2.8982e-05    1.65841e-05      331    
simulate:largeant:larg4Main                   0.00327349      0.75061       95.1577      0.264598       5.27098        331    
simulate:IonAndScint:IonAndScint              6.5254e-05     0.0336264      10.6181     0.000129582    0.582675        331    
simulate:elecDrift:SimDriftElectrons          3.1143e-05     0.220123       67.4094     5.3903e-05      3.69897        331    
simulate:PDFastSim:PDFastSimPAR               4.4024e-05      22.7352       6943.44     7.5623e-05      381.011        331    
simulate:muonfilter:LArG4ParticleFilter        1.078e-05    7.58957e-05   0.00422428    2.2944e-05    0.000318823      331    
[art]:TriggerResults:TriggerResultInserter     7.681e-06    1.28681e-05     4.8e-05     1.2336e-05    4.44678e-06      331    
end_path:out1:RootOutput                       5.781e-06    9.73821e-06   0.000236367    8.952e-06    1.25928e-05      331    
end_path:out1:RootOutput(write)                2.427e-06    0.00200693     0.336917      3.786e-06     0.0223063       330    
================================================================================================================================
%MSG-i NuRandomService:  RootOutput:out1@EndJob 19-Aug-2024 20:00:18 MDT  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1
   Configured value          Last value   ModuleLabel.InstanceName
        (per event)           135540678   IonAndScint.ISCalcAlg
        (per event)           312321013   PDFastSim.photon
        (per event)            38795196   PDFastSim.scinttime
        (per event)           317736782   elecDrift
        (per event)           505276073   largeant

%MSG

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 17963.9 MB
  Peak resident set size usage (VmHWM): 14247 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 331 passed = 5 failed = 326

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport          5          5          0 out1

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 7878.012664 Real = 7895.048192

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 17963.9 VmHWM = 14247

terminate called after throwing an instance of 'cet::coded_exception<art::errors::ErrorCodes, &art::ExceptionDetail::translate[abi:cxx11]>'
  what():  ---- FatalRootError BEGIN
  Fatal Root Error: TBufferFile::AutoExpand
  Request to expand to a negative size, likely due to an integer overflow: 0x80000002 for a max of 0x7ffffffe.
  ROOT severity: 6000
---- FatalRootError END

=== End last 100 lines of lar log file ===
lar exit code 0
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_75_03d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_75_03d00/bin/extractor_prod.py", line 373, in main
    mddict = expSpecificMetadata.getmetadata()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_75_03d00/bin/extractor_prod.py", line 344, in getmetadata
    jobt = self.get_job(proc)
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_75_03d00/bin/extractor_prod.py", line 69, in get_job
    raise RuntimeError('sam_metadata_dumper returned nonzero exit status {}.'.format(rc))
RuntimeError: sam_metadata_dumper returned nonzero exit status 1.
extractor_prod.py exit code 1
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
pdjson2metadata exit code 1
.:
total 120024
-rw-r--r-- 1 dunepilot gridusers 121649913 Aug 19 20:00 RootOutput-3491-c2d7-79c3-30e2.root
-rw-r--r-- 1 dunepilot gridusers   1235719 Aug 19 20:00 000073_reco_data_2024-08-16T_161654Z_reco_2024-08-19T_234720Z.log
-rw-r--r-- 1 dunepilot gridusers      7722 Aug 19 20:00 jobscript.log
-rw-r--r-- 1 dunepilot gridusers       274 Aug 19 17:47 TFileService-f6f7-ca75-1120-682e.root
-rw-r--r-- 1 dunepilot gridusers       104 Aug 19 17:47 all-input-dids.txt
-rw-r--r-- 1 dunepilot gridusers         0 Aug 19 20:00 000073_reco_data_2024-08-16T_161654Z_reco_data_2024-08-19T_234720Z.root.ext.json
-rw-r--r-- 1 dunepilot gridusers         0 Aug 19 20:00 000073_reco_data_2024-08-16T_161654Z_reco_data_2024-08-19T_234720Z.root.json
justIN time: 2024-11-17 08:34:25 UTC       justIN version: 01.01.09