justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 59996.1@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID59996.1@justin-prod-sched02.dune.hep.ac.uk
Workflow ID2869
Stage ID1
User nameamoor@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-08-19 23:17:31
SiteUS_UChicago
EntryEngage_US_MWT2_uct2_condce_mcore
Last heartbeat2024-08-20 02:07:33
From worker nodeHostnameuct2-c579.mwt2.org
cpuinfoAMD EPYC 7302 16-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit86400 (24 hours)
Inner Apptainer?True
Job statejobscript_error
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-08-19 23:19:26
Input filesusertests:000915_reco_data_2024-08-16T_163036Z.root
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Outputting started 
Output files
Finished2024-08-20 02:07:33
Saved logsjustin-logs:59996.1-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

AFM g4 jobscript.
Input PFN = root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/f3/10/000915_reco_data_2024-08-16T_163036Z.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_4_3b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libXrdPosixPreload.so
../justin-jobscript: line 72:  1280 Aborted                 (core dumped) lar -c $FCL_FILE $events_option -o $outFile "$pfn" > ${fname}_reco_${now}.log 2>&1
=== Start last 100 lines of lar log file ===
Generating 1 particles
%MSG
%MSG-i ParticleListActionService:  larg4Main:largeant@BeginModule  19-Aug-2024 18:29:05 CDT run: 20000031 subRun: 0 event: 76
Not Stored Process summary:
	compt : 1992311
	annihil : 96760
	Pair : 72
	phot : 644150
	Brem : 591253
	conv : 96680
	Ion : 254605
%MSG
%MSG-i endOfEventAction:  larg4Main:largeant@BeginModule  19-Aug-2024 18:29:05 CDT run: 20000031 subRun: 0 event: 76 ParticleListAction.cc:701
MCTruth Handles Size: 1
%MSG
%MSG-i endOfEventAction:  larg4Main:largeant@BeginModule  19-Aug-2024 18:29:05 CDT run: 20000031 subRun: 0 event: 76 ParticleListAction.cc:708
mclistHandle Size: 1
%MSG
%MSG-i endOfEventAction:  larg4Main:largeant@BeginModule  19-Aug-2024 18:29:05 CDT run: 20000031 subRun: 0 event: 76 ParticleListAction.cc:711
Found 1 particles
%MSG
%MSG-i NuRandomService:  IonAndScint:IonAndScint@BeginModule  19-Aug-2024 18:29:05 CDT run: 20000031 subRun: 0 event: 76
Random seed for this event, engine 'IonAndScint.ISCalcAlg': 384344049
%MSG
IonAndScint Module Producer
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCPlaneUInner
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveInner
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCInner
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCPlaneVInner
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCPlaneZInner
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveOuter
%MSG-i NuRandomService:  SimDriftElectrons:elecDrift@BeginModule  19-Aug-2024 18:29:17 CDT run: 20000031 subRun: 0 event: 76
Random seed for this event, engine 'elecDrift': 401822636
%MSG
%MSG-i NuRandomService:  PDFastSimPAR:PDFastSim@BeginModule  19-Aug-2024 18:30:56 CDT run: 20000031 subRun: 0 event: 76
Random seed for this event, engine 'PDFastSim.photon': 487903501
%MSG
%MSG-i NuRandomService:  PDFastSimPAR:PDFastSim@BeginModule  19-Aug-2024 18:30:56 CDT run: 20000031 subRun: 0 event: 76
Random seed for this event, engine 'PDFastSim.scinttime': 310890478
%MSG
IonAndScint endJob.
19-Aug-2024 21:06:23 CDT  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/f3/10/000915_reco_data_2024-08-16T_163036Z.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.0129636      127.985       9525.81      0.324765       1085.19        76     
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        0.000111559   0.000264069   0.00391571    0.000150553   0.000460029      76     
simulate:rns:RandomNumberSaver                1.6791e-05     2.815e-05    0.000261241   2.33785e-05   2.78175e-05      76     
simulate:largeant:larg4Main                    0.0112116      2.13943       106.335      0.312935       12.1486        76     
simulate:IonAndScint:IonAndScint              0.000111068    0.148284       11.1115     0.000148218     1.26594        76     
simulate:elecDrift:SimDriftElectrons          3.5075e-05      1.32315       99.0192     5.34105e-05     11.2812        76     
simulate:PDFastSim:PDFastSimPAR               4.6908e-05      124.371       9309.34     6.8348e-05      1060.61        76     
simulate:muonfilter:LArG4ParticleFilter       1.2383e-05    7.50267e-05   0.00219002    2.19215e-05   0.000256628      76     
[art]:TriggerResults:TriggerResultInserter     7.945e-06    1.21541e-05   4.5075e-05    1.18575e-05   5.43395e-06      76     
end_path:out1:RootOutput                       6.673e-06    1.38819e-05   0.000293691    9.087e-06    3.26246e-05      76     
end_path:out1:RootOutput(write)                2.505e-06    0.00224371     0.167898      4.719e-06     0.0192569       75     
================================================================================================================================
%MSG-i NuRandomService:  RootOutput:out1@EndJob 19-Aug-2024 21:06:23 CDT  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1
   Configured value          Last value   ModuleLabel.InstanceName
        (per event)           384344049   IonAndScint.ISCalcAlg
        (per event)           487903501   PDFastSim.photon
        (per event)           310890478   PDFastSim.scinttime
        (per event)           401822636   elecDrift
        (per event)           410620001   largeant

%MSG

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 17858.5 MB
  Peak resident set size usage (VmHWM): 15122.9 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 76 passed = 2 failed = 74

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport          2          2          0 out1

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 9551.965967 Real = 9758.967814

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 17858.5 VmHWM = 15122.9

terminate called after throwing an instance of 'cet::coded_exception<art::errors::ErrorCodes, &art::ExceptionDetail::translate[abi:cxx11]>'
  what():  ---- FatalRootError BEGIN
  Fatal Root Error: TBufferFile::AutoExpand
  Request to expand to a negative size, likely due to an integer overflow: 0x80000002 for a max of 0x7ffffffe.
  ROOT severity: 6000
---- FatalRootError END

=== End last 100 lines of lar log file ===
lar exit code 0
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_75_03d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_75_03d00/bin/extractor_prod.py", line 373, in main
    mddict = expSpecificMetadata.getmetadata()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_75_03d00/bin/extractor_prod.py", line 344, in getmetadata
    jobt = self.get_job(proc)
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_75_03d00/bin/extractor_prod.py", line 69, in get_job
    raise RuntimeError('sam_metadata_dumper returned nonzero exit status {}.'.format(rc))
RuntimeError: sam_metadata_dumper returned nonzero exit status 1.
extractor_prod.py exit code 1
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
pdjson2metadata exit code 1
.:
total 94796
-rw-r--r--. 1 dune osgvo 96709977 Aug 19 21:06 RootOutput-544f-00c9-aaf1-a860.root
-rw-r--r--. 1 dune osgvo   333934 Aug 19 21:06 000915_reco_data_2024-08-16T_163036Z_reco_2024-08-19T_232040Z.log
-rw-r--r--. 1 dune osgvo     7717 Aug 19 21:06 jobscript.log
-rw-r--r--. 1 dune osgvo      274 Aug 19 18:22 TFileService-2da4-c452-9051-1393.root
-rw-r--r--. 1 dune osgvo      104 Aug 19 18:19 all-input-dids.txt
-rw-r--r--. 1 dune osgvo        0 Aug 19 21:06 000915_reco_data_2024-08-16T_163036Z_reco_data_2024-08-19T_232040Z.root.ext.json
-rw-r--r--. 1 dune osgvo        0 Aug 19 21:06 000915_reco_data_2024-08-16T_163036Z_reco_data_2024-08-19T_232040Z.root.json
justIN time: 2024-11-17 08:19:35 UTC       justIN version: 01.01.09