justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 245014.0@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID245014.0@justin-prod-sched01.dune.hep.ac.uk
Workflow ID2869
Stage ID1
User nameamoor@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-08-19 19:20:17
SiteUS_Colorado
EntryCMSHTPC_T3_US_Colorado_heposg01-colorado
Last heartbeat2024-08-19 21:20:11
From worker nodeHostnamelnxfarm311.colorado.edu
cpuinfoIntel(R) Core(TM) i7-7820X CPU @ 3.60GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
Inner Apptainer?True
Job statejobscript_error
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-08-19 19:21:10
Input filesusertests:000793_reco_data_2024-08-16T_162901Z.root
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Outputting started 
Output files
Finished2024-08-19 21:20:11
Saved logsjustin-logs:245014.0-justin-prod-sched01.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

AFM g4 jobscript.
Input PFN = root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/a9/f1/000793_reco_data_2024-08-16T_162901Z.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_4_3b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libXrdPosixPreload.so
../justin-jobscript: line 72:  1280 Segmentation fault      (core dumped) lar -c $FCL_FILE $events_option -o $outFile "$pfn" > ${fname}_reco_${now}.log 2>&1
=== Start last 100 lines of lar log file ===
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-i NuRandomService:  larg4Main:largeant@BeginModule  19-Aug-2024 13:35:46 MDT run: 20000031 subRun: 0 event: 335
Random seed for this event, engine 'largeant': 678251670
%MSG
%MSG-i generatePrimaries:  larg4Main:largeant@BeginModule  19-Aug-2024 13:35:46 MDT run: 20000031 subRun: 0 event: 335 MCTruthEventAction.cc:112
Generating 1 particles
%MSG
%MSG-i ParticleListActionService:  larg4Main:largeant@BeginModule  19-Aug-2024 13:37:03 MDT run: 20000031 subRun: 0 event: 335
Not Stored Process summary:
	compt : 1791680
	Pair : 78
	annihil : 86002
	phot : 575211
	Brem : 527959
	conv : 85924
	Ion : 226220
%MSG
%MSG-i endOfEventAction:  larg4Main:largeant@BeginModule  19-Aug-2024 13:37:03 MDT run: 20000031 subRun: 0 event: 335 ParticleListAction.cc:701
MCTruth Handles Size: 1
%MSG
%MSG-i endOfEventAction:  larg4Main:largeant@BeginModule  19-Aug-2024 13:37:03 MDT run: 20000031 subRun: 0 event: 335 ParticleListAction.cc:708
mclistHandle Size: 1
%MSG
%MSG-i endOfEventAction:  larg4Main:largeant@BeginModule  19-Aug-2024 13:37:03 MDT run: 20000031 subRun: 0 event: 335 ParticleListAction.cc:711
Found 1 particles
%MSG
%MSG-i NuRandomService:  IonAndScint:IonAndScint@BeginModule  19-Aug-2024 13:37:03 MDT run: 20000031 subRun: 0 event: 335
Random seed for this event, engine 'IonAndScint.ISCalcAlg': 497143033
%MSG
IonAndScint Module Producer
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCPlaneUInner
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveInner
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCInner
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCPlaneVInner
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCPlaneZInner
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveOuter
%MSG-i NuRandomService:  SimDriftElectrons:elecDrift@BeginModule  19-Aug-2024 13:37:10 MDT run: 20000031 subRun: 0 event: 335
Random seed for this event, engine 'elecDrift': 238155930
%MSG
%MSG-i NuRandomService:  PDFastSimPAR:PDFastSim@BeginModule  19-Aug-2024 13:38:04 MDT run: 20000031 subRun: 0 event: 335
Random seed for this event, engine 'PDFastSim.photon': 159279575
%MSG
%MSG-i NuRandomService:  PDFastSimPAR:PDFastSim@BeginModule  19-Aug-2024 13:38:04 MDT run: 20000031 subRun: 0 event: 335
Random seed for this event, engine 'PDFastSim.scinttime': 710343838
%MSG
Plugin version SecClnt v5.4.3 is incompatible with secztn v5.6.8 (must be <= 5.4.x) in sec.protocol libXrdSecztn-5.so
IonAndScint endJob.
19-Aug-2024 15:19:52 MDT  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/a9/f1/000793_reco_data_2024-08-16T_162901Z.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                    0.00601184      20.7817        6169        0.197739       336.664        335    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        6.7344e-05     0.0005259     0.0259773    0.000119967   0.00305622       335    
simulate:rns:RandomNumberSaver                1.2788e-05    2.50337e-05   0.000188483   2.3983e-05    1.02771e-05      335    
simulate:largeant:larg4Main                   0.00542925     0.637676       77.4524      0.186936       4.3872         335    
simulate:IonAndScint:IonAndScint              6.3454e-05     0.0218393      6.73691     9.5422e-05     0.367602        335    
simulate:elecDrift:SimDriftElectrons          2.7173e-05     0.178289       53.6497     3.6032e-05      2.92811        335    
simulate:PDFastSim:PDFastSimPAR               3.8867e-05      19.942        6031.16     5.4936e-05      329.158        335    
simulate:muonfilter:LArG4ParticleFilter       1.0984e-05    5.17597e-05   0.00236576    1.7278e-05    0.000189052      335    
[art]:TriggerResults:TriggerResultInserter     6.524e-06    9.98965e-06   3.1517e-05     9.099e-06    2.46896e-06      335    
end_path:out1:RootOutput                       4.517e-06    8.08305e-06   0.000149576    7.438e-06    7.82234e-06      335    
end_path:out1:RootOutput(write)                2.151e-06    0.000899482    0.299432      2.643e-06     0.0163595       334    
================================================================================================================================
%MSG-i NuRandomService:  RootOutput:out1@EndJob 19-Aug-2024 15:19:52 MDT  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1
   Configured value          Last value   ModuleLabel.InstanceName
        (per event)           497143033   IonAndScint.ISCalcAlg
        (per event)           159279575   PDFastSim.photon
        (per event)           710343838   PDFastSim.scinttime
        (per event)           238155930   elecDrift
        (per event)           678251670   largeant

%MSG

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 16446.4 MB
  Peak resident set size usage (VmHWM): 13229.9 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 335 passed = 2 failed = 333

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport          2          2          0 out1

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 6971.510031 Real = 7052.773730

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 16446.4 VmHWM = 13229.9

=== End last 100 lines of lar log file ===
lar exit code 0
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_75_03d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_75_03d00/bin/extractor_prod.py", line 373, in main
    mddict = expSpecificMetadata.getmetadata()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_75_03d00/bin/extractor_prod.py", line 344, in getmetadata
    jobt = self.get_job(proc)
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_75_03d00/bin/extractor_prod.py", line 69, in get_job
    raise RuntimeError('sam_metadata_dumper returned nonzero exit status {}.'.format(rc))
RuntimeError: sam_metadata_dumper returned nonzero exit status 1.
extractor_prod.py exit code 1
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
pdjson2metadata exit code 1
.:
total 81512
-rw-r--r-- 1 dunepilot gridusers 82195603 Aug 19 15:19 RootOutput-982d-971c-a356-44c8.root
-rw-r--r-- 1 dunepilot gridusers  1249585 Aug 19 15:19 000793_reco_data_2024-08-16T_162901Z_reco_2024-08-19T_192114Z.log
-rw-r--r-- 1 dunepilot gridusers     7898 Aug 19 15:20 jobscript.log
-rw-r--r-- 1 dunepilot gridusers      274 Aug 19 13:21 TFileService-afcd-92bb-b84e-d657.root
-rw-r--r-- 1 dunepilot gridusers      104 Aug 19 13:21 all-input-dids.txt
-rw-r--r-- 1 dunepilot gridusers        0 Aug 19 15:19 000793_reco_data_2024-08-16T_162901Z_reco_data_2024-08-19T_192114Z.root.ext.json
-rw-r--r-- 1 dunepilot gridusers        0 Aug 19 15:20 000793_reco_data_2024-08-16T_162901Z_reco_data_2024-08-19T_192114Z.root.json
justIN time: 2024-11-17 14:29:29 UTC       justIN version: 01.01.09