justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 59786.0@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID59786.0@justin-prod-sched02.dune.hep.ac.uk
Workflow ID2869
Stage ID1
User nameamoor@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-08-19 17:21:59
SiteUS_Colorado
EntryCMSHTPC_T3_US_Colorado_heposg01-colorado
Last heartbeat2024-08-19 19:21:40
From worker nodeHostnamelnxfarm311.colorado.edu
cpuinfoIntel(R) Core(TM) i7-7820X CPU @ 3.60GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
Inner Apptainer?True
Job statejobscript_error
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-08-19 17:23:28
Input filesusertests:000793_reco_data_2024-08-16T_162901Z.root
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Outputting started 
Output files
Finished2024-08-19 19:21:40
Saved logsjustin-logs:59786.0-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

AFM g4 jobscript.
Input PFN = root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/a9/f1/000793_reco_data_2024-08-16T_162901Z.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_4_3b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libXrdPosixPreload.so
../justin-jobscript: line 72:  1280 Segmentation fault      (core dumped) lar -c $FCL_FILE $events_option -o $outFile "$pfn" > ${fname}_reco_${now}.log 2>&1
=== Start last 100 lines of lar log file ===
Rebuild failed to get the OpDetBTRs. This is expected when running on a generation or simulation step.
%MSG
%MSG-i NuRandomService:  larg4Main:largeant@BeginModule  19-Aug-2024 11:36:09 MDT run: 20000031 subRun: 0 event: 335
Random seed for this event, engine 'largeant': 678251670
%MSG
%MSG-i generatePrimaries:  larg4Main:largeant@BeginModule  19-Aug-2024 11:36:09 MDT run: 20000031 subRun: 0 event: 335 MCTruthEventAction.cc:112
Generating 1 particles
%MSG
%MSG-i ParticleListActionService:  larg4Main:largeant@BeginModule  19-Aug-2024 11:37:16 MDT run: 20000031 subRun: 0 event: 335
Not Stored Process summary:
	compt : 1791680
	Pair : 78
	annihil : 86002
	phot : 575211
	Brem : 527959
	conv : 85924
	Ion : 226220
%MSG
%MSG-i endOfEventAction:  larg4Main:largeant@BeginModule  19-Aug-2024 11:37:16 MDT run: 20000031 subRun: 0 event: 335 ParticleListAction.cc:701
MCTruth Handles Size: 1
%MSG
%MSG-i endOfEventAction:  larg4Main:largeant@BeginModule  19-Aug-2024 11:37:16 MDT run: 20000031 subRun: 0 event: 335 ParticleListAction.cc:708
mclistHandle Size: 1
%MSG
%MSG-i endOfEventAction:  larg4Main:largeant@BeginModule  19-Aug-2024 11:37:16 MDT run: 20000031 subRun: 0 event: 335 ParticleListAction.cc:711
Found 1 particles
%MSG
%MSG-i NuRandomService:  IonAndScint:IonAndScint@BeginModule  19-Aug-2024 11:37:16 MDT run: 20000031 subRun: 0 event: 335
Random seed for this event, engine 'IonAndScint.ISCalcAlg': 497143033
%MSG
IonAndScint Module Producer
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCPlaneUInner
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveInner
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCInner
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCPlaneVInner
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCPlaneZInner
SimEnergyDeposit input module: largeant, instance name: LArG4DetectorServicevolTPCActiveOuter
%MSG-i NuRandomService:  SimDriftElectrons:elecDrift@BeginModule  19-Aug-2024 11:37:23 MDT run: 20000031 subRun: 0 event: 335
Random seed for this event, engine 'elecDrift': 238155930
%MSG
%MSG-i NuRandomService:  PDFastSimPAR:PDFastSim@BeginModule  19-Aug-2024 11:38:17 MDT run: 20000031 subRun: 0 event: 335
Random seed for this event, engine 'PDFastSim.photon': 159279575
%MSG
%MSG-i NuRandomService:  PDFastSimPAR:PDFastSim@BeginModule  19-Aug-2024 11:38:17 MDT run: 20000031 subRun: 0 event: 335
Random seed for this event, engine 'PDFastSim.scinttime': 710343838
%MSG
Plugin version SecClnt v5.4.3 is incompatible with secztn v5.6.8 (must be <= 5.4.x) in sec.protocol libXrdSecztn-5.so
IonAndScint endJob.
19-Aug-2024 13:20:19 MDT  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/a9/f1/000793_reco_data_2024-08-16T_162901Z.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                    0.00527649      20.4729       6168.41      0.184266       336.522        335    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                        6.0068e-05    0.000535126    0.0335002    0.000110443   0.00325635       335    
simulate:rns:RandomNumberSaver                1.2888e-05    2.35843e-05   0.000245391   2.2645e-05    1.2738e-05       335    
simulate:largeant:larg4Main                   0.00469361     0.583338       67.146       0.177571       3.83875        335    
simulate:IonAndScint:IonAndScint              6.1058e-05     0.0216677      6.74499     9.1045e-05     0.367962        335    
simulate:elecDrift:SimDriftElectrons          2.7553e-05     0.176951       53.8469     3.4127e-05      2.93785        335    
simulate:PDFastSim:PDFastSimPAR               3.7873e-05      19.6891       6040.68     5.2187e-05      329.57         335    
simulate:muonfilter:LArG4ParticleFilter        8.848e-06    4.79918e-05   0.00233298    1.5354e-05    0.000177471      335    
[art]:TriggerResults:TriggerResultInserter     6.268e-06    9.34519e-06   4.4427e-05     8.658e-06    2.59781e-06      335    
end_path:out1:RootOutput                       4.193e-06    7.76053e-06   0.00022663     6.892e-06    1.20184e-05      335    
end_path:out1:RootOutput(write)                2.215e-06    0.000750708    0.249766     2.6305e-06     0.0136459       334    
================================================================================================================================
%MSG-i NuRandomService:  RootOutput:out1@EndJob 19-Aug-2024 13:20:19 MDT  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1
   Configured value          Last value   ModuleLabel.InstanceName
        (per event)           497143033   IonAndScint.ISCalcAlg
        (per event)           159279575   PDFastSim.photon
        (per event)           710343838   PDFastSim.scinttime
        (per event)           238155930   elecDrift
        (per event)           678251670   largeant

%MSG

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 16445.1 MB
  Peak resident set size usage (VmHWM): 13105.5 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 335 passed = 2 failed = 333

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport          2          2          0 out1

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 6869.848134 Real = 6952.923371

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 16445.1 VmHWM = 13105.5

=== End last 100 lines of lar log file ===
lar exit code 0
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_75_03d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_75_03d00/bin/extractor_prod.py", line 373, in main
    mddict = expSpecificMetadata.getmetadata()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_75_03d00/bin/extractor_prod.py", line 344, in getmetadata
    jobt = self.get_job(proc)
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_75_03d00/bin/extractor_prod.py", line 69, in get_job
    raise RuntimeError('sam_metadata_dumper returned nonzero exit status {}.'.format(rc))
RuntimeError: sam_metadata_dumper returned nonzero exit status 1.
extractor_prod.py exit code 1
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
pdjson2metadata exit code 1
.:
total 81512
-rw-r--r-- 1 dunepilot gridusers 82195602 Aug 19 13:20 RootOutput-a614-8a5e-a488-d1ff.root
-rw-r--r-- 1 dunepilot gridusers  1249585 Aug 19 13:20 000793_reco_data_2024-08-16T_162901Z_reco_2024-08-19T_172331Z.log
-rw-r--r-- 1 dunepilot gridusers     7898 Aug 19 13:20 jobscript.log
-rw-r--r-- 1 dunepilot gridusers      274 Aug 19 11:23 TFileService-0dbe-b7e2-b851-0178.root
-rw-r--r-- 1 dunepilot gridusers      104 Aug 19 11:23 all-input-dids.txt
-rw-r--r-- 1 dunepilot gridusers        0 Aug 19 13:20 000793_reco_data_2024-08-16T_162901Z_reco_data_2024-08-19T_172331Z.root.ext.json
-rw-r--r-- 1 dunepilot gridusers        0 Aug 19 13:20 000793_reco_data_2024-08-16T_162901Z_reco_data_2024-08-19T_172331Z.root.json
justIN time: 2024-11-17 14:31:48 UTC       justIN version: 01.01.09