justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 271945.1@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID271945.1@justin-prod-sched01.dune.hep.ac.uk
Workflow ID3652
Stage ID1
User nameamoor@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-10-09 02:53:15
SiteUK_RAL-Tier1
EntryLIGO_UK_RAL_arc_ce01
Last heartbeat2024-10-09 08:13:31
From worker nodeHostnamedune001-3163484.0-lcg2580.gridpp.rl.ac.uk
cpuinfoAMD EPYC 7452 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit216000 (60 hours)
Inner Apptainer?True
Job statejobscript_error
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-10-09 02:55:13
Input filesusertests:000504_reco_data_2024-10-07T_115700Z.root
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Outputting started 
Output files
Finished2024-10-09 08:13:31
Saved logsjustin-logs:271945.1-justin-prod-sched01.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

ciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/Matrix.pcm
14b806400000-14b806488000 r-xp 00000000 00:151 286003                    /cvmfs/larsoft.opensciencegrid.org/products/canvas/v3_14_00/slf7.x86_64.e20.prof/lib/libcanvas.so
14b806488000-14b806687000 ---p 00088000 00:151 286003                    /cvmfs/larsoft.opensciencegrid.org/products/canvas/v3_14_00/slf7.x86_64.e20.prof/lib/libcanvas.so
14b806687000-14b806689000 r--p 00087000 00:151 286003                    /cvmfs/larsoft.opensciencegrid.org/products/canvas/v3_14_00/slf7.x86_64.e20.prof/lib/libcanvas.so
14b806689000-14b80668b000 rw-p 00089000 00:151 286003                    /cvmfs/larsoft.opensciencegrid.org/products/canvas/v3_14_00/slf7.x86_64.e20.prof/lib/libcanvas.so
14b80668b000-14b80668c000 rw-p 00000000 00:00 0 
14b8066e9000-14b806800000 r--p 00000000 00:151 194627                    /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/Net.pcm
14b806800000-14b806843000 r-xp 00000000 00:151 694198                    /cvmfs/larsoft.opensciencegrid.org/products/art/v3_12_00/slf7.x86_64.e20.prof/lib/libart_Utilities.so
14b806843000-14b806a42000 ---p 00043000 00:151 694198                    /cvmfs/larsoft.opensciencegrid.org/products/art/v3_12_00/slf7.x86_64.e20.prof/lib/libart_Utilities.so
14b806a42000-14b806a43000 r--p 00042000 00:151 694198                    /cvmfs/larsoft.opensciencegrid.org/products/art/v3_12_00/slf7.x86_64.e20.prof/lib/libart_Utilities.so
14b806a43000-14b806a44000 rw-p 00043000 00:151 694198                    /cvmfs/larsoft.opensciencegrid.org/products/art/v3_12_00/slf7.x86_64.e20.prof/lib/libart_Utilities.so
14b806a44000-14b806a45000 rw-p 00000000 00:00 0 
14b806a4e000-14b806c00000 r--p 00000000 00:151 194295                    /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/Imt.pcm
14b806c00000-14b806c63000 r-xp 00000000 00:151 298031                    /cvmfs/larsoft.opensciencegrid.org/products/boost/v1_80_0/Linux64bit+3.10-2.17-e20-prof/lib/libboost_program_options.so.1.80.0
14b806c63000-14b806e62000 ---p 00063000 00:151 298031                    /cvmfs/larsoft.opensciencegrid.org/products/boost/v1_80_0/Linux64bit+3.10-2.17-e20-prof/lib/libboost_program_options.so.1.80.0
14b806e62000-14b806e65000 r--p 00062000 00:151 298031                    /cvmfs/larsoft.opensciencegrid.org/products/boost/v1_80_0/Linux64bit+3.10-2.17-e20-prof/lib/libboost_program_options.so.1.80.0
14b806e65000-14b806e66000 rw-p 00065000 00:151 298031                    /cvmfs/larsoft.opensciencegrid.org/products/boost/v1_80_0/Linux64bit+3.10-2.17-e20-prof/lib/libboost_program_options.so.1.80.0
14b806e76000-14b806e77000 rw-p 00000000 00:00 0 
14b806e77000-14b806e80000 r-xp 00000000 00:00 0 
14b806e80000-14b807000000 r--p 00000000 00:151 194448                    /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/Thread.pcm
14b807000000-14b8070d8000 r-xp 00000000 00:151 694205                    /cvmfs/larsoft.opensciencegrid.org/products/art/v3_12_00/slf7.x86_64.e20.prof/lib/libart_Framework_Art.so
14b8070d8000-14b8072d8000 ---p 000d8000 00:151 694205                    /cvmfs/larsoft.opensciencegrid.org/products/art/v3_12_00/slf7.x86_64.e20.prof/lib/libart_Framework_Art.so
14b8072d8000-14b8072db000 r--p 000d8000 00:151 694205                    /cvmfs/larsoft.opensciencegrid.org/products/art/v3_12_00/slf7.x86_64.e20.prof/lib/libart_Framework_Art.so
14b8072db000-14b8072dd000 rw-p 000db000 00:151 694205                    /cvmfs/larsoft.opensciencegrid.org/products/art/v3_12_00/slf7.x86_64.e20.prof/lib/libart_Framework_Art.so
14b8072dd000-14b8072e1000 r-xp 00000000 00:00 0 
14b8072e1000-14b807400000 r--p 00000000 00:151 194468                    /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/MultiProc.pcm
14b807400000-14b80740f000 r-xp 00000000 00:151 270586                    /cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_4_3b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libXrdPosixPreload.so.2.0.0
14b80740f000-14b80760e000 ---p 0000f000 00:151 270586                    /cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_4_3b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libXrdPosixPreload.so.2.0.0
14b80760e000-14b80760f000 r--p 0000e000 00:151 270586                    /cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_4_3b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libXrdPosixPreload.so.2.0.0
14b80760f000-14b807610000 rw-p 0000f000 00:151 270586                    /cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_4_3b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libXrdPosixPreload.so.2.0.0
14b807610000-14b807627000 r-xp 00000000 00:00 0 
14b807627000-14b80762e000 r--s 00000000 00:34 4866412                    /usr/lib64/gconv/gconv-modules.cache
14b80762e000-14b807635000 r-xp 00000000 00:00 0 
14b807635000-14b8076c1000 r--p 00000000 00:151 194660                    /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/Unfold.pcm
14b8076c1000-14b80772b000 r--p 00000000 00:151 194027                    /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/Rint.pcm
14b80772b000-14b8077be000 r--p 00000000 00:151 199120                    /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/ROOT_Foundation_Stage1_NoRTTI.pcm
14b8077be000-14b8077dc000 r--p 00000000 00:151 196024                    /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/ROOT_Config.pcm
14b8077dc000-14b807800000 r--p 00000000 00:151 199173                    /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/ROOT_Rtypes.pcm
14b807800000-14b807822000 r-xp 00000000 00:34 4554812                    /usr/lib64/ld-2.17.so
14b807822000-14b807824000 r-xp 00000000 00:00 0 
14b807824000-14b807877000 r--p 00000000 00:151 195352                    /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/ROOT_Foundation_C.pcm
14b807877000-14b807894000 r--p 00000000 00:151 197743                    /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/Cling_Runtime_Extra.pcm
14b807894000-14b8078de000 r--p 00000000 00:151 196724                    /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/modules.idx
14b8078de000-14b807a21000 r--p 00000000 00:151 194773                    /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libc.pcm
14b807a21000-14b807a22000 r--p 00021000 00:34 4554812                    /usr/lib64/ld-2.17.so
14b807a22000-14b807a23000 rw-p 00022000 00:34 4554812                    /usr/lib64/ld-2.17.so
14b807a23000-14b807a24000 rw-p 00000000 00:00 0 
14b807a24000-14b807a25000 r-xp 00000000 00:00 0 
14b807a25000-14b807a2a000 r-xp 00000000 00:00 0 
14b807a2a000-14b807a49000 r--p 00000000 00:151 198585                    /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/Cling_Runtime.pcm
14b807a49000-14b807a56000 r--p 00000000 00:151 198053                    /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/_Builtin_stddef_max_align_t.pcm
14b807a56000-14b807a64000 r--p 00000000 00:151 195500                    /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/xlocale.h.pcm
14b807a64000-14b807ae2000 rw-p 00000000 00:00 0 
14b807ae2000-14b807ae3000 r-xp 00000000 00:00 0 
14b807ae3000-14b807aeb000 r-xp 00000000 00:00 0 
14b807aeb000-14b807aec000 rw-p 00000000 00:00 0 
14b807aec000-14b807aee000 r-xp 00000000 00:00 0 
14b807aee000-14b807afa000 rw-p 00000000 00:00 0 
14b807afa000-14b807afe000 r--p 00000000 00:00 0                          [vvar]
14b807afe000-14b807b00000 r-xp 00000000 00:00 0                          [vdso]
7fffdc268000-7fffdc2a7000 rw-p 00000000 00:00 0                          [stack]
ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0                  [vsyscall]
=== End last 100 lines of lar log file ===
lar exit code 0
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_75_03d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_75_03d00/bin/extractor_prod.py", line 373, in main
    mddict = expSpecificMetadata.getmetadata()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_75_03d00/bin/extractor_prod.py", line 344, in getmetadata
    jobt = self.get_job(proc)
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_75_03d00/bin/extractor_prod.py", line 69, in get_job
    raise RuntimeError('sam_metadata_dumper returned nonzero exit status {}.'.format(rc))
RuntimeError: sam_metadata_dumper returned nonzero exit status 1.
extractor_prod.py exit code 1
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
pdjson2metadata exit code 1
.:
total 181216
-rw-r--r-- 1 dune001 dune 183524656 Oct  9 08:12 RootOutput-e8dd-8bcc-c144-e11c.root
-rw-r--r-- 1 dune001 dune   2011903 Oct  9 08:13 000504_reco_data_2024-10-07T_115700Z_reco_2024-10-09T_025530Z.log
-rw-r--r-- 1 dune001 dune     16117 Oct  9 08:13 jobscript.log
-rw-r--r-- 1 dune001 dune       274 Oct  9 02:55 TFileService-bd1b-f5d3-2d67-f2f4.root
-rw-r--r-- 1 dune001 dune       104 Oct  9 02:55 all-input-dids.txt
-rw-r--r-- 1 dune001 dune         0 Oct  9 08:13 000504_reco_data_2024-10-07T_115700Z_reco_data_2024-10-09T_025530Z.root.ext.json
-rw-r--r-- 1 dune001 dune         0 Oct  9 08:13 000504_reco_data_2024-10-07T_115700Z_reco_data_2024-10-09T_025530Z.root.json
justIN time: 2024-11-24 07:37:32 UTC       justIN version: 01.01.09