justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 271761.0@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID271761.0@justin-prod-sched01.dune.hep.ac.uk
Workflow ID3652
Stage ID1
User nameamoor@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-10-08 15:43:01
SiteCZ_FZU
EntryNova_CZ_FZU_osgce2
Last heartbeat2024-10-08 20:28:21
From worker nodeHostnamemff-magic17.farm.particle.cz
cpuinfoIntel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit172800 (48 hours)
Inner Apptainer?True
Job statejobscript_error
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-10-08 15:53:39
Input filesusertests:000504_reco_data_2024-10-07T_115700Z.root
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Outputting started 
Output files
Finished2024-10-08 20:28:21
Saved logsjustin-logs:271761.0-justin-prod-sched01.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

/cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/XMLParser.pcm
152f93ae9000-152f93c00000 r--p 00000000 00:33 739404                     /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/Net.pcm
152f93c00000-152f93c88000 r-xp 00000000 00:33 826534                     /cvmfs/larsoft.opensciencegrid.org/products/canvas/v3_14_00/slf7.x86_64.e20.prof/lib/libcanvas.so
152f93c88000-152f93e87000 ---p 00088000 00:33 826534                     /cvmfs/larsoft.opensciencegrid.org/products/canvas/v3_14_00/slf7.x86_64.e20.prof/lib/libcanvas.so
152f93e87000-152f93e89000 r--p 00087000 00:33 826534                     /cvmfs/larsoft.opensciencegrid.org/products/canvas/v3_14_00/slf7.x86_64.e20.prof/lib/libcanvas.so
152f93e89000-152f93e8b000 rw-p 00089000 00:33 826534                     /cvmfs/larsoft.opensciencegrid.org/products/canvas/v3_14_00/slf7.x86_64.e20.prof/lib/libcanvas.so
152f93e8b000-152f93e8c000 rw-p 00000000 00:00 0 
152f93ee1000-152f94000000 r--p 00000000 00:33 739245                     /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/MultiProc.pcm
152f94000000-152f94043000 r-xp 00000000 00:33 1190147                    /cvmfs/larsoft.opensciencegrid.org/products/art/v3_12_00/slf7.x86_64.e20.prof/lib/libart_Utilities.so
152f94043000-152f94242000 ---p 00043000 00:33 1190147                    /cvmfs/larsoft.opensciencegrid.org/products/art/v3_12_00/slf7.x86_64.e20.prof/lib/libart_Utilities.so
152f94242000-152f94243000 r--p 00042000 00:33 1190147                    /cvmfs/larsoft.opensciencegrid.org/products/art/v3_12_00/slf7.x86_64.e20.prof/lib/libart_Utilities.so
152f94243000-152f94244000 rw-p 00043000 00:33 1190147                    /cvmfs/larsoft.opensciencegrid.org/products/art/v3_12_00/slf7.x86_64.e20.prof/lib/libart_Utilities.so
152f94244000-152f94245000 rw-p 00000000 00:00 0 
152f9424e000-152f94400000 r--p 00000000 00:33 739072                     /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/Imt.pcm
152f94400000-152f94463000 r-xp 00000000 00:33 838219                     /cvmfs/larsoft.opensciencegrid.org/products/boost/v1_80_0/Linux64bit+3.10-2.17-e20-prof/lib/libboost_program_options.so.1.80.0
152f94463000-152f94662000 ---p 00063000 00:33 838219                     /cvmfs/larsoft.opensciencegrid.org/products/boost/v1_80_0/Linux64bit+3.10-2.17-e20-prof/lib/libboost_program_options.so.1.80.0
152f94662000-152f94665000 r--p 00062000 00:33 838219                     /cvmfs/larsoft.opensciencegrid.org/products/boost/v1_80_0/Linux64bit+3.10-2.17-e20-prof/lib/libboost_program_options.so.1.80.0
152f94665000-152f94666000 rw-p 00065000 00:33 838219                     /cvmfs/larsoft.opensciencegrid.org/products/boost/v1_80_0/Linux64bit+3.10-2.17-e20-prof/lib/libboost_program_options.so.1.80.0
152f94680000-152f94800000 r--p 00000000 00:33 739225                     /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/Thread.pcm
152f94800000-152f948d8000 r-xp 00000000 00:33 1190154                    /cvmfs/larsoft.opensciencegrid.org/products/art/v3_12_00/slf7.x86_64.e20.prof/lib/libart_Framework_Art.so
152f948d8000-152f94ad8000 ---p 000d8000 00:33 1190154                    /cvmfs/larsoft.opensciencegrid.org/products/art/v3_12_00/slf7.x86_64.e20.prof/lib/libart_Framework_Art.so
152f94ad8000-152f94adb000 r--p 000d8000 00:33 1190154                    /cvmfs/larsoft.opensciencegrid.org/products/art/v3_12_00/slf7.x86_64.e20.prof/lib/libart_Framework_Art.so
152f94adb000-152f94add000 rw-p 000db000 00:33 1190154                    /cvmfs/larsoft.opensciencegrid.org/products/art/v3_12_00/slf7.x86_64.e20.prof/lib/libart_Framework_Art.so
152f94b0a000-152f94b96000 r--p 00000000 00:33 739437                     /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/Unfold.pcm
152f94b96000-152f94c00000 r--p 00000000 00:33 738804                     /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/Rint.pcm
152f94c00000-152f94c0f000 r-xp 00000000 00:33 815363                     /cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_4_3b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libXrdPosixPreload.so.2.0.0
152f94c0f000-152f94e0e000 ---p 0000f000 00:33 815363                     /cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_4_3b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libXrdPosixPreload.so.2.0.0
152f94e0e000-152f94e0f000 r--p 0000e000 00:33 815363                     /cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_4_3b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libXrdPosixPreload.so.2.0.0
152f94e0f000-152f94e10000 rw-p 0000f000 00:33 815363                     /cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_4_3b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libXrdPosixPreload.so.2.0.0
152f94e25000-152f94e26000 rw-p 00000000 00:00 0 
152f94e26000-152f94e4a000 r-xp 00000000 00:00 0 
152f94e4a000-152f94e51000 r--s 00000000 00:3b 17329                      /usr/lib64/gconv/gconv-modules.cache
152f94e51000-152f94e52000 r-xp 00000000 00:00 0 
152f94e52000-152f94ee5000 r--p 00000000 00:33 743897                     /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/ROOT_Foundation_Stage1_NoRTTI.pcm
152f94ee5000-152f94f03000 r--p 00000000 00:33 740801                     /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/ROOT_Config.pcm
152f94f03000-152f94f27000 r--p 00000000 00:33 743950                     /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/ROOT_Rtypes.pcm
152f94f27000-152f94f7a000 r--p 00000000 00:33 740129                     /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/ROOT_Foundation_C.pcm
152f94f7a000-152f94f97000 r--p 00000000 00:33 742520                     /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/Cling_Runtime_Extra.pcm
152f94f97000-152f94fb6000 r--p 00000000 00:33 743362                     /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/Cling_Runtime.pcm
152f94fb6000-152f95000000 r--p 00000000 00:33 741501                     /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/modules.idx
152f95000000-152f95022000 r-xp 00000000 00:3b 17927                      /usr/lib64/ld-2.17.so
152f95022000-152f9502e000 r-xp 00000000 00:00 0 
152f9502e000-152f9503b000 r--p 00000000 00:33 742830                     /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/_Builtin_stddef_max_align_t.pcm
152f9503b000-152f95049000 r--p 00000000 00:33 740277                     /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/xlocale.h.pcm
152f95049000-152f9518c000 r--p 00000000 00:33 739550                     /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libc.pcm
152f9518c000-152f95221000 rw-p 00000000 00:00 0 
152f95221000-152f95222000 r--p 00021000 00:3b 17927                      /usr/lib64/ld-2.17.so
152f95222000-152f95223000 rw-p 00022000 00:3b 17927                      /usr/lib64/ld-2.17.so
152f95223000-152f95224000 rw-p 00000000 00:00 0 
152f95224000-152f95225000 r-xp 00000000 00:00 0 
152f95225000-152f95226000 r-xp 00000000 00:00 0 
152f95226000-152f95250000 rw-p 00000000 00:00 0 
152f95250000-152f95251000 r-xp 00000000 00:00 0 
152f95251000-152f95259000 r-xp 00000000 00:00 0 
152f95259000-152f9525a000 rw-p 00000000 00:00 0 
152f9525a000-152f9525c000 r-xp 00000000 00:00 0 
152f9525c000-152f95268000 rw-p 00000000 00:00 0 
7fffbbc06000-7fffbbc45000 rw-p 00000000 00:00 0                          [stack]
7fffbbd91000-7fffbbd95000 r--p 00000000 00:00 0                          [vvar]
7fffbbd95000-7fffbbd97000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0                  [vsyscall]
=== End last 100 lines of lar log file ===
lar exit code 0
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_75_03d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_75_03d00/bin/extractor_prod.py", line 373, in main
    mddict = expSpecificMetadata.getmetadata()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_75_03d00/bin/extractor_prod.py", line 344, in getmetadata
    jobt = self.get_job(proc)
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_75_03d00/bin/extractor_prod.py", line 69, in get_job
    raise RuntimeError('sam_metadata_dumper returned nonzero exit status {}.'.format(rc))
RuntimeError: sam_metadata_dumper returned nonzero exit status 1.
extractor_prod.py exit code 1
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
pdjson2metadata exit code 1
.:
total 181228
-rw-r--r-- 1 fermilab users 183524656 Oct  8 22:26 RootOutput-f4a0-e7b9-f7f9-e384.root
-rw-r--r-- 1 fermilab users   2018872 Oct  8 22:27 000504_reco_data_2024-10-07T_115700Z_reco_2024-10-08T_155345Z.log
-rw-r--r-- 1 fermilab users     16119 Oct  8 22:27 jobscript.log
-rw-r--r-- 1 fermilab users       274 Oct  8 17:54 TFileService-9411-83d9-9390-f7a1.root
-rw-r--r-- 1 fermilab users       104 Oct  8 17:53 all-input-dids.txt
-rw-r--r-- 1 fermilab users         0 Oct  8 22:27 000504_reco_data_2024-10-07T_115700Z_reco_data_2024-10-08T_155345Z.root.ext.json
-rw-r--r-- 1 fermilab users         0 Oct  8 22:27 000504_reco_data_2024-10-07T_115700Z_reco_data_2024-10-08T_155345Z.root.json
justIN time: 2024-11-24 09:56:37 UTC       justIN version: 01.01.09