justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 271474.119@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID271474.119@justin-prod-sched01.dune.hep.ac.uk
Workflow ID3652
Stage ID1
User nameamoor@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-10-08 08:32:27
SiteUK_Liverpool
EntryDUNE_UK_Liverpool_hepgrid5
Last heartbeat2024-10-08 11:12:26
From worker nodeHostnamer12-n06.ph.liv.ac.uk
cpuinfoAMD EPYC 7453 28-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit259200 (72 hours)
Inner Apptainer?True
Job statejobscript_error
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-10-08 09:02:38
Input filesusertests:000135_reco_data_2024-10-07T_114846Z.root
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Outputting started 
Output files
Finished2024-10-08 11:12:26
Saved logsjustin-logs:271474.119-justin-prod-sched01.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

org/products/dk2nudata/v01_10_01d/Linux64bit+3.10-2.17-e20-prof/lib/libdk2nuTree.so
2ad954e04000-2ad954e06000 rw-p 00065000 00:192 2973211                   /cvmfs/larsoft.opensciencegrid.org/products/dk2nudata/v01_10_01d/Linux64bit+3.10-2.17-e20-prof/lib/libdk2nuTree.so
2ad954e06000-2ad954e08000 rw-p 00000000 00:00 0 
2ad954e08000-2ad954ee4000 r-xp 00000000 00:192 3003594                   /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libROOTDataFrame.so.6.26.07
2ad954ee4000-2ad9550e3000 ---p 000dc000 00:192 3003594                   /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libROOTDataFrame.so.6.26.07
2ad9550e3000-2ad9550e7000 r--p 000db000 00:192 3003594                   /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libROOTDataFrame.so.6.26.07
2ad9550e7000-2ad9550e9000 rw-p 000df000 00:192 3003594                   /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libROOTDataFrame.so.6.26.07
2ad9550e9000-2ad9550ec000 rw-p 00000000 00:00 0 
2ad9550ec000-2ad955263000 r-xp 00000000 00:192 3004332                   /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libTreePlayer.so.6.26.07
2ad955263000-2ad955462000 ---p 00177000 00:192 3004332                   /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libTreePlayer.so.6.26.07
2ad955462000-2ad95546d000 r--p 00176000 00:192 3004332                   /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libTreePlayer.so.6.26.07
2ad95546d000-2ad955470000 rw-p 00181000 00:192 3004332                   /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libTreePlayer.so.6.26.07
2ad955470000-2ad955476000 rw-p 00000000 00:00 0 
2ad955476000-2ad955519000 r-xp 00000000 00:192 3002364                   /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libROOTVecOps.so.6.26.07
2ad955519000-2ad955719000 ---p 000a3000 00:192 3002364                   /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libROOTVecOps.so.6.26.07
2ad955719000-2ad95571b000 r--p 000a3000 00:192 3002364                   /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libROOTVecOps.so.6.26.07
2ad95571b000-2ad95571c000 rw-p 000a5000 00:192 3002364                   /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libROOTVecOps.so.6.26.07
2ad95571c000-2ad95571d000 rw-p 00000000 00:00 0 
2ad95571d000-2ad9557e7000 r-xp 00000000 00:192 3002994                   /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libROOTNTuple.so.6.26.07
2ad9557e7000-2ad9559e7000 ---p 000ca000 00:192 3002994                   /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libROOTNTuple.so.6.26.07
2ad9559e7000-2ad9559ea000 r--p 000ca000 00:192 3002994                   /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libROOTNTuple.so.6.26.07
2ad9559ea000-2ad9559ec000 rw-p 000cd000 00:192 3002994                   /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libROOTNTuple.so.6.26.07
2ad9559ec000-2ad955a04000 r-xp 00000000 00:192 3503687                   /cvmfs/larsoft.opensciencegrid.org/products/larcoreobj/v09_09_00/slf7.x86_64.e20.prof/lib/liblarcoreobj_SummaryData_dict.so
2ad955a04000-2ad955c04000 ---p 00018000 00:192 3503687                   /cvmfs/larsoft.opensciencegrid.org/products/larcoreobj/v09_09_00/slf7.x86_64.e20.prof/lib/liblarcoreobj_SummaryData_dict.so
2ad955c04000-2ad955c05000 r--p 00018000 00:192 3503687                   /cvmfs/larsoft.opensciencegrid.org/products/larcoreobj/v09_09_00/slf7.x86_64.e20.prof/lib/liblarcoreobj_SummaryData_dict.so
2ad955c05000-2ad955c06000 rw-p 00019000 00:192 3503687                   /cvmfs/larsoft.opensciencegrid.org/products/larcoreobj/v09_09_00/slf7.x86_64.e20.prof/lib/liblarcoreobj_SummaryData_dict.so
2ad955c06000-2ad95600a000 rw-p 00000000 00:00 0 
2ad95600a000-2ad9560f3000 r-xp 00000000 00:192 3503420                   /cvmfs/larsoft.opensciencegrid.org/products/lardataobj/v09_16_01/slf7.x86_64.e20.prof/lib/liblardataobj_Simulation_dict.so
2ad9560f3000-2ad9562f2000 ---p 000e9000 00:192 3503420                   /cvmfs/larsoft.opensciencegrid.org/products/lardataobj/v09_16_01/slf7.x86_64.e20.prof/lib/liblardataobj_Simulation_dict.so
2ad9562f2000-2ad9562f7000 r--p 000e8000 00:192 3503420                   /cvmfs/larsoft.opensciencegrid.org/products/lardataobj/v09_16_01/slf7.x86_64.e20.prof/lib/liblardataobj_Simulation_dict.so
2ad9562f7000-2ad9562f9000 rw-p 000ed000 00:192 3503420                   /cvmfs/larsoft.opensciencegrid.org/products/lardataobj/v09_16_01/slf7.x86_64.e20.prof/lib/liblardataobj_Simulation_dict.so
2ad9562f9000-2ad9562fe000 rw-p 00000000 00:00 0 
2ad9562fe000-2ad956300000 r-xp 00000000 00:192 3503073                   /cvmfs/larsoft.opensciencegrid.org/products/lardataobj/v09_16_01/slf7.x86_64.e20.prof/lib/liblardataobj_Simulation_Compatibility.so
2ad956300000-2ad9564ff000 ---p 00002000 00:192 3503073                   /cvmfs/larsoft.opensciencegrid.org/products/lardataobj/v09_16_01/slf7.x86_64.e20.prof/lib/liblardataobj_Simulation_Compatibility.so
2ad9564ff000-2ad956500000 r--p 00001000 00:192 3503073                   /cvmfs/larsoft.opensciencegrid.org/products/lardataobj/v09_16_01/slf7.x86_64.e20.prof/lib/liblardataobj_Simulation_Compatibility.so
2ad956500000-2ad956501000 rw-p 00002000 00:192 3503073                   /cvmfs/larsoft.opensciencegrid.org/products/lardataobj/v09_16_01/slf7.x86_64.e20.prof/lib/liblardataobj_Simulation_Compatibility.so
2ad956501000-2ad95650b000 r-xp 00000000 00:192 3503676                   /cvmfs/larsoft.opensciencegrid.org/products/larcoreobj/v09_09_00/slf7.x86_64.e20.prof/lib/liblarcoreobj_SimpleTypesAndConstants_dict.so
2ad95650b000-2ad95670b000 ---p 0000a000 00:192 3503676                   /cvmfs/larsoft.opensciencegrid.org/products/larcoreobj/v09_09_00/slf7.x86_64.e20.prof/lib/liblarcoreobj_SimpleTypesAndConstants_dict.so
2ad95670b000-2ad95670c000 r--p 0000a000 00:192 3503676                   /cvmfs/larsoft.opensciencegrid.org/products/larcoreobj/v09_09_00/slf7.x86_64.e20.prof/lib/liblarcoreobj_SimpleTypesAndConstants_dict.so
2ad95670c000-2ad95670d000 rw-p 0000b000 00:192 3503676                   /cvmfs/larsoft.opensciencegrid.org/products/larcoreobj/v09_09_00/slf7.x86_64.e20.prof/lib/liblarcoreobj_SimpleTypesAndConstants_dict.so
2ad95670d000-2ad956778000 r-xp 00000000 00:192 3441325                   /cvmfs/larsoft.opensciencegrid.org/products/canvas_root_io/v1_11_02/slf7.x86_64.e20.prof/lib/libcanvas_root_io_Dictionaries_std_dict.so
2ad956778000-2ad956978000 ---p 0006b000 00:192 3441325                   /cvmfs/larsoft.opensciencegrid.org/products/canvas_root_io/v1_11_02/slf7.x86_64.e20.prof/lib/libcanvas_root_io_Dictionaries_std_dict.so
2ad956978000-2ad95697b000 r--p 0006b000 00:192 3441325                   /cvmfs/larsoft.opensciencegrid.org/products/canvas_root_io/v1_11_02/slf7.x86_64.e20.prof/lib/libcanvas_root_io_Dictionaries_std_dict.so
2ad95697b000-2ad95697c000 rw-p 0006e000 00:192 3441325                   /cvmfs/larsoft.opensciencegrid.org/products/canvas_root_io/v1_11_02/slf7.x86_64.e20.prof/lib/libcanvas_root_io_Dictionaries_std_dict.so
2ad95697c000-2ad9591c2000 rw-p 00000000 00:00 0 
2ad95a5ad000-2ad9625ad000 rw-p 00000000 00:00 0 
2ad9625ad000-2ad966691000 rw-p 00000000 00:00 0 
2ad9685ae000-2ad9745af000 rw-p 00000000 00:00 0 
2ad97d1df000-2ada09797000 rw-p 00000000 00:00 0 
2ada1c5b2000-2adadc5b3000 rw-p 00000000 00:00 0 
7ffd66628000-7ffd66667000 rw-p 00000000 00:00 0                          [stack]
7ffd6669a000-7ffd6669c000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
=== End last 100 lines of lar log file ===
lar exit code 0
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_75_03d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_75_03d00/bin/extractor_prod.py", line 373, in main
    mddict = expSpecificMetadata.getmetadata()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_75_03d00/bin/extractor_prod.py", line 344, in getmetadata
    jobt = self.get_job(proc)
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_75_03d00/bin/extractor_prod.py", line 69, in get_job
    raise RuntimeError('sam_metadata_dumper returned nonzero exit status {}.'.format(rc))
RuntimeError: sam_metadata_dumper returned nonzero exit status 1.
extractor_prod.py exit code 1
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
pdjson2metadata exit code 1
.:
total 90824
-rw-r--r-- 1 pildun13 dunepil 90529758 Oct  8 12:11 RootOutput-d518-ea8d-81d5-746f.root
-rw-r--r-- 1 pildun13 dunepil  2443345 Oct  8 12:11 000135_reco_data_2024-10-07T_114846Z_reco_2024-10-08T_090242Z.log
-rw-r--r-- 1 pildun13 dunepil    16472 Oct  8 12:12 jobscript.log
-rw-r--r-- 1 pildun13 dunepil      274 Oct  8 10:02 TFileService-3374-4d07-e865-004e.root
-rw-r--r-- 1 pildun13 dunepil      104 Oct  8 10:02 all-input-dids.txt
-rw-r--r-- 1 pildun13 dunepil        0 Oct  8 12:12 000135_reco_data_2024-10-07T_114846Z_reco_data_2024-10-08T_090242Z.root.ext.json
-rw-r--r-- 1 pildun13 dunepil        0 Oct  8 12:12 000135_reco_data_2024-10-07T_114846Z_reco_data_2024-10-08T_090242Z.root.json
justIN time: 2024-11-24 09:58:23 UTC       justIN version: 01.01.09