justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 271906.0@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID271906.0@justin-prod-sched01.dune.hep.ac.uk
Workflow ID3652
Stage ID1
User nameamoor@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-10-08 21:43:29
SiteFR_CCIN2P3
EntryDUNE_FR_CCIN2P3_cccondorce01
Last heartbeat2024-10-08 23:53:20
From worker nodeHostnameccwcondor0642
cpuinfoAMD EPYC 7453 28-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit106200 (29 hours)
Inner Apptainer?True
Job statejobscript_error
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-10-08 21:47:44
Input filesusertests:000104_reco_data_2024-10-07T_114813Z.root
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Outputting started 
Output files
Finished2024-10-08 23:53:20
Saved logsjustin-logs:271906.0-justin-prod-sched01.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

mfs/larsoft.opensciencegrid.org/products/dk2nudata/v01_10_01d/Linux64bit+3.10-2.17-e20-prof/lib/libdk2nuTree.so
2b50def59000-2b50def5b000 rw-p 00065000 00:4c 43830613                   /cvmfs/larsoft.opensciencegrid.org/products/dk2nudata/v01_10_01d/Linux64bit+3.10-2.17-e20-prof/lib/libdk2nuTree.so
2b50def5b000-2b50def5d000 rw-p 00000000 00:00 0 
2b50def5d000-2b50df039000 r-xp 00000000 00:4c 43860996                   /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libROOTDataFrame.so.6.26.07
2b50df039000-2b50df238000 ---p 000dc000 00:4c 43860996                   /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libROOTDataFrame.so.6.26.07
2b50df238000-2b50df23c000 r--p 000db000 00:4c 43860996                   /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libROOTDataFrame.so.6.26.07
2b50df23c000-2b50df23e000 rw-p 000df000 00:4c 43860996                   /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libROOTDataFrame.so.6.26.07
2b50df23e000-2b50df241000 rw-p 00000000 00:00 0 
2b50df241000-2b50df3b8000 r-xp 00000000 00:4c 43861734                   /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libTreePlayer.so.6.26.07
2b50df3b8000-2b50df5b7000 ---p 00177000 00:4c 43861734                   /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libTreePlayer.so.6.26.07
2b50df5b7000-2b50df5c2000 r--p 00176000 00:4c 43861734                   /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libTreePlayer.so.6.26.07
2b50df5c2000-2b50df5c5000 rw-p 00181000 00:4c 43861734                   /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libTreePlayer.so.6.26.07
2b50df5c5000-2b50df5cb000 rw-p 00000000 00:00 0 
2b50df5cb000-2b50df66e000 r-xp 00000000 00:4c 43859766                   /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libROOTVecOps.so.6.26.07
2b50df66e000-2b50df86e000 ---p 000a3000 00:4c 43859766                   /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libROOTVecOps.so.6.26.07
2b50df86e000-2b50df870000 r--p 000a3000 00:4c 43859766                   /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libROOTVecOps.so.6.26.07
2b50df870000-2b50df871000 rw-p 000a5000 00:4c 43859766                   /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libROOTVecOps.so.6.26.07
2b50df871000-2b50df872000 rw-p 00000000 00:00 0 
2b50df872000-2b50df93c000 r-xp 00000000 00:4c 43860396                   /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libROOTNTuple.so.6.26.07
2b50df93c000-2b50dfb3c000 ---p 000ca000 00:4c 43860396                   /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libROOTNTuple.so.6.26.07
2b50dfb3c000-2b50dfb3f000 r--p 000ca000 00:4c 43860396                   /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libROOTNTuple.so.6.26.07
2b50dfb3f000-2b50dfb41000 rw-p 000cd000 00:4c 43860396                   /cvmfs/larsoft.opensciencegrid.org/products/root/v6_26_06b/Linux64bit+3.10-2.17-e20-p3913-prof/lib/libROOTNTuple.so.6.26.07
2b50dfb41000-2b50dfb59000 r-xp 00000000 00:4c 44372392                   /cvmfs/larsoft.opensciencegrid.org/products/larcoreobj/v09_09_00/slf7.x86_64.e20.prof/lib/liblarcoreobj_SummaryData_dict.so
2b50dfb59000-2b50dfd59000 ---p 00018000 00:4c 44372392                   /cvmfs/larsoft.opensciencegrid.org/products/larcoreobj/v09_09_00/slf7.x86_64.e20.prof/lib/liblarcoreobj_SummaryData_dict.so
2b50dfd59000-2b50dfd5a000 r--p 00018000 00:4c 44372392                   /cvmfs/larsoft.opensciencegrid.org/products/larcoreobj/v09_09_00/slf7.x86_64.e20.prof/lib/liblarcoreobj_SummaryData_dict.so
2b50dfd5a000-2b50dfd5b000 rw-p 00019000 00:4c 44372392                   /cvmfs/larsoft.opensciencegrid.org/products/larcoreobj/v09_09_00/slf7.x86_64.e20.prof/lib/liblarcoreobj_SummaryData_dict.so
2b50dfd5b000-2b50e015f000 rw-p 00000000 00:00 0 
2b50e015f000-2b50e0248000 r-xp 00000000 00:4c 44372125                   /cvmfs/larsoft.opensciencegrid.org/products/lardataobj/v09_16_01/slf7.x86_64.e20.prof/lib/liblardataobj_Simulation_dict.so
2b50e0248000-2b50e0447000 ---p 000e9000 00:4c 44372125                   /cvmfs/larsoft.opensciencegrid.org/products/lardataobj/v09_16_01/slf7.x86_64.e20.prof/lib/liblardataobj_Simulation_dict.so
2b50e0447000-2b50e044c000 r--p 000e8000 00:4c 44372125                   /cvmfs/larsoft.opensciencegrid.org/products/lardataobj/v09_16_01/slf7.x86_64.e20.prof/lib/liblardataobj_Simulation_dict.so
2b50e044c000-2b50e044e000 rw-p 000ed000 00:4c 44372125                   /cvmfs/larsoft.opensciencegrid.org/products/lardataobj/v09_16_01/slf7.x86_64.e20.prof/lib/liblardataobj_Simulation_dict.so
2b50e044e000-2b50e0453000 rw-p 00000000 00:00 0 
2b50e0453000-2b50e0455000 r-xp 00000000 00:4c 44371778                   /cvmfs/larsoft.opensciencegrid.org/products/lardataobj/v09_16_01/slf7.x86_64.e20.prof/lib/liblardataobj_Simulation_Compatibility.so
2b50e0455000-2b50e0654000 ---p 00002000 00:4c 44371778                   /cvmfs/larsoft.opensciencegrid.org/products/lardataobj/v09_16_01/slf7.x86_64.e20.prof/lib/liblardataobj_Simulation_Compatibility.so
2b50e0654000-2b50e0655000 r--p 00001000 00:4c 44371778                   /cvmfs/larsoft.opensciencegrid.org/products/lardataobj/v09_16_01/slf7.x86_64.e20.prof/lib/liblardataobj_Simulation_Compatibility.so
2b50e0655000-2b50e0656000 rw-p 00002000 00:4c 44371778                   /cvmfs/larsoft.opensciencegrid.org/products/lardataobj/v09_16_01/slf7.x86_64.e20.prof/lib/liblardataobj_Simulation_Compatibility.so
2b50e0656000-2b50e0660000 r-xp 00000000 00:4c 44372381                   /cvmfs/larsoft.opensciencegrid.org/products/larcoreobj/v09_09_00/slf7.x86_64.e20.prof/lib/liblarcoreobj_SimpleTypesAndConstants_dict.so
2b50e0660000-2b50e0860000 ---p 0000a000 00:4c 44372381                   /cvmfs/larsoft.opensciencegrid.org/products/larcoreobj/v09_09_00/slf7.x86_64.e20.prof/lib/liblarcoreobj_SimpleTypesAndConstants_dict.so
2b50e0860000-2b50e0861000 r--p 0000a000 00:4c 44372381                   /cvmfs/larsoft.opensciencegrid.org/products/larcoreobj/v09_09_00/slf7.x86_64.e20.prof/lib/liblarcoreobj_SimpleTypesAndConstants_dict.so
2b50e0861000-2b50e0862000 rw-p 0000b000 00:4c 44372381                   /cvmfs/larsoft.opensciencegrid.org/products/larcoreobj/v09_09_00/slf7.x86_64.e20.prof/lib/liblarcoreobj_SimpleTypesAndConstants_dict.so
2b50e0862000-2b50e08cd000 r-xp 00000000 00:4c 44310030                   /cvmfs/larsoft.opensciencegrid.org/products/canvas_root_io/v1_11_02/slf7.x86_64.e20.prof/lib/libcanvas_root_io_Dictionaries_std_dict.so
2b50e08cd000-2b50e0acd000 ---p 0006b000 00:4c 44310030                   /cvmfs/larsoft.opensciencegrid.org/products/canvas_root_io/v1_11_02/slf7.x86_64.e20.prof/lib/libcanvas_root_io_Dictionaries_std_dict.so
2b50e0acd000-2b50e0ad0000 r--p 0006b000 00:4c 44310030                   /cvmfs/larsoft.opensciencegrid.org/products/canvas_root_io/v1_11_02/slf7.x86_64.e20.prof/lib/libcanvas_root_io_Dictionaries_std_dict.so
2b50e0ad0000-2b50e0ad1000 rw-p 0006e000 00:4c 44310030                   /cvmfs/larsoft.opensciencegrid.org/products/canvas_root_io/v1_11_02/slf7.x86_64.e20.prof/lib/libcanvas_root_io_Dictionaries_std_dict.so
2b50e0ad1000-2b50e0ad5000 rw-p 00000000 00:00 0 
2b50e245b000-2b50ea45b000 rw-p 00000000 00:00 0 
2b50ea45b000-2b50eed69000 rw-p 00000000 00:00 0 
2b50f045d000-2b50fd501000 rw-p 00000000 00:00 0 
2b51a4460000-2b5264461000 rw-p 00000000 00:00 0 
2b53fea5b000-2b547ea5c000 rw-p 00000000 00:00 0 
7ffe4df94000-7ffe4dfd3000 rw-p 00000000 00:00 0                          [stack]
7ffe4dfe1000-7ffe4dfe3000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
=== End last 100 lines of lar log file ===
lar exit code 0
Traceback (most recent call last):
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_75_03d00/bin/extractor_prod.py", line 434, in <module>
    main()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_75_03d00/bin/extractor_prod.py", line 373, in main
    mddict = expSpecificMetadata.getmetadata()
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_75_03d00/bin/extractor_prod.py", line 344, in getmetadata
    jobt = self.get_job(proc)
  File "/cvmfs/dune.opensciencegrid.org/products/dune/duneutil/v09_75_03d00/bin/extractor_prod.py", line 69, in get_job
    raise RuntimeError('sam_metadata_dumper returned nonzero exit status {}.'.format(rc))
RuntimeError: sam_metadata_dumper returned nonzero exit status 1.
extractor_prod.py exit code 1
Error reading metadata from file: Expecting value: line 1 column 1 (char 0)
pdjson2metadata exit code 1
.:
total 89016
-rw-r--r-- 1 dune100 lbno 90523605 Oct  9 01:52 RootOutput-1739-7405-f22a-db24.root
-rw-r--r-- 1 dune100 lbno   595289 Oct  9 01:52 000104_reco_data_2024-10-07T_114813Z_reco_2024-10-08T_214752Z.log
-rw-r--r-- 1 dune100 lbno    16893 Oct  9 01:53 jobscript.log
-rw-r--r-- 1 dune100 lbno      274 Oct  8 23:48 TFileService-0adb-a846-1e5d-079f.root
-rw-r--r-- 1 dune100 lbno      104 Oct  8 23:47 all-input-dids.txt
-rw-r--r-- 1 dune100 lbno        0 Oct  9 01:52 000104_reco_data_2024-10-07T_114813Z_reco_data_2024-10-08T_214752Z.root.ext.json
-rw-r--r-- 1 dune100 lbno        0 Oct  9 01:53 000104_reco_data_2024-10-07T_114813Z_reco_data_2024-10-08T_214752Z.root.json
justIN time: 2024-11-24 07:54:32 UTC       justIN version: 01.01.09