justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 168151.0@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID168151.0@justin-prod-sched02.dune.hep.ac.uk
Workflow ID5837
Stage ID1
User nametwalton@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
GPUNo
RSS bytes2097152000 (2000 MiB)
Wall seconds limit3600 (1 hours)
Submitted time2025-04-01 07:58:49
SiteUS_NERSC-CPU
Entrydune_t3_us_nersc_perlmutter_cpu_sl7
Last heartbeat2025-04-01 08:09:15
From worker nodeHostnamenid006973
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit84600 (23 hours)
GPU
Inner Apptainer?False
Job statejobscript_error
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-04-01 08:00:36
Input filesneardet-2x2-lar-charge:binary-0050011-2024_07_07_13_59_33_CDT.hdf5
JobscriptExit code1
Real time0m (0s)
CPU time0m (0s = 0%)
Max RSS bytes0 (0 MiB)
Outputting started 
Output files
Finished2025-04-01 08:09:15
Saved logsjustin-logs:168151.0-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/neardet-2x2-lar-charge/87/32/binary-0050011-2024_07_07_13_59_33_CDT.hdf5]
	The file Rucio storage element (RSE) is [DUNE_US_FNAL_DISK_STAGE]

The input file is binary-0050011-2024_07_07_13_59_33_CDT.hdf5
Using rucio to download file [neardet-2x2-lar-charge:binary-0050011-2024_07_07_13_59_33_CDT.hdf5]
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
status     : ACTIVE
account    : justinreadonly
account_type : USER
deleted_at : None
created_at : 2024-12-23T15:57:24
suspended_at : None
email      : timm@fnal.gov
updated_at : 2024-12-23T15:57:24
2025-04-01 08:00:39,272	INFO	Processing 1 item(s) for input
2025-04-01 08:00:39,930	INFO	No preferred protocol impl in rucio.cfg: No section: 'download'
2025-04-01 08:00:39,931	INFO	Using main thread to download 1 file(s)
2025-04-01 08:00:39,931	INFO	Preparing download of neardet-2x2-lar-charge:binary-0050011-2024_07_07_13_59_33_CDT.hdf5
2025-04-01 08:00:40,237	INFO	Trying to download with root and timeout of 547s from DUNE_US_FNAL_DISK_STAGE: neardet-2x2-lar-charge:binary-0050011-2024_07_07_13_59_33_CDT.hdf5 
2025-04-01 08:00:40,281	INFO	Using PFN: root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/neardet-2x2-lar-charge/87/32/binary-0050011-2024_07_07_13_59_33_CDT.hdf5
 error in request of checking file QoS:   is not a valid HTTP or Webdav URL

(process:1432087): GLib-WARNING **: 08:00:40.282: GError set over the top of a previous GError or uninitialized memory.
This indicates a bug in someone's code. You must ensure an error is NULL before it's set.
The overwriting error message was:  \x01 is not a valid HTTP or Webdav URL
2025-04-01 08:02:37,801	WARNING	Download attempt failed. Try 1/2
/cvmfs/dune.opensciencegrid.org/products/dune/rucio/v35_4_0/NULL/lib/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'dune-rucio.fnal.gov'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
 error in request of checking file QoS:   is not a valid HTTP or Webdav URL

(process:1432087): GLib-WARNING **: 08:02:38.035: GError set over the top of a previous GError or uninitialized memory.
This indicates a bug in someone's code. You must ensure an error is NULL before it's set.
The overwriting error message was:  \x01 is not a valid HTTP or Webdav URL
2025-04-01 08:03:38,932	WARNING	Download attempt failed. Try 2/2
/cvmfs/dune.opensciencegrid.org/products/dune/rucio/v35_4_0/NULL/lib/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'dune-rucio.fnal.gov'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
2025-04-01 08:03:39,231	INFO	Trying to download with davs and timeout of 547s from DUNE_US_FNAL_DISK_STAGE: neardet-2x2-lar-charge:binary-0050011-2024_07_07_13_59_33_CDT.hdf5 
2025-04-01 08:03:39,236	INFO	Using PFN: davs://fndca1.fnal.gov:2880/dune/persistent/staging/neardet-2x2-lar-charge/87/32/binary-0050011-2024_07_07_13_59_33_CDT.hdf5
 error in request of checking file QoS:   is not a valid HTTP or Webdav URL

(process:1432087): GLib-WARNING **: 08:03:39.237: GError set over the top of a previous GError or uninitialized memory.
This indicates a bug in someone's code. You must ensure an error is NULL before it's set.
The overwriting error message was:  \x01 is not a valid HTTP or Webdav URL
/cvmfs/dune.opensciencegrid.org/products/dune/rucio/v35_4_0/NULL/lib/python3.9/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'dune-rucio.fnal.gov'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
2025-04-01 08:03:41,218	INFO	File neardet-2x2-lar-charge:binary-0050011-2024_07_07_13_59_33_CDT.hdf5 successfully downloaded. 243.555 MB in 1.59 seconds = 153.18 MBps
----------------------------------
Download summary
----------------------------------------
DID neardet-2x2-lar-charge:binary-0050011-2024_07_07_13_59_33_CDT.hdf5
Total files (DID):                            1
Total files (filtered):                       1
Downloaded files:                             1
Files already found locally:                  0
Files that cannot be downloaded:              0
-rw-r--r-- 1 dunepro dunepro   67 Apr  1 08:00 all-input-dids.txt
-rwxr-xr-x 1 dunepro dunepro 233M Apr  1 08:03 binary-0050011-2024_07_07_13_59_33_CDT.hdf5
-rw-r--r-- 1 dunepro dunepro 1.2K Apr  1 08:00 env_5837.1.168151.0.justin-prod-sched02.dune.hep.ac.uk.log
-rw-r--r-- 1 dunepro dunepro  43K Apr  1 08:03 jobscript.log
Downloading the matching light files for the charge+light combination workflow.
	Setting up the data management tools.
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
	Running the command [ python /cvmfs/minerva.opensciencegrid.org/minerva2x2/2x2tmp/v1.1.1/ndlar_scripts/GetInputList.py --file=neardet-2x2-lar-charge:binary-0050011-2024_07_07_13_59_33_CDT.hdf5 ].

Enter getting the input list of light files for the 2x2 production jobs

	Charge file is [neardet-2x2-lar-charge:binary-0050011-2024_07_07_13_59_33_CDT.hdf5]
		The matching runs/subruns list [ [(50011, 32), (50011, 33)] ]

	The paths for neardet-2x2-lar-light:mpd_run_hvramp_rctl_098_p32.data is [['root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/tape_backed/dunepro//neardet-2x2-lar-light/raw/2024/detector/commissioning/None/00/05/00/11/mpd_run_hvramp_rctl_098_p32.data', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/neardet-2x2-lar-light/6e/78/mpd_run_hvramp_rctl_098_p32.data', '']].

	Downloading the light raw file [neardet-2x2-lar-light:mpd_run_hvramp_rctl_098_p32.data]
----------------------------------
Download summary
----------------------------------------
DID neardet-2x2-lar-light:mpd_run_hvramp_rctl_098_p32.data
Total files (DID):                            1
Total files (filtered):                       1
Downloaded files:                             1
Files already found locally:                  0
Files that cannot be downloaded:              0

		Successfully download the file [neardet-2x2-lar-light:mpd_run_hvramp_rctl_098_p32.data].
	The paths for neardet-2x2-lar-light:mpd_run_hvramp_rctl_098_p33.data is [['root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/tape_backed/dunepro//neardet-2x2-lar-light/raw/2024/detector/commissioning/None/00/05/00/11/mpd_run_hvramp_rctl_098_p33.data', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/neardet-2x2-lar-light/88/47/mpd_run_hvramp_rctl_098_p33.data', '']].

	Downloading the light raw file [neardet-2x2-lar-light:mpd_run_hvramp_rctl_098_p33.data]
----------------------------------
Download summary
----------------------------------------
DID neardet-2x2-lar-light:mpd_run_hvramp_rctl_098_p33.data
Total files (DID):                            1
Total files (filtered):                       1
Downloaded files:                             1
Files already found locally:                  0
Files that cannot be downloaded:              0

		Successfully download the file [neardet-2x2-lar-light:mpd_run_hvramp_rctl_098_p33.data].
Exit getting the input list of matching light files.
	The number of files is [2].

	The parent files are [neardet-2x2-lar-charge:binary-0050011-2024_07_07_13_59_33_CDT.hdf5 neardet-2x2-lar-light:mpd_run_hvramp_rctl_098_p32.data neardet-2x2-lar-light:mpd_run_hvramp_rctl_098_p33.data].

The output files are placed in the directory [/tmp/glide_zVcUWv/execute/dir_1427867/home/workspace]

total 8.6G
-rw-r--r-- 1 dunepro dunepro   67 Apr  1 08:00 all-input-dids.txt
-rwxr-xr-x 1 dunepro dunepro 233M Apr  1 08:03 binary-0050011-2024_07_07_13_59_33_CDT.hdf5
-rwxr-xr-x 1 dunepro dunepro 4.3G Apr  1 08:06 mpd_run_hvramp_rctl_098_p32.data
-rwxr-xr-x 1 dunepro dunepro 4.1G Apr  1 08:09 mpd_run_hvramp_rctl_098_p33.data
-rw-r--r-- 1 dunepro dunepro  46K Apr  1 08:09 jobscript.log
-rw-r--r-- 1 dunepro dunepro 4.3K Apr  1 08:09 env_5837.1.168151.0.justin-prod-sched02.dune.hep.ac.uk.log
Setup and enter the conda environment for the software release [v1.1.1]
	Running [ source /cvmfs/minerva.opensciencegrid.org/minerva2x2/2x2tmp/v00_00_01/anaconda/etc/profile.d/conda.sh ]
	The current conda virtual environment is activated: [/cvmfs/minerva.opensciencegrid.org/minerva2x2/2x2tmp/v00_00_01/conda_envs/ndlar_flow_v1.1.1]
	Copying the configuration directory [/cvmfs/minerva.opensciencegrid.org/minerva2x2/2x2tmp/v1.1.1/ndlar_flow/yamls/proto_nd_flow] to the workspace [/tmp/glide_zVcUWv/execute/dir_1427867/home/workspace/ndlar_flow/yamls/proto_nd_flow]
	Copying the constants directory [/cvmfs/minerva.opensciencegrid.org/minerva2x2/2x2tmp/v1.1.1/ndlar_flow/data/proto_nd_flow] to the workspace [/tmp/glide_zVcUWv/execute/dir_1427867/home/workspace/ndlar_flow/data/proto_nd_flow]



Enter executing the charge raw files to hdf5 packet workflow for data stream [combined]
	Running the charge raw data to packet data conversion workflow.
	[ python /cvmfs/minerva.opensciencegrid.org/minerva2x2/2x2tmp/v1.1.1/larpix-control/scripts/convert_rawhdf5_to_hdf5.py -i /tmp/glide_zVcUWv/execute/dir_1427867/home/workspace/binary-0050011-2024_07_07_13_59_33_CDT.hdf5 -o packet-0050011-2024_07_07_13_59_33_CDT.hdf5 --direct >> packet-0050011-2024_07_07_13_59_33_CDT.log ]
	Moving the file(s) to the outfiles directory
FATAL::The file [packet-0050011-2024_07_07_13_59_33_CDT.hdf5] does not exist! Will not continue.
justIN time: 2025-04-03 08:13:25 UTC       justIN version: 01.03.00