21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 423697.2@justin-prod-sched01.dune.hep.ac.uk
Jobsub ID | 423697.2@justin-prod-sched01.dune.hep.ac.uk | |
Workflow ID | 7991 | |
Stage ID | 1 | |
User name | avizcaya@fnal.gov | |
HTCondor Group | group_dune.prod_mcsim | |
Requested | Processors | 1 |
GPU | No | |
RSS bytes | 4193255424 (3999 MiB) | |
Wall seconds limit | 80000 (22 hours) | |
Submitted time | 2025-06-27 14:13:42 | |
Site | NL_NIKHEF | |
Entry | VIRGO_NL_NIKHEF_dissel | |
Last heartbeat | 2025-06-27 14:17:52 | |
From worker node | Hostname | wn-snel-031.farm.nikhef.nl |
cpuinfo | AMD EPYC 7H12 64-Core Processor | |
OS release | Scientific Linux release 7.9 (Nitrogen) | |
Processors | 1 | |
RSS bytes | 4193255424 (3999 MiB) | |
Wall seconds limit | 129600 (36 hours) | |
GPU | ||
Inner Apptainer? | True | |
Job state | jobscript_error | |
Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
Started | 2025-06-27 14:15:22 | |
Input files | monte-carlo-007991-000085 | |
Jobscript | Exit code | 1 |
Real time | 0m (0s) | |
CPU time | 0m (0s = 0%) | |
Max RSS bytes | 0 (0 MiB) | |
Outputting started | ||
Output files | ||
Finished | 2025-06-27 14:17:52 | |
Saved logs | justin-logs:423697.2-justin-prod-sched01.dune.hep.ac.uk.logs.tgz | |
List job events Wrapper job log |
Jobscript log (last 10,000 characters)
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/ Justin processors: 1 did_pfn_rse monte-carlo-007991-000085 000085 MONTECARLO 2 423697 Querying ehn1-beam-np04:avizcaya_g4bl_mom5-w7584s1p1 for 100 files Query: files from ehn1-beam-np04:avizcaya_g4bl_mom5-w7584s1p1 where dune.output_status=confirmed ordered skip 8400 limit 100 Traceback (most recent call last): File "/cvmfs/dune.opensciencegrid.org/products/dune/rucio/v35_4_0/NULL/lib/python3.9/site-packages/urllib3/connection.py", line 174, in _new_conn conn = connection.create_connection( File "/cvmfs/dune.opensciencegrid.org/products/dune/rucio/v35_4_0/NULL/lib/python3.9/site-packages/urllib3/util/connection.py", line 95, in create_connection raise err File "/cvmfs/dune.opensciencegrid.org/products/dune/rucio/v35_4_0/NULL/lib/python3.9/site-packages/urllib3/util/connection.py", line 85, in create_connection sock.connect(sa) TimeoutError: [Errno 110] Connection timed out During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/cvmfs/dune.opensciencegrid.org/products/dune/rucio/v35_4_0/NULL/lib/python3.9/site-packages/urllib3/connectionpool.py", line 715, in urlopen httplib_response = self._make_request( File "/cvmfs/dune.opensciencegrid.org/products/dune/rucio/v35_4_0/NULL/lib/python3.9/site-packages/urllib3/connectionpool.py", line 404, in _make_request self._validate_conn(conn) File "/cvmfs/dune.opensciencegrid.org/products/dune/rucio/v35_4_0/NULL/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1058, in _validate_conn conn.connect() File "/cvmfs/dune.opensciencegrid.org/products/dune/rucio/v35_4_0/NULL/lib/python3.9/site-packages/urllib3/connection.py", line 363, in connect self.sock = conn = self._new_conn() File "/cvmfs/dune.opensciencegrid.org/products/dune/rucio/v35_4_0/NULL/lib/python3.9/site-packages/urllib3/connection.py", line 186, in _new_conn raise NewConnectionError( urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x14fefc3f87c0>: Failed to establish a new connection: [Errno 110] Connection timed out During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/cvmfs/dune.opensciencegrid.org/products/dune/rucio/v35_4_0/NULL/lib/python3.9/site-packages/requests/adapters.py", line 667, in send resp = conn.urlopen( File "/cvmfs/dune.opensciencegrid.org/products/dune/rucio/v35_4_0/NULL/lib/python3.9/site-packages/urllib3/connectionpool.py", line 799, in urlopen retries = retries.increment( File "/cvmfs/dune.opensciencegrid.org/products/dune/rucio/v35_4_0/NULL/lib/python3.9/site-packages/urllib3/util/retry.py", line 592, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='metacat.fnal.gov', port=9443): Max retries exceeded with url: /dune_meta_prod/app/data/query?with_meta=yes&with_provenance=no (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x14fefc3f87c0>: Failed to establish a new connection: [Errno 110] Connection timed out')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/7bf312a0a99f42cae01e4f8cfdd3c3bdaaedc832/merge_g4bl.py", line 433, in <module> do_merge(args) File "/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/7bf312a0a99f42cae01e4f8cfdd3c3bdaaedc832/merge_g4bl.py", line 31, in do_merge files = query(args) File "/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/7bf312a0a99f42cae01e4f8cfdd3c3bdaaedc832/merge_g4bl.py", line 273, in query files = mc.query(query, with_metadata=True, with_provenance=with_parents) File "/cvmfs/dune.opensciencegrid.org/products/dune/metacat/v4_0_2/NULL/lib/python3.9/site-packages/metacat/webapi/webapi.py", line 1318, in query results = self.post_json(url, query) File "/cvmfs/dune.opensciencegrid.org/products/dune/metacat/v4_0_2/NULL/lib/python3.9/site-packages/metacat/webapi/webapi.py", line 215, in post_json response = self.send_request("post", uri_suffix, data=data, headers=headers, stream=True) File "/cvmfs/dune.opensciencegrid.org/products/dune/metacat/v4_0_2/NULL/lib/python3.9/site-packages/metacat/webapi/webapi.py", line 154, in send_request self.LastResponse = response = self.retry_request(method, url, headers=headers, **args) File "/cvmfs/dune.opensciencegrid.org/products/dune/metacat/v4_0_2/NULL/lib/python3.9/site-packages/metacat/webapi/webapi.py", line 133, in retry_request response = requests.post(url, timeout=self.Timeout, **args) File "/cvmfs/dune.opensciencegrid.org/products/dune/rucio/v35_4_0/NULL/lib/python3.9/site-packages/requests/api.py", line 115, in post return request("post", url, data=data, json=json, **kwargs) File "/cvmfs/dune.opensciencegrid.org/products/dune/rucio/v35_4_0/NULL/lib/python3.9/site-packages/requests/api.py", line 59, in request return session.request(method=method, url=url, **kwargs) File "/cvmfs/dune.opensciencegrid.org/products/dune/rucio/v35_4_0/NULL/lib/python3.9/site-packages/requests/sessions.py", line 589, in request resp = self.send(prep, **send_kwargs) File "/cvmfs/dune.opensciencegrid.org/products/dune/rucio/v35_4_0/NULL/lib/python3.9/site-packages/requests/sessions.py", line 703, in send r = adapter.send(request, **kwargs) File "/cvmfs/dune.opensciencegrid.org/products/dune/rucio/v35_4_0/NULL/lib/python3.9/site-packages/requests/adapters.py", line 700, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPSConnectionPool(host='metacat.fnal.gov', port=9443): Max retries exceeded with url: /dune_meta_prod/app/data/query?with_meta=yes&with_provenance=no (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x14fefc3f87c0>: Failed to establish a new connection: [Errno 110] Connection timed out')) Exiting with error