21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 231319.13@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 231319.13@justin-prod-sched02.dune.hep.ac.uk | |
| Workflow ID | 7983 | |
| Stage ID | 1 | |
| User name | imawby@fnal.gov | |
| HTCondor Group | group_dune | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 43200 (12 hours) | |
| Submitted time | 2025-06-27 12:47:19 | |
| Site | UK_Edinburgh | |
| Entry | DUNE_UK_SGridECDF_ce1 | |
| Last heartbeat | 2025-06-27 12:49:31 | |
| From worker node | Hostname | node2b05.ecdf.ed.ac.uk |
| cpuinfo | Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 171000 (47 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-06-27 12:48:46 | |
| Input files | fardet-hd:nutau_dune10kt_1x2x6_1112_750_20230827T115620Z_gen_g4_detsim_hitreco__20240220T184604Z_reco2.root | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-06-27 12:49:31 | |
| Saved logs | justin-logs:231319.13-justin-prod-sched02.dune.hep.ac.uk.logs.tgz | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
directory Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak). Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak). Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak). 27-Jun-2025 13:49:16 BST Initiating request to open input file "root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/0c/43/nutau_dune10kt_1x2x6_1112_750_20230827T115620Z_gen_g4_detsim_hitreco__20240220T184604Z_reco2.root" [2025-06-27 13:49:16.535764 +0100][Debug ][Utility ][ 1304] Env: overriding entry: connectionwindow=120 with 30 [2025-06-27 13:49:16.535800 +0100][Debug ][Utility ][ 1304] Env: overriding entry: requesttimeout=1800 with 4096 [2025-06-27 13:49:16.535809 +0100][Debug ][Utility ][ 1304] Env: overriding entry: requesttimeout=4096 with 14400 [2025-06-27 13:49:16.535829 +0100][Debug ][Utility ][ 1304] Env: overriding entry: redirectlimit=16 with 64 [2025-06-27 13:49:16.535848 +0100][Debug ][Utility ][ 1304] Env: overriding entry: multiprotocol=0 with 1 [2025-06-27 13:49:16.536162 +0100][Debug ][File ][ 1304] [0x9e52fd0@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/0c/43/nutau_dune10kt_1x2x6_1112_750_20230827T115620Z_gen_g4_detsim_hitreco__20240220T184604Z_reco2.root?xrdcl.requuid=889e39f4-b142-42b2-bdfb-56777783befb] Sending an open command [2025-06-27 13:49:16.536310 +0100][Debug ][Utility ][ 1304] Env: trying to get a non-existent string entry: pollerpreference [2025-06-27 13:49:16.536339 +0100][Debug ][Poller ][ 1304] Available pollers: built-in [2025-06-27 13:49:16.536343 +0100][Debug ][Poller ][ 1304] Attempting to create a poller according to preference: built-in [2025-06-27 13:49:16.536348 +0100][Debug ][Poller ][ 1304] Creating poller: built-in [2025-06-27 13:49:16.536372 +0100][Debug ][Poller ][ 1304] Creating and starting the built-in poller... [2025-06-27 13:49:16.536781 +0100][Debug ][Poller ][ 1304] Using 1 poller threads [2025-06-27 13:49:16.536803 +0100][Debug ][TaskMgr ][ 1304] Starting the task manager... [2025-06-27 13:49:16.536842 +0100][Debug ][TaskMgr ][ 1304] Task manager started [2025-06-27 13:49:16.536856 +0100][Debug ][JobMgr ][ 1304] Starting the job manager... [2025-06-27 13:49:16.536943 +0100][Debug ][JobMgr ][ 1304] Job manager started, 3 workers [2025-06-27 13:49:16.536971 +0100][Debug ][TaskMgr ][ 1304] Registering task: "FileTimer task" to be run at: [2025-06-27 13:49:16 +0100] [2025-06-27 13:49:16.537070 +0100][Debug ][ExDbgMsg ][ 1304] [se1.farm.particle.cz:1094] MsgHandler created: 0x9ee4b90 (message: kXR_open (file: /dune/RSE/fardet-hd/0c/43/nutau_dune10kt_1x2x6_1112_750_20230827T115620Z_gen_g4_detsim_hitreco__20240220T184604Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ). [2025-06-27 13:49:16.537203 +0100][Debug ][PostMaster ][ 1304] Creating new channel to: root://se1.farm.particle.cz:1094/ [2025-06-27 13:49:16.537421 +0100][Debug ][PostMaster ][ 1304] [se1.farm.particle.cz:1094] Stream parameters: Network Stack: IPAuto, Connection Window: 30, ConnectionRetry: 5, Stream Error Window: 1800 [2025-06-27 13:49:16.537464 +0100][Debug ][TaskMgr ][ 1304] Registering task: "TickGeneratorTask for: root://se1.farm.particle.cz:1094/" to be run at: [2025-06-27 13:49:31 +0100] [2025-06-27 13:49:16.538209 +0100][Debug ][PostMaster ][ 1304] [se1.farm.particle.cz:1094] Found 1 address(es): [::ffff:147.231.25.100]:1094 [2025-06-27 13:49:16.538250 +0100][Debug ][AsyncSock ][ 1304] [se1.farm.particle.cz:1094.0] Attempting connection to [::ffff:147.231.25.100]:1094 [2025-06-27 13:49:16.538288 +0100][Debug ][Poller ][ 1304] Adding socket 0xb0a36e0 to the poller [2025-06-27 13:49:16.569861 +0100][Debug ][AsyncSock ][ 1304] [se1.farm.particle.cz:1094.0] Async connection call returned [2025-06-27 13:49:16.569993 +0100][Debug ][XRootDTransport ][ 1304] [se1.farm.particle.cz:1094.0] Sending out the initial hand shake + kXR_protocol [2025-06-27 13:49:16.601703 +0100][Debug ][XRootDTransport ][ 1304] [se1.farm.particle.cz:1094.0] Got the server hand shake response (type: manager [], protocol version 500) [2025-06-27 13:49:16.601751 +0100][Debug ][XRootDTransport ][ 1304] [se1.farm.particle.cz:1094.0] kXR_protocol successful (type: manager [], protocol version 500) [2025-06-27 13:49:16.602425 +0100][Debug ][XRootDTransport ][ 1304] [se1.farm.particle.cz:1094.0] Sending out kXR_login request, username: gl05pi6, cgi: xrd.cc=uk&xrd.tz=0&xrd.appname=lar&xrd.info=&xrd.hostname=node2b05.ecdf.ed.ac.uk&xrd.rn=v5.5.5, dual-stack: false, private IPv4: false, private IPv6: false [2025-06-27 13:49:16.633923 +0100][Debug ][XRootDTransport ][ 1304] [se1.farm.particle.cz:1094.0] Logged in, session: 88d72bd3702f6c6c8f39b24586fe5436 [2025-06-27 13:49:16.633937 +0100][Debug ][XRootDTransport ][ 1304] [se1.farm.particle.cz:1094.0] Authentication is required: &P=gsi,v:10400,c:ssl,ca:9c979c2b&P=unix [2025-06-27 13:49:16.633948 +0100][Debug ][XRootDTransport ][ 1304] [se1.farm.particle.cz:1094.0] Sending authentication data [2025-06-27 13:49:16.650498 +0100][Debug ][XRootDTransport ][ 1304] [se1.farm.particle.cz:1094.0] Trying to authenticate using gsi [2025-06-27 13:49:16.986963 +0100][Debug ][XRootDTransport ][ 1304] [se1.farm.particle.cz:1094.0] Sending more authentication data for gsi [2025-06-27 13:49:17.033615 +0100][Debug ][XRootDTransport ][ 1304] [se1.farm.particle.cz:1094.0] Authenticated with gsi. [2025-06-27 13:49:17.033678 +0100][Debug ][PostMaster ][ 1304] [se1.farm.particle.cz:1094] Stream 0 connected (IPv4). [2025-06-27 13:49:17.033696 +0100][Debug ][Utility ][ 1304] Monitor library name not set. No monitoring [2025-06-27 13:49:17.033790 +0100][Debug ][ExDbgMsg ][ 1304] [se1.farm.particle.cz:1094] Moving MsgHandler: 0x9ee4b90 (message: kXR_open (file: /dune/RSE/fardet-hd/0c/43/nutau_dune10kt_1x2x6_1112_750_20230827T115620Z_gen_g4_detsim_hitreco__20240220T184604Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) from out-queu to in-queue. [2025-06-27 13:49:17.071691 +0100][Debug ][ExDbgMsg ][ 1304] [msg: 0xb0a5790] Assigned MsgHandler: 0x9ee4b90. [2025-06-27 13:49:17.071705 +0100][Debug ][ExDbgMsg ][ 1304] [handler: 0x9ee4b90] Removed MsgHandler: 0x9ee4b90 from the in-queue. [2025-06-27 13:49:17.071796 +0100][Debug ][XRootD ][ 1304] [se1.farm.particle.cz:1094] Handling error while processing kXR_open (file: /dune/RSE/fardet-hd/0c/43/nutau_dune10kt_1x2x6_1112_750_20230827T115620Z_gen_g4_detsim_hitreco__20240220T184604Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ): [ERROR] Error response: no such file or directory. [2025-06-27 13:49:17.071849 +0100][Debug ][ExDbgMsg ][ 1304] [se1.farm.particle.cz:1094] Calling MsgHandler: 0x9ee4b90 (message: kXR_open (file: /dune/RSE/fardet-hd/0c/43/nutau_dune10kt_1x2x6_1112_750_20230827T115620Z_gen_g4_detsim_hitreco__20240220T184604Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) with status: [ERROR] Error response: no such file or directory. [2025-06-27 13:49:17.071946 +0100][Debug ][File ][ 1304] [0x9e52fd0@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/0c/43/nutau_dune10kt_1x2x6_1112_750_20230827T115620Z_gen_g4_detsim_hitreco__20240220T184604Z_reco2.root?xrdcl.requuid=889e39f4-b142-42b2-bdfb-56777783befb] Open has returned with status [ERROR] Server responded with an error: [3011] No such file [2025-06-27 13:49:17.071962 +0100][Debug ][File ][ 1304] [0x9e52fd0@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/0c/43/nutau_dune10kt_1x2x6_1112_750_20230827T115620Z_gen_g4_detsim_hitreco__20240220T184604Z_reco2.root?xrdcl.requuid=889e39f4-b142-42b2-bdfb-56777783befb] Error while opening at se1.farm.particle.cz:1094: [ERROR] Server responded with an error: [3011] No such file [2025-06-27 13:49:17.072009 +0100][Debug ][ExDbgMsg ][ 1304] [se1.farm.particle.cz:1094] Destroying MsgHandler: 0x9ee4b90. MultiPandoraApiImpl::DeletePandoraInstances - unable to find daughter instances associated with primary 0 %MSG-s ArtException: TriggerResultInserter:TriggerResults@Construction 27-Jun-2025 13:49:17 BST ModuleConstruction cet::exception caught in art ---- FileOpenError BEGIN RootInputFileSequence::initFile(): Input file root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/0c/43/nutau_dune10kt_1x2x6_1112_750_20230827T115620Z_gen_g4_detsim_hitreco__20240220T184604Z_reco2.root was not found or could not be opened. ---- FileOpenError END %MSG Art has completed and will exit with status 20. [2025-06-27 13:49:17.266228 +0100][Debug ][JobMgr ][ 1304] Stopping the job manager... [2025-06-27 13:49:17.267249 +0100][Debug ][JobMgr ][ 1304] Job manager stopped [2025-06-27 13:49:17.268021 +0100][Debug ][TaskMgr ][ 1304] Stopping the task manager... [2025-06-27 13:49:17.268556 +0100][Debug ][TaskMgr ][ 1304] Task manager stopped [2025-06-27 13:49:17.268654 +0100][Debug ][Poller ][ 1304] Stopping the poller... [2025-06-27 13:49:17.268859 +0100][Debug ][AsyncSock ][ 1304] [se1.farm.particle.cz:1094.0] Closing the socket [2025-06-27 13:49:17.268933 +0100][Debug ][Poller ][ 1304] <[::ffff:192.41.105.38]:53724><--><[::ffff:147.231.25.100]:1094> Removing socket from the poller [2025-06-27 13:49:17.269025 +0100][Debug ][PostMaster ][ 1304] [se1.farm.particle.cz:1094] Destroying stream [2025-06-27 13:49:17.269081 +0100][Debug ][AsyncSock ][ 1304] [se1.farm.particle.cz:1094.0] Closing the socket === End last 100 lines of lar log file === mv: cannot stat 'HierarchyRecoPerformance.root': No such file or directory