21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 231322.48@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 231322.48@justin-prod-sched02.dune.hep.ac.uk | |
| Workflow ID | 7983 | |
| Stage ID | 1 | |
| User name | imawby@fnal.gov | |
| HTCondor Group | group_dune | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 43200 (12 hours) | |
| Submitted time | 2025-06-27 12:49:44 | |
| Site | UK_Edinburgh | |
| Entry | DUNE_UK_SGridECDF_ce1 | |
| Last heartbeat | 2025-06-27 12:51:02 | |
| From worker node | Hostname | node2b05.ecdf.ed.ac.uk |
| cpuinfo | Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 171000 (47 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | jobscript_error | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-06-27 12:50:16 | |
| Input files | fardet-hd:nutau_dune10kt_1x2x6_1417_166_20230826T233305Z_gen_g4_detsim_hitreco__20240219T182848Z_reco2.root | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-06-27 12:51:02 | |
| Saved logs | justin-logs:231322.48-justin-prod-sched02.dune.hep.ac.uk.logs.tgz | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
directory Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_U (Potential memory leak). Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_V (Potential memory leak). Warning in <TFile::Append>: Replacing existing TH1: FieldResponse_Y (Potential memory leak). 27-Jun-2025 13:50:47 BST Initiating request to open input file "root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/c8/6e/nutau_dune10kt_1x2x6_1417_166_20230826T233305Z_gen_g4_detsim_hitreco__20240219T182848Z_reco2.root" [2025-06-27 13:50:47.492484 +0100][Debug ][Utility ][ 1304] Env: overriding entry: connectionwindow=120 with 30 [2025-06-27 13:50:47.492528 +0100][Debug ][Utility ][ 1304] Env: overriding entry: requesttimeout=1800 with 4096 [2025-06-27 13:50:47.492537 +0100][Debug ][Utility ][ 1304] Env: overriding entry: requesttimeout=4096 with 14400 [2025-06-27 13:50:47.492558 +0100][Debug ][Utility ][ 1304] Env: overriding entry: redirectlimit=16 with 64 [2025-06-27 13:50:47.492577 +0100][Debug ][Utility ][ 1304] Env: overriding entry: multiprotocol=0 with 1 [2025-06-27 13:50:47.492880 +0100][Debug ][File ][ 1304] [0xa353600@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/c8/6e/nutau_dune10kt_1x2x6_1417_166_20230826T233305Z_gen_g4_detsim_hitreco__20240219T182848Z_reco2.root?xrdcl.requuid=dfc1cbe6-76c3-42d9-b4dc-8dfaf94ac51e] Sending an open command [2025-06-27 13:50:47.493002 +0100][Debug ][Utility ][ 1304] Env: trying to get a non-existent string entry: pollerpreference [2025-06-27 13:50:47.493025 +0100][Debug ][Poller ][ 1304] Available pollers: built-in [2025-06-27 13:50:47.493029 +0100][Debug ][Poller ][ 1304] Attempting to create a poller according to preference: built-in [2025-06-27 13:50:47.493034 +0100][Debug ][Poller ][ 1304] Creating poller: built-in [2025-06-27 13:50:47.493054 +0100][Debug ][Poller ][ 1304] Creating and starting the built-in poller... [2025-06-27 13:50:47.493469 +0100][Debug ][Poller ][ 1304] Using 1 poller threads [2025-06-27 13:50:47.493489 +0100][Debug ][TaskMgr ][ 1304] Starting the task manager... [2025-06-27 13:50:47.493532 +0100][Debug ][TaskMgr ][ 1304] Task manager started [2025-06-27 13:50:47.493543 +0100][Debug ][JobMgr ][ 1304] Starting the job manager... [2025-06-27 13:50:47.493643 +0100][Debug ][JobMgr ][ 1304] Job manager started, 3 workers [2025-06-27 13:50:47.493667 +0100][Debug ][TaskMgr ][ 1304] Registering task: "FileTimer task" to be run at: [2025-06-27 13:50:47 +0100] [2025-06-27 13:50:47.493751 +0100][Debug ][ExDbgMsg ][ 1304] [se1.farm.particle.cz:1094] MsgHandler created: 0xd6b8d80 (message: kXR_open (file: /dune/RSE/fardet-hd/c8/6e/nutau_dune10kt_1x2x6_1417_166_20230826T233305Z_gen_g4_detsim_hitreco__20240219T182848Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ). [2025-06-27 13:50:47.493859 +0100][Debug ][PostMaster ][ 1304] Creating new channel to: root://se1.farm.particle.cz:1094/ [2025-06-27 13:50:47.494089 +0100][Debug ][PostMaster ][ 1304] [se1.farm.particle.cz:1094] Stream parameters: Network Stack: IPAuto, Connection Window: 30, ConnectionRetry: 5, Stream Error Window: 1800 [2025-06-27 13:50:47.494127 +0100][Debug ][TaskMgr ][ 1304] Registering task: "TickGeneratorTask for: root://se1.farm.particle.cz:1094/" to be run at: [2025-06-27 13:51:02 +0100] [2025-06-27 13:50:47.495365 +0100][Debug ][PostMaster ][ 1304] [se1.farm.particle.cz:1094] Found 1 address(es): [::ffff:147.231.25.100]:1094 [2025-06-27 13:50:47.495405 +0100][Debug ][AsyncSock ][ 1304] [se1.farm.particle.cz:1094.0] Attempting connection to [::ffff:147.231.25.100]:1094 [2025-06-27 13:50:47.495449 +0100][Debug ][Poller ][ 1304] Adding socket 0xa3541e0 to the poller [2025-06-27 13:50:47.527147 +0100][Debug ][AsyncSock ][ 1304] [se1.farm.particle.cz:1094.0] Async connection call returned [2025-06-27 13:50:47.527260 +0100][Debug ][XRootDTransport ][ 1304] [se1.farm.particle.cz:1094.0] Sending out the initial hand shake + kXR_protocol [2025-06-27 13:50:47.559025 +0100][Debug ][XRootDTransport ][ 1304] [se1.farm.particle.cz:1094.0] Got the server hand shake response (type: manager [], protocol version 500) [2025-06-27 13:50:47.559088 +0100][Debug ][XRootDTransport ][ 1304] [se1.farm.particle.cz:1094.0] kXR_protocol successful (type: manager [], protocol version 500) [2025-06-27 13:50:47.560033 +0100][Debug ][XRootDTransport ][ 1304] [se1.farm.particle.cz:1094.0] Sending out kXR_login request, username: gl05pi6, cgi: xrd.cc=uk&xrd.tz=0&xrd.appname=lar&xrd.info=&xrd.hostname=node2b05.ecdf.ed.ac.uk&xrd.rn=v5.5.5, dual-stack: false, private IPv4: false, private IPv6: false [2025-06-27 13:50:47.591802 +0100][Debug ][XRootDTransport ][ 1304] [se1.farm.particle.cz:1094.0] Logged in, session: 2cecbec80437b1d199326582fdd394b2 [2025-06-27 13:50:47.591830 +0100][Debug ][XRootDTransport ][ 1304] [se1.farm.particle.cz:1094.0] Authentication is required: &P=gsi,v:10400,c:ssl,ca:9c979c2b&P=unix [2025-06-27 13:50:47.591842 +0100][Debug ][XRootDTransport ][ 1304] [se1.farm.particle.cz:1094.0] Sending authentication data [2025-06-27 13:50:47.606947 +0100][Debug ][XRootDTransport ][ 1304] [se1.farm.particle.cz:1094.0] Trying to authenticate using gsi [2025-06-27 13:50:48.765298 +0100][Debug ][XRootDTransport ][ 1304] [se1.farm.particle.cz:1094.0] Sending more authentication data for gsi [2025-06-27 13:50:48.813383 +0100][Debug ][XRootDTransport ][ 1304] [se1.farm.particle.cz:1094.0] Authenticated with gsi. [2025-06-27 13:50:48.813442 +0100][Debug ][PostMaster ][ 1304] [se1.farm.particle.cz:1094] Stream 0 connected (IPv4). [2025-06-27 13:50:48.813462 +0100][Debug ][Utility ][ 1304] Monitor library name not set. No monitoring [2025-06-27 13:50:48.813553 +0100][Debug ][ExDbgMsg ][ 1304] [se1.farm.particle.cz:1094] Moving MsgHandler: 0xd6b8d80 (message: kXR_open (file: /dune/RSE/fardet-hd/c8/6e/nutau_dune10kt_1x2x6_1417_166_20230826T233305Z_gen_g4_detsim_hitreco__20240219T182848Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) from out-queu to in-queue. [2025-06-27 13:50:48.850153 +0100][Debug ][ExDbgMsg ][ 1304] [msg: 0xb8ff6e0] Assigned MsgHandler: 0xd6b8d80. [2025-06-27 13:50:48.850168 +0100][Debug ][ExDbgMsg ][ 1304] [handler: 0xd6b8d80] Removed MsgHandler: 0xd6b8d80 from the in-queue. [2025-06-27 13:50:48.850251 +0100][Debug ][XRootD ][ 1304] [se1.farm.particle.cz:1094] Handling error while processing kXR_open (file: /dune/RSE/fardet-hd/c8/6e/nutau_dune10kt_1x2x6_1417_166_20230826T233305Z_gen_g4_detsim_hitreco__20240219T182848Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ): [ERROR] Error response: no such file or directory. [2025-06-27 13:50:48.850297 +0100][Debug ][ExDbgMsg ][ 1304] [se1.farm.particle.cz:1094] Calling MsgHandler: 0xd6b8d80 (message: kXR_open (file: /dune/RSE/fardet-hd/c8/6e/nutau_dune10kt_1x2x6_1417_166_20230826T233305Z_gen_g4_detsim_hitreco__20240219T182848Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) with status: [ERROR] Error response: no such file or directory. [2025-06-27 13:50:48.850378 +0100][Debug ][File ][ 1304] [0xa353600@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/c8/6e/nutau_dune10kt_1x2x6_1417_166_20230826T233305Z_gen_g4_detsim_hitreco__20240219T182848Z_reco2.root?xrdcl.requuid=dfc1cbe6-76c3-42d9-b4dc-8dfaf94ac51e] Open has returned with status [ERROR] Server responded with an error: [3011] No such file [2025-06-27 13:50:48.850392 +0100][Debug ][File ][ 1304] [0xa353600@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/c8/6e/nutau_dune10kt_1x2x6_1417_166_20230826T233305Z_gen_g4_detsim_hitreco__20240219T182848Z_reco2.root?xrdcl.requuid=dfc1cbe6-76c3-42d9-b4dc-8dfaf94ac51e] Error while opening at se1.farm.particle.cz:1094: [ERROR] Server responded with an error: [3011] No such file [2025-06-27 13:50:48.850430 +0100][Debug ][ExDbgMsg ][ 1304] [se1.farm.particle.cz:1094] Destroying MsgHandler: 0xd6b8d80. MultiPandoraApiImpl::DeletePandoraInstances - unable to find daughter instances associated with primary 0 %MSG-s ArtException: TriggerResultInserter:TriggerResults@Construction 27-Jun-2025 13:50:48 BST ModuleConstruction cet::exception caught in art ---- FileOpenError BEGIN RootInputFileSequence::initFile(): Input file root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/c8/6e/nutau_dune10kt_1x2x6_1417_166_20230826T233305Z_gen_g4_detsim_hitreco__20240219T182848Z_reco2.root was not found or could not be opened. ---- FileOpenError END %MSG Art has completed and will exit with status 20. [2025-06-27 13:50:49.055601 +0100][Debug ][JobMgr ][ 1304] Stopping the job manager... [2025-06-27 13:50:49.056594 +0100][Debug ][JobMgr ][ 1304] Job manager stopped [2025-06-27 13:50:49.057352 +0100][Debug ][TaskMgr ][ 1304] Stopping the task manager... [2025-06-27 13:50:49.057625 +0100][Debug ][TaskMgr ][ 1304] Task manager stopped [2025-06-27 13:50:49.057743 +0100][Debug ][Poller ][ 1304] Stopping the poller... [2025-06-27 13:50:49.057931 +0100][Debug ][AsyncSock ][ 1304] [se1.farm.particle.cz:1094.0] Closing the socket [2025-06-27 13:50:49.058218 +0100][Debug ][Poller ][ 1304] <[::ffff:192.41.105.38]:45960><--><[::ffff:147.231.25.100]:1094> Removing socket from the poller [2025-06-27 13:50:49.058306 +0100][Debug ][PostMaster ][ 1304] [se1.farm.particle.cz:1094] Destroying stream [2025-06-27 13:50:49.058335 +0100][Debug ][AsyncSock ][ 1304] [se1.farm.particle.cz:1094.0] Closing the socket === End last 100 lines of lar log file === mv: cannot stat 'HierarchyRecoPerformance.root': No such file or directory