21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 413902.56@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 413902.56@justin-prod-sched01.dune.hep.ac.uk | |
| Workflow ID | 7650 | |
| Stage ID | 1 | |
| User name | hsouza@fnal.gov | |
| HTCondor Group | group_dune | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 3145728000 (3000 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-06-14 22:11:22 | |
| Site | UK_Edinburgh | |
| Entry | DUNE_UK_SGridECDF_ce1_multicore | |
| Last heartbeat | 2025-06-14 22:30:06 | |
| From worker node | Hostname | node2b23.ecdf.ed.ac.uk |
| cpuinfo | Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 3145728000 (3000 MiB) | |
| Wall seconds limit | 171000 (47 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | outputting_failed | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-06-14 22:12:37 | |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_186_20231203T180154Z_gen_g4_detsim_hitreco__20240509T203907Z_reco2.root | |
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) | |
| CPU time | 0m (0s = 0%) | |
| Max RSS bytes | 0 (0 MiB) | |
| Outputting started | ||
| Output files | ||
| Finished | 2025-06-14 22:30:06 | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
ity ][ 1314] Env: overriding entry: requesttimeout=1800 with 4096 [2025-06-14 23:12:54.851392 +0100][Debug ][Utility ][ 1314] Env: overriding entry: requesttimeout=4096 with 14400 [2025-06-14 23:12:54.851412 +0100][Debug ][Utility ][ 1314] Env: overriding entry: redirectlimit=16 with 64 [2025-06-14 23:12:54.851431 +0100][Debug ][Utility ][ 1314] Env: overriding entry: multiprotocol=0 with 1 [2025-06-14 23:12:54.851727 +0100][Debug ][File ][ 1314] [0x88cced0@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/df/1a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_186_20231203T180154Z_gen_g4_detsim_hitreco__20240509T203907Z_reco2.root?xrdcl.requuid=3933f958-c426-4dd2-aeda-b87c252c7d7f] Sending an open command [2025-06-14 23:12:54.851846 +0100][Debug ][Utility ][ 1314] Env: trying to get a non-existent string entry: pollerpreference [2025-06-14 23:12:54.851882 +0100][Debug ][Poller ][ 1314] Available pollers: built-in [2025-06-14 23:12:54.851886 +0100][Debug ][Poller ][ 1314] Attempting to create a poller according to preference: built-in [2025-06-14 23:12:54.851891 +0100][Debug ][Poller ][ 1314] Creating poller: built-in [2025-06-14 23:12:54.851912 +0100][Debug ][Poller ][ 1314] Creating and starting the built-in poller... [2025-06-14 23:12:54.852307 +0100][Debug ][Poller ][ 1314] Using 1 poller threads [2025-06-14 23:12:54.852327 +0100][Debug ][TaskMgr ][ 1314] Starting the task manager... [2025-06-14 23:12:54.852365 +0100][Debug ][TaskMgr ][ 1314] Task manager started [2025-06-14 23:12:54.852377 +0100][Debug ][JobMgr ][ 1314] Starting the job manager... [2025-06-14 23:12:54.852476 +0100][Debug ][JobMgr ][ 1314] Job manager started, 3 workers [2025-06-14 23:12:54.852501 +0100][Debug ][TaskMgr ][ 1314] Registering task: "FileTimer task" to be run at: [2025-06-14 23:12:54 +0100] [2025-06-14 23:12:54.852588 +0100][Debug ][ExDbgMsg ][ 1314] [se1.farm.particle.cz:1094] MsgHandler created: 0x884e530 (message: kXR_open (file: /dune/RSE/fardet-hd/df/1a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_186_20231203T180154Z_gen_g4_detsim_hitreco__20240509T203907Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ). [2025-06-14 23:12:54.852702 +0100][Debug ][PostMaster ][ 1314] Creating new channel to: root://se1.farm.particle.cz:1094/ [2025-06-14 23:12:54.852927 +0100][Debug ][PostMaster ][ 1314] [se1.farm.particle.cz:1094] Stream parameters: Network Stack: IPAuto, Connection Window: 30, ConnectionRetry: 5, Stream Error Window: 1800 [2025-06-14 23:12:54.852966 +0100][Debug ][TaskMgr ][ 1314] Registering task: "TickGeneratorTask for: root://se1.farm.particle.cz:1094/" to be run at: [2025-06-14 23:13:09 +0100] [2025-06-14 23:12:54.854856 +0100][Debug ][PostMaster ][ 1314] [se1.farm.particle.cz:1094] Found 1 address(es): [::ffff:147.231.25.100]:1094 [2025-06-14 23:12:54.854909 +0100][Debug ][AsyncSock ][ 1314] [se1.farm.particle.cz:1094.0] Attempting connection to [::ffff:147.231.25.100]:1094 [2025-06-14 23:12:54.854956 +0100][Debug ][Poller ][ 1314] Adding socket 0x6c76400 to the poller [2025-06-14 23:12:54.886581 +0100][Debug ][AsyncSock ][ 1314] [se1.farm.particle.cz:1094.0] Async connection call returned [2025-06-14 23:12:54.886683 +0100][Debug ][XRootDTransport ][ 1314] [se1.farm.particle.cz:1094.0] Sending out the initial hand shake + kXR_protocol [2025-06-14 23:12:54.918358 +0100][Debug ][XRootDTransport ][ 1314] [se1.farm.particle.cz:1094.0] Got the server hand shake response (type: manager [], protocol version 500) [2025-06-14 23:12:54.918934 +0100][Debug ][XRootDTransport ][ 1314] [se1.farm.particle.cz:1094.0] kXR_protocol successful (type: manager [], protocol version 500) [2025-06-14 23:12:54.919616 +0100][Debug ][XRootDTransport ][ 1314] [se1.farm.particle.cz:1094.0] Sending out kXR_login request, username: gl05pi6, cgi: xrd.cc=uk&xrd.tz=0&xrd.appname=lar&xrd.info=&xrd.hostname=node2b23.ecdf.ed.ac.uk&xrd.rn=v5.5.5, dual-stack: false, private IPv4: false, private IPv6: false [2025-06-14 23:12:54.951339 +0100][Debug ][XRootDTransport ][ 1314] [se1.farm.particle.cz:1094.0] Logged in, session: 47874ffba54fa75026c3a90575742632 [2025-06-14 23:12:54.951373 +0100][Debug ][XRootDTransport ][ 1314] [se1.farm.particle.cz:1094.0] Authentication is required: &P=gsi,v:10400,c:ssl,ca:9c979c2b&P=unix [2025-06-14 23:12:54.951385 +0100][Debug ][XRootDTransport ][ 1314] [se1.farm.particle.cz:1094.0] Sending authentication data [2025-06-14 23:12:54.988738 +0100][Debug ][XRootDTransport ][ 1314] [se1.farm.particle.cz:1094.0] Trying to authenticate using gsi [2025-06-14 23:12:55.068190 +0100][Debug ][XRootDTransport ][ 1314] [se1.farm.particle.cz:1094.0] Sending more authentication data for gsi [2025-06-14 23:12:55.115087 +0100][Debug ][XRootDTransport ][ 1314] [se1.farm.particle.cz:1094.0] Authenticated with gsi. [2025-06-14 23:12:55.115148 +0100][Debug ][PostMaster ][ 1314] [se1.farm.particle.cz:1094] Stream 0 connected (IPv4). [2025-06-14 23:12:55.115165 +0100][Debug ][Utility ][ 1314] Monitor library name not set. No monitoring [2025-06-14 23:12:55.115264 +0100][Debug ][ExDbgMsg ][ 1314] [se1.farm.particle.cz:1094] Moving MsgHandler: 0x884e530 (message: kXR_open (file: /dune/RSE/fardet-hd/df/1a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_186_20231203T180154Z_gen_g4_detsim_hitreco__20240509T203907Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) from out-queu to in-queue. [2025-06-14 23:12:55.270694 +0100][Debug ][ExDbgMsg ][ 1314] [msg: 0x88b3420] Assigned MsgHandler: 0x884e530. [2025-06-14 23:12:55.270741 +0100][Debug ][ExDbgMsg ][ 1314] [handler: 0x884e530] Removed MsgHandler: 0x884e530 from the in-queue. [2025-06-14 23:12:55.270839 +0100][Debug ][XRootD ][ 1314] [se1.farm.particle.cz:1094] Handling error while processing kXR_open (file: /dune/RSE/fardet-hd/df/1a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_186_20231203T180154Z_gen_g4_detsim_hitreco__20240509T203907Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ): [ERROR] Error response: no such file or directory. [2025-06-14 23:12:55.270890 +0100][Debug ][ExDbgMsg ][ 1314] [se1.farm.particle.cz:1094] Calling MsgHandler: 0x884e530 (message: kXR_open (file: /dune/RSE/fardet-hd/df/1a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_186_20231203T180154Z_gen_g4_detsim_hitreco__20240509T203907Z_reco2.root, mode: 00, flags: kXR_open_read kXR_async kXR_retstat ) ) with status: [ERROR] Error response: no such file or directory. [2025-06-14 23:12:55.270976 +0100][Debug ][File ][ 1314] [0x88cced0@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/df/1a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_186_20231203T180154Z_gen_g4_detsim_hitreco__20240509T203907Z_reco2.root?xrdcl.requuid=3933f958-c426-4dd2-aeda-b87c252c7d7f] Open has returned with status [ERROR] Server responded with an error: [3011] No such file [2025-06-14 23:12:55.270991 +0100][Debug ][File ][ 1314] [0x88cced0@root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/df/1a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_186_20231203T180154Z_gen_g4_detsim_hitreco__20240509T203907Z_reco2.root?xrdcl.requuid=3933f958-c426-4dd2-aeda-b87c252c7d7f] Error while opening at se1.farm.particle.cz:1094: [ERROR] Server responded with an error: [3011] No such file [2025-06-14 23:12:55.275362 +0100][Debug ][ExDbgMsg ][ 1314] [se1.farm.particle.cz:1094] Destroying MsgHandler: 0x884e530. %MSG-s ArtException: TriggerResultInserter:TriggerResults@Construction 14-Jun-2025 23:12:55 BST ModuleConstruction cet::exception caught in art ---- FileOpenError BEGIN RootInputFileSequence::initFile(): Input file root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/df/1a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_186_20231203T180154Z_gen_g4_detsim_hitreco__20240509T203907Z_reco2.root was not found or could not be opened. ---- FileOpenError END %MSG Art has completed and will exit with status 20. [2025-06-14 23:12:55.396132 +0100][Debug ][JobMgr ][ 1314] Stopping the job manager... [2025-06-14 23:12:55.397041 +0100][Debug ][JobMgr ][ 1314] Job manager stopped [2025-06-14 23:12:55.397485 +0100][Debug ][TaskMgr ][ 1314] Stopping the task manager... [2025-06-14 23:12:55.397553 +0100][Debug ][TaskMgr ][ 1314] Task manager stopped [2025-06-14 23:12:55.397561 +0100][Debug ][Poller ][ 1314] Stopping the poller... [2025-06-14 23:12:55.397771 +0100][Debug ][AsyncSock ][ 1314] [se1.farm.particle.cz:1094.0] Closing the socket [2025-06-14 23:12:55.397796 +0100][Debug ][Poller ][ 1314] <[::ffff:192.41.105.56]:48878><--><[::ffff:147.231.25.100]:1094> Removing socket from the poller [2025-06-14 23:12:55.397925 +0100][Debug ][PostMaster ][ 1314] [se1.farm.particle.cz:1094] Destroying stream [2025-06-14 23:12:55.397947 +0100][Debug ][AsyncSock ][ 1314] [se1.farm.particle.cz:1094.0] Closing the socket === End last 100 lines of lar log file === .: total 60 -rw-r--r-- 1 gl05pi6 eddie_users 33080 Jun 14 23:12 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_186_20231203T180154Z_gen_g4_detsim_hitreco__20240509T203907Z_reco2_reco_2025-06-14T_221244Z.log -rw-r--r-- 1 gl05pi6 eddie_users 16317 Jun 14 23:12 jobscript.log -rw-r--r-- 1 gl05pi6 eddie_users 519 Jun 14 23:12 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577091_186_20231203T180154Z_gen_g4_detsim_hitreco__20240509T203907Z_reco2_pida_2025-06-14T_221244Z.root -rw-r--r-- 1 gl05pi6 eddie_users 138 Jun 14 23:12 all-input-dids.txt -rw-r--r-- 1 gl05pi6 eddie_users 0 Jun 14 23:12 debugprod.log