21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 413943.58@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 413943.58@justin-prod-sched01.dune.hep.ac.uk | |
| Workflow ID | 7650 | |
| Stage ID | 1 | |
| User name | hsouza@fnal.gov | |
| HTCondor Group | group_dune | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 3145728000 (3000 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-06-14 22:57:06 | |
| Site | UK_Edinburgh | |
| Entry | DUNE_UK_SGridECDF_ce1_multicore | |
| Last heartbeat | 2025-06-14 23:25:13 | |
| From worker node | Hostname | node2b23.ecdf.ed.ac.uk |
| cpuinfo | Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 3145728000 (3000 MiB) | |
| Wall seconds limit | 171000 (47 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | outputting_failed | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-06-14 22:58:41 | |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577113_1143_20231204T041046Z_gen_g4_detsim_hitreco__20240509T213133Z_reco2.root | |
| Jobscript | Exit code | 0 |
| Real time | 3m (180s) | |
| CPU time | 0m (38s = 21%) | |
| Max RSS bytes | 1300865024 (1240 MiB) | |
| Outputting started | 2025-06-14 23:01:42 | |
| Output files | ||
| Finished | 2025-06-14 23:25:13 | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
fd48a0] Assigned MsgHandler: 0xe32a9e0. [2025-06-15 00:01:42.049426 +0100][Debug ][ExDbgMsg ][ 1314] [handler: 0xe32a9e0] Removed MsgHandler: 0xe32a9e0 from the in-queue. [2025-06-15 00:01:42.049461 +0100][Debug ][ExDbgMsg ][ 1314] [hooipolder-03.nikhef.nl:33297] Calling MsgHandler: 0xe32a9e0 (message: kXR_read (handle: 0x00000000, offset: 1472468667, size: 107) ) with status: [SUCCESS] . [2025-06-15 00:01:42.049477 +0100][Debug ][ExDbgMsg ][ 1314] [hooipolder-03.nikhef.nl:33297] Destroying MsgHandler: 0xe32a9e0. [2025-06-15 00:01:42.049509 +0100][Debug ][File ][ 1314] [0x7a20e40@root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/2f/be/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577113_1143_20231204T041046Z_gen_g4_detsim_hitreco__20240509T213133Z_reco2.root?xrdcl.requuid=e517610c-2135-41e0-b765-9cfe827dcfd8] Sending a read command for handle 0x0 to hooipolder-03.nikhef.nl:33297 [2025-06-15 00:01:42.049531 +0100][Debug ][ExDbgMsg ][ 1314] [hooipolder-03.nikhef.nl:33297] MsgHandler created: 0x11fd44e0 (message: kXR_read (handle: 0x00000000, offset: 1472468774, size: 113) ). [2025-06-15 00:01:42.049585 +0100][Debug ][ExDbgMsg ][ 1314] [hooipolder-03.nikhef.nl:33297] Moving MsgHandler: 0x11fd44e0 (message: kXR_read (handle: 0x00000000, offset: 1472468774, size: 113) ) from out-queu to in-queue. [2025-06-15 00:01:42.067624 +0100][Debug ][ExDbgMsg ][ 1314] [msg: 0xe32ada0] Assigned MsgHandler: 0x11fd44e0. [2025-06-15 00:01:42.067661 +0100][Debug ][ExDbgMsg ][ 1314] [handler: 0x11fd44e0] Removed MsgHandler: 0x11fd44e0 from the in-queue. [2025-06-15 00:01:42.067696 +0100][Debug ][ExDbgMsg ][ 1314] [hooipolder-03.nikhef.nl:33297] Calling MsgHandler: 0x11fd44e0 (message: kXR_read (handle: 0x00000000, offset: 1472468774, size: 113) ) with status: [SUCCESS] . [2025-06-15 00:01:42.067710 +0100][Debug ][ExDbgMsg ][ 1314] [hooipolder-03.nikhef.nl:33297] Destroying MsgHandler: 0x11fd44e0. [2025-06-15 00:01:42.067743 +0100][Debug ][File ][ 1314] [0x7a20e40@root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/2f/be/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577113_1143_20231204T041046Z_gen_g4_detsim_hitreco__20240509T213133Z_reco2.root?xrdcl.requuid=e517610c-2135-41e0-b765-9cfe827dcfd8] Sending a read command for handle 0x0 to hooipolder-03.nikhef.nl:33297 [2025-06-15 00:01:42.067763 +0100][Debug ][ExDbgMsg ][ 1314] [hooipolder-03.nikhef.nl:33297] MsgHandler created: 0xe32a9e0 (message: kXR_read (handle: 0x00000000, offset: 1472468887, size: 226) ). [2025-06-15 00:01:42.067816 +0100][Debug ][ExDbgMsg ][ 1314] [hooipolder-03.nikhef.nl:33297] Moving MsgHandler: 0xe32a9e0 (message: kXR_read (handle: 0x00000000, offset: 1472468887, size: 226) ) from out-queu to in-queue. [2025-06-15 00:01:42.085770 +0100][Debug ][ExDbgMsg ][ 1314] [msg: 0x11fd48a0] Assigned MsgHandler: 0xe32a9e0. [2025-06-15 00:01:42.085808 +0100][Debug ][ExDbgMsg ][ 1314] [handler: 0xe32a9e0] Removed MsgHandler: 0xe32a9e0 from the in-queue. [2025-06-15 00:01:42.085837 +0100][Debug ][ExDbgMsg ][ 1314] [hooipolder-03.nikhef.nl:33297] Calling MsgHandler: 0xe32a9e0 (message: kXR_read (handle: 0x00000000, offset: 1472468887, size: 226) ) with status: [SUCCESS] . [2025-06-15 00:01:42.085853 +0100][Debug ][ExDbgMsg ][ 1314] [hooipolder-03.nikhef.nl:33297] Destroying MsgHandler: 0xe32a9e0. [2025-06-15 00:01:42.090700 +0100][Debug ][File ][ 1314] [0x7a20e40@root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/2f/be/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577113_1143_20231204T041046Z_gen_g4_detsim_hitreco__20240509T213133Z_reco2.root?xrdcl.requuid=e517610c-2135-41e0-b765-9cfe827dcfd8] Sending a close command for handle 0x0 to hooipolder-03.nikhef.nl:33297 [2025-06-15 00:01:42.090989 +0100][Debug ][ExDbgMsg ][ 1314] [hooipolder-03.nikhef.nl:33297] MsgHandler created: 0x16d63870 (message: kXR_close (handle: 0x00000000) ). [2025-06-15 00:01:42.091083 +0100][Debug ][ExDbgMsg ][ 1314] [hooipolder-03.nikhef.nl:33297] Moving MsgHandler: 0x16d63870 (message: kXR_close (handle: 0x00000000) ) from out-queu to in-queue. [2025-06-15 00:01:42.110408 +0100][Debug ][ExDbgMsg ][ 1314] [msg: 0xb27ca80] Assigned MsgHandler: 0x16d63870. [2025-06-15 00:01:42.110452 +0100][Debug ][ExDbgMsg ][ 1314] [handler: 0x16d63870] Removed MsgHandler: 0x16d63870 from the in-queue. [2025-06-15 00:01:42.110486 +0100][Debug ][ExDbgMsg ][ 1314] [hooipolder-03.nikhef.nl:33297] Calling MsgHandler: 0x16d63870 (message: kXR_close (handle: 0x00000000) ) with status: [SUCCESS] . [2025-06-15 00:01:42.110511 +0100][Debug ][File ][ 1314] [0x7a20e40@root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/2f/be/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577113_1143_20231204T041046Z_gen_g4_detsim_hitreco__20240509T213133Z_reco2.root?xrdcl.requuid=e517610c-2135-41e0-b765-9cfe827dcfd8] Close returned from hooipolder-03.nikhef.nl:33297 with: [SUCCESS] [2025-06-15 00:01:42.110532 +0100][Debug ][ExDbgMsg ][ 1314] [hooipolder-03.nikhef.nl:33297] Destroying MsgHandler: 0x16d63870. 15-Jun-2025 00:01:42 BST Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/2f/be/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577113_1143_20231204T041046Z_gen_g4_detsim_hitreco__20240509T213133Z_reco2.root" ================================================================================================================================ TimeTracker printout (sec) Min Avg Max Median RMS nEvts ================================================================================================================================ Full event 0.343965 1.3015 3.92855 1.22459 0.514535 100 -------------------------------------------------------------------------------------------------------------------------------- source:RootInput(read) 0.0184286 0.038051 0.0812811 0.0367474 0.0134853 100 prod:pandorapid:Chi2ParticleID 0.108221 0.188595 0.388574 0.189586 0.0399789 100 prod:energyreconue:EnergyReco 0.135544 0.435282 0.992082 0.408734 0.129409 100 prod:energyreconumu:EnergyReco 0.00231512 0.115202 1.40026 0.111004 0.155569 100 prod:energyreconc:EnergyReco 0.00281511 0.042268 0.281351 0.0325751 0.0472152 100 [art]:TriggerResults:TriggerResultInserter 9.814e-06 1.2244e-05 6.8051e-05 1.1023e-05 6.85594e-06 100 end_path:ana:PIDASelectionAna 0.075237 0.481768 1.76074 0.441508 0.230961 100 ================================================================================================================================ ==================================================================================================== MemoryTracker summary (base-10 MB units used) Peak virtual memory usage (VmPeak) : 11081.7 MB Peak resident set size usage (VmHWM): 1300.87 MB ==================================================================================================== Art has completed and will exit with status 0. [2025-06-15 00:01:42.318260 +0100][Debug ][JobMgr ][ 1314] Stopping the job manager... [2025-06-15 00:01:42.319218 +0100][Debug ][JobMgr ][ 1314] Job manager stopped [2025-06-15 00:01:42.319663 +0100][Debug ][TaskMgr ][ 1314] Stopping the task manager... [2025-06-15 00:01:42.319913 +0100][Debug ][TaskMgr ][ 1314] Task manager stopped [2025-06-15 00:01:42.319963 +0100][Debug ][Poller ][ 1314] Stopping the poller... [2025-06-15 00:01:42.320183 +0100][Debug ][AsyncSock ][ 1314] [dune.dcache.nikhef.nl:1094.0] Closing the socket [2025-06-15 00:01:42.320214 +0100][Debug ][Poller ][ 1314] <[::ffff:192.41.105.56]:43582><--><[::ffff:194.171.96.151]:1094> Removing socket from the poller [2025-06-15 00:01:42.320278 +0100][Debug ][PostMaster ][ 1314] [dune.dcache.nikhef.nl:1094] Destroying stream [2025-06-15 00:01:42.320298 +0100][Debug ][AsyncSock ][ 1314] [dune.dcache.nikhef.nl:1094.0] Closing the socket [2025-06-15 00:01:42.320319 +0100][Debug ][AsyncSock ][ 1314] [hooipolder-03.nikhef.nl:33297.0] Closing the socket [2025-06-15 00:01:42.320371 +0100][Debug ][Poller ][ 1314] <[::ffff:192.41.105.56]:44656><--><[::ffff:194.171.96.151]:33297> Removing socket from the poller [2025-06-15 00:01:42.320532 +0100][Debug ][PostMaster ][ 1314] [hooipolder-03.nikhef.nl:33297] Destroying stream [2025-06-15 00:01:42.320538 +0100][Debug ][AsyncSock ][ 1314] [hooipolder-03.nikhef.nl:33297.0] Closing the socket === End last 100 lines of lar log file === .: total 10044 -rw-r--r-- 1 gl05pi6 eddie_users 10176800 Jun 15 00:01 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577113_1143_20231204T041046Z_gen_g4_detsim_hitreco__20240509T213133Z_reco2_reco_2025-06-14T_225846Z.log -rw-r--r-- 1 gl05pi6 eddie_users 73860 Jun 15 00:01 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577113_1143_20231204T041046Z_gen_g4_detsim_hitreco__20240509T213133Z_reco2_pida_2025-06-14T_225846Z.root -rw-r--r-- 1 gl05pi6 eddie_users 19728 Jun 15 00:01 jobscript.log -rw-r--r-- 1 gl05pi6 eddie_users 218 Jun 15 00:01 justin-processed-pfns.txt -rw-r--r-- 1 gl05pi6 eddie_users 139 Jun 14 23:58 all-input-dids.txt -rw-r--r-- 1 gl05pi6 eddie_users 0 Jun 14 23:58 debugprod.log