Jobsub ID 394146.1@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 394146.1@justin-prod-sched01.dune.hep.ac.uk |
| Workflow ID | 6731 |
| Stage ID | 1 |
| User name | calcuttj@fnal.gov |
| HTCondor Group | group_dune |
| Requested | Processors | 1 |
| GPU | No |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 80000 (22 hours) |
| Submitted time | 2025-05-08 18:52:47 |
| Site | UK_Edinburgh |
| Entry | DUNE_UK_SGridECDF_ce1 |
| Last heartbeat | 2025-05-08 18:56:28 |
| From worker node | Hostname | node2b23.ecdf.ed.ac.uk |
| cpuinfo | Intel(R) Xeon(R) Gold 6338 CPU @ 2.00GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 171000 (47 hours) |
| GPU | |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2025-05-08 18:54:21 |
| Input files | monte-carlo-006731-000269
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Max RSS bytes | 0 (0 MiB) |
| Outputting started | |
| Output files | |
| Finished | 2025-05-08 18:56:28 |
| Saved logs | justin-logs:394146.1-justin-prod-sched01.dune.hep.ac.uk.logs.tgz |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
e560@root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/usertests/b4/97/H4_v34b_1GeV_-27.7_1_20250430T233848Z_006006.root?xrdcl.requuid=d23b583a-c705-4a07-85cb-b7685fcaaf5a] Close returned from st-120-100gb-pvrn9f.cern.ch:1095 with: [SUCCESS]
[2025-05-08 19:56:03.006479 +0100][Debug ][ExDbgMsg ] [st-120-100gb-pvrn9f.cern.ch:1095] Destroying MsgHandler: 0x33ca010.
[2025-05-08 19:56:03.006566 +0100][Debug ][File ] [0x3851c50@root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/4d/61/H4_v34b_1GeV_-27.7_1_20250501T001135Z_004446.root?xrdcl.requuid=76564b32-872b-4d38-81e9-abb5f657e7c6] Sending a close command for handle 0x0 to pubstor2302.fnal.gov:22471
[2025-05-08 19:56:03.006587 +0100][Debug ][ExDbgMsg ] [pubstor2302.fnal.gov:22471] MsgHandler created: 0x33ca010 (message: kXR_close (handle: 0x00000000) ).
[2025-05-08 19:56:03.006640 +0100][Debug ][ExDbgMsg ] [pubstor2302.fnal.gov:22471] Moving MsgHandler: 0x33ca010 (message: kXR_close (handle: 0x00000000) ) from out-queu to in-queue.
[2025-05-08 19:56:12.362194 +0100][Debug ][ExDbgMsg ] [msg: 0x32c2c20] Assigned MsgHandler: 0x33ca010.
[2025-05-08 19:56:12.362248 +0100][Debug ][ExDbgMsg ] [handler: 0x33ca010] Removed MsgHandler: 0x33ca010 from the in-queue.
[2025-05-08 19:56:12.362288 +0100][Debug ][ExDbgMsg ] [pubstor2302.fnal.gov:22471] Calling MsgHandler: 0x33ca010 (message: kXR_close (handle: 0x00000000) ) with status: [SUCCESS] .
[2025-05-08 19:56:12.362300 +0100][Debug ][File ] [0x3851c50@root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/4d/61/H4_v34b_1GeV_-27.7_1_20250501T001135Z_004446.root?xrdcl.requuid=76564b32-872b-4d38-81e9-abb5f657e7c6] Close returned from pubstor2302.fnal.gov:22471 with: [SUCCESS]
[2025-05-08 19:56:12.362315 +0100][Debug ][ExDbgMsg ] [pubstor2302.fnal.gov:22471] Destroying MsgHandler: 0x33ca010.
[2025-05-08 19:56:12.362730 +0100][Debug ][JobMgr ] Stopping the job manager...
[2025-05-08 19:56:12.362956 +0100][Debug ][JobMgr ] Job manager stopped
[2025-05-08 19:56:12.362970 +0100][Debug ][TaskMgr ] Stopping the task manager...
[2025-05-08 19:56:12.363117 +0100][Debug ][TaskMgr ] Task manager stopped
[2025-05-08 19:56:12.363127 +0100][Debug ][Poller ] Stopping the poller...
[2025-05-08 19:56:12.363222 +0100][Debug ][AsyncSock ] [ceph-svc02.gridpp.rl.ac.uk:1094.0] Closing the socket
[2025-05-08 19:56:12.363236 +0100][Debug ][Poller ] <[::ffff:192.41.105.56]:37438><--><[::ffff:130.246.178.115]:1094> Removing socket from the poller
[2025-05-08 19:56:12.363355 +0100][Debug ][PostMaster ] [ceph-svc02.gridpp.rl.ac.uk:1094] Destroying stream
[2025-05-08 19:56:12.363370 +0100][Debug ][AsyncSock ] [ceph-svc02.gridpp.rl.ac.uk:1094.0] Closing the socket
[2025-05-08 19:56:12.363387 +0100][Debug ][AsyncSock ] [ceph-svc17.gridpp.rl.ac.uk:1094.0] Closing the socket
[2025-05-08 19:56:12.363392 +0100][Debug ][Poller ] <[::ffff:192.41.105.56]:55548><--><[::ffff:130.246.179.51]:1094> Removing socket from the poller
[2025-05-08 19:56:12.363435 +0100][Debug ][PostMaster ] [ceph-svc17.gridpp.rl.ac.uk:1094] Destroying stream
[2025-05-08 19:56:12.363441 +0100][Debug ][AsyncSock ] [ceph-svc17.gridpp.rl.ac.uk:1094.0] Closing the socket
[2025-05-08 19:56:12.363448 +0100][Debug ][AsyncSock ] [eospublic.cern.ch:1094.0] Closing the socket
[2025-05-08 19:56:12.363453 +0100][Debug ][Poller ] <[::ffff:192.41.105.56]:60296><--><[::ffff:128.142.160.145]:1094> Removing socket from the poller
[2025-05-08 19:56:12.363463 +0100][Debug ][PostMaster ] [eospublic.cern.ch:1094] Destroying stream
[2025-05-08 19:56:12.363467 +0100][Debug ][AsyncSock ] [eospublic.cern.ch:1094.0] Closing the socket
[2025-05-08 19:56:12.363476 +0100][Debug ][AsyncSock ] [fndca1.fnal.gov:1094.0] Closing the socket
[2025-05-08 19:56:12.363481 +0100][Debug ][Poller ] <[::ffff:192.41.105.56]:48866><--><[::ffff:131.225.69.121]:1094> Removing socket from the poller
[2025-05-08 19:56:12.363512 +0100][Debug ][PostMaster ] [fndca1.fnal.gov:1094] Destroying stream
[2025-05-08 19:56:12.363516 +0100][Debug ][AsyncSock ] [fndca1.fnal.gov:1094.0] Closing the socket
[2025-05-08 19:56:12.363525 +0100][Debug ][AsyncSock ] [pubstor2302.fnal.gov:22471.0] Closing the socket
[2025-05-08 19:56:12.363529 +0100][Debug ][Poller ] <[::ffff:192.41.105.56]:48474><--><[::ffff:131.225.69.89]:22471> Removing socket from the poller
[2025-05-08 19:56:12.363537 +0100][Debug ][PostMaster ] [pubstor2302.fnal.gov:22471] Destroying stream
[2025-05-08 19:56:12.363541 +0100][Debug ][AsyncSock ] [pubstor2302.fnal.gov:22471.0] Closing the socket
[2025-05-08 19:56:12.363548 +0100][Debug ][AsyncSock ] [st-096-100gb-ip306-59247.cern.ch:1095.0] Closing the socket
[2025-05-08 19:56:12.363553 +0100][Debug ][Poller ] <[::ffff:192.41.105.56]:47836><--><[::ffff:128.142.218.82]:1095> Removing socket from the poller
[2025-05-08 19:56:12.363561 +0100][Debug ][PostMaster ] [st-096-100gb-ip306-59247.cern.ch:1095] Destroying stream
[2025-05-08 19:56:12.363565 +0100][Debug ][AsyncSock ] [st-096-100gb-ip306-59247.cern.ch:1095.0] Closing the socket
[2025-05-08 19:56:12.363574 +0100][Debug ][AsyncSock ] [st-120-100gb-pvrn9f.cern.ch:1095.0] Closing the socket
[2025-05-08 19:56:12.363578 +0100][Debug ][Poller ] <[::ffff:192.41.105.56]:35500><--><[::ffff:128.142.197.18]:1095> Removing socket from the poller
[2025-05-08 19:56:12.363586 +0100][Debug ][PostMaster ] [st-120-100gb-pvrn9f.cern.ch:1095] Destroying stream
[2025-05-08 19:56:12.363590 +0100][Debug ][AsyncSock ] [st-120-100gb-pvrn9f.cern.ch:1095.0] Closing the socket
[2025-05-08 19:56:12.363597 +0100][Debug ][AsyncSock ] [stkendca2042.fnal.gov:23550.0] Closing the socket
[2025-05-08 19:56:12.363601 +0100][Debug ][Poller ] <[::ffff:192.41.105.56]:46050><--><[::ffff:131.225.69.171]:23550> Removing socket from the poller
[2025-05-08 19:56:12.363610 +0100][Debug ][PostMaster ] [stkendca2042.fnal.gov:23550] Destroying stream
[2025-05-08 19:56:12.363614 +0100][Debug ][AsyncSock ] [stkendca2042.fnal.gov:23550.0] Closing the socket
[2025-05-08 19:56:12.363620 +0100][Debug ][AsyncSock ] [stkendca2044.fnal.gov:21418.0] Closing the socket
[2025-05-08 19:56:12.363624 +0100][Debug ][Poller ] <[::ffff:192.41.105.56]:53284><--><[::ffff:131.225.69.173]:21418> Removing socket from the poller
[2025-05-08 19:56:12.363632 +0100][Debug ][PostMaster ] [stkendca2044.fnal.gov:21418] Destroying stream
[2025-05-08 19:56:12.363636 +0100][Debug ][AsyncSock ] [stkendca2044.fnal.gov:21418.0] Closing the socket
[2025-05-08 19:56:12.363643 +0100][Debug ][AsyncSock ] [xrootd.echo.stfc.ac.uk:1094.0] Closing the socket
[2025-05-08 19:56:12.363647 +0100][Debug ][Poller ] <[::ffff:192.41.105.56]:48126><--><[::ffff:130.246.217.8]:1094> Removing socket from the poller
[2025-05-08 19:56:12.363724 +0100][Debug ][PostMaster ] [xrootd.echo.stfc.ac.uk:1094] Destroying stream
[2025-05-08 19:56:12.363733 +0100][Debug ][AsyncSock ] [xrootd.echo.stfc.ac.uk:1094.0] Closing the socket
Querying usertests:calcuttj_g4bl_prod_full_1_042825-w6502s1p1 for 10 files
Query: files from usertests:calcuttj_g4bl_prod_full_1_042825-w6502s1p1 where dune.output_status=confirmed ordered skip 2680 limit 10
Getting names and metadata
done
{'beam.momentum': 1.0, 'beam.polarity': 1, 'core.data_stream': 'g4beamline', 'core.data_tier': 'root-tuple', 'core.file_format': 'root', 'core.file_type': 'mc', 'core.group': 'dune', 'core.run_type': 'ehn1-beam-np04', 'dune.output_status': 'confirmed', 'retention.class': 'physics', 'retention.status': 'active', 'core.runs': [394146], 'core.runs_subruns': [39414600001]}
Getting paths from rucio
Got 10 paths from 10 files
['hadd', 'H4_v34b_1GeV_-27.7_1_394146_1_20250508T185424.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/b0/98/H4_v34b_1GeV_-27.7_1_20250430T201338Z_003245.root', 'root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/usertests/95/fe/H4_v34b_1GeV_-27.7_1_20250430T203117Z_003452.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/16/2e/H4_v34b_1GeV_-27.7_1_20250430T225455Z_004014.root', 'root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/usertests/b4/97/H4_v34b_1GeV_-27.7_1_20250430T233848Z_006006.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/4d/61/H4_v34b_1GeV_-27.7_1_20250501T001135Z_004446.root', 'root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/usertests/a3/c6/H4_v34b_1GeV_-27.7_1_20250501T010444Z_001661.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/71/77/H4_v34b_1GeV_-27.7_1_20250501T024522Z_004304.root', 'root://eospublic.cern.ch:1094//eos/experiment/neutplatform/protodune/dune/usertests/c1/82/H4_v34b_1GeV_-27.7_1_20250501T040028Z_009448.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/31/c6/H4_v34b_1GeV_-27.7_1_20250501T040919Z_009074.root', 'root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/a0/dc/H4_v34b_1GeV_-27.7_1_20250501T044002Z_008248.root']
Traceback (most recent call last):
File "/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/4e9b42dda8c1cbee7b07e2de7059f47384a3867b/merge_g4bl.py", line 259, in <module>
do_merge(args)
File "/cvmfs/fifeuser3.opensciencegrid.org/sw/dune/4e9b42dda8c1cbee7b07e2de7059f47384a3867b/merge_g4bl.py", line 111, in do_merge
raise Exception('Error in hadd')
Exception: Error in hadd
Exiting with error 