justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 265321.0@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID265321.0@justin-prod-sched01.dune.hep.ac.uk
Workflow ID3537
Stage ID1
User namegvittist@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-09-27 19:16:54
SiteUK_Edinburgh
EntryDUNE_UK_SGridECDF_ce1
Last heartbeat2024-09-27 19:20:19
From worker nodeHostnamenode3a01.ecdf.ed.ac.uk
cpuinfoIntel(R) Xeon(R) Gold 6242 CPU @ 2.80GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
Inner Apptainer?True
Job statefinished
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-09-27 19:18:33
Input fileshd-protodune:np04_hd_run023169_0001_dataflow0_datawriter_0_20231113T082725.hdf5
JobscriptExit code0
Real time1m (63s)
CPU time0m (48s = 76%)
Outputting started2024-09-27 19:19:37
Output filesusertests:np04_hd_run023169_0001_dataflow0_datawriter_0_20231113T082725_reco_2024-09-27T_191836Z.root
Finished2024-09-27 19:20:19
Saved logsjustin-logs:265321.0-justin-prod-sched01.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

ooipolder-04.nikhef.nl:33279] Moving MsgHandler: 0xe812db0 (message: kXR_read (handle: 0x00000000, offset: 28168, size: 512) ) from out-queu to in-queue.
[2024-09-27 20:19:14.222524 +0100][Debug  ][ExDbgMsg          ][ 1288] [msg: 0xe813c70] Assigned MsgHandler: 0xe812db0.
[2024-09-27 20:19:14.222584 +0100][Debug  ][ExDbgMsg          ][ 1288] [handler: 0xe812db0] Removed MsgHandler: 0xe812db0 from the in-queue.
[2024-09-27 20:19:14.222636 +0100][Debug  ][ExDbgMsg          ][ 1288] [hooipolder-04.nikhef.nl:33279] Calling MsgHandler: 0xe812db0 (message: kXR_read (handle: 0x00000000, offset: 28168, size: 512) ) with status: [SUCCESS] .
[2024-09-27 20:19:14.222662 +0100][Debug  ][ExDbgMsg          ][ 1288] [hooipolder-04.nikhef.nl:33279] Destroying MsgHandler: 0xe812db0.
[2024-09-27 20:19:14.222820 +0100][Debug  ][File              ][ 1288] [0x11460960@root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/hd-protodune/47/9c/np04_hd_run023169_0001_dataflow0_datawriter_0_20231113T082725.hdf5?xrdcl.requuid=61b10b62-87b1-43b1-944c-abc2593b7686] Sending a read command for handle 0x0 to hooipolder-04.nikhef.nl:33279
[2024-09-27 20:19:14.222854 +0100][Debug  ][ExDbgMsg          ][ 1288] [hooipolder-04.nikhef.nl:33279] MsgHandler created: 0x144f0230 (message: kXR_read (handle: 0x00000000, offset: 29756, size: 128) ).
[2024-09-27 20:19:14.222932 +0100][Debug  ][ExDbgMsg          ][ 1288] [hooipolder-04.nikhef.nl:33279] Moving MsgHandler: 0x144f0230 (message: kXR_read (handle: 0x00000000, offset: 29756, size: 128) ) from out-queu to in-queue.
[2024-09-27 20:19:14.240467 +0100][Debug  ][ExDbgMsg          ][ 1288] [msg: 0x144f0c20] Assigned MsgHandler: 0x144f0230.
[2024-09-27 20:19:14.240532 +0100][Debug  ][ExDbgMsg          ][ 1288] [handler: 0x144f0230] Removed MsgHandler: 0x144f0230 from the in-queue.
[2024-09-27 20:19:14.240589 +0100][Debug  ][ExDbgMsg          ][ 1288] [hooipolder-04.nikhef.nl:33279] Calling MsgHandler: 0x144f0230 (message: kXR_read (handle: 0x00000000, offset: 29756, size: 128) ) with status: [SUCCESS] .
[2024-09-27 20:19:14.240618 +0100][Debug  ][ExDbgMsg          ][ 1288] [hooipolder-04.nikhef.nl:33279] Destroying MsgHandler: 0x144f0230.
[2024-09-27 20:19:14.241881 +0100][Debug  ][File              ][ 1288] [0x11460960@root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/hd-protodune/47/9c/np04_hd_run023169_0001_dataflow0_datawriter_0_20231113T082725.hdf5?xrdcl.requuid=61b10b62-87b1-43b1-944c-abc2593b7686] Sending a read command for handle 0x0 to hooipolder-04.nikhef.nl:33279
[2024-09-27 20:19:14.241938 +0100][Debug  ][ExDbgMsg          ][ 1288] [hooipolder-04.nikhef.nl:33279] MsgHandler created: 0x144f0e80 (message: kXR_read (handle: 0x00000000, offset: 22368, size: 5240) ).
[2024-09-27 20:19:14.242051 +0100][Debug  ][ExDbgMsg          ][ 1288] [hooipolder-04.nikhef.nl:33279] Moving MsgHandler: 0x144f0e80 (message: kXR_read (handle: 0x00000000, offset: 22368, size: 5240) ) from out-queu to in-queue.
[2024-09-27 20:19:14.259670 +0100][Debug  ][ExDbgMsg          ][ 1288] [msg: 0xe812540] Assigned MsgHandler: 0x144f0e80.
[2024-09-27 20:19:14.259743 +0100][Debug  ][ExDbgMsg          ][ 1288] [handler: 0x144f0e80] Removed MsgHandler: 0x144f0e80 from the in-queue.
[2024-09-27 20:19:14.259816 +0100][Debug  ][ExDbgMsg          ][ 1288] [hooipolder-04.nikhef.nl:33279] Calling MsgHandler: 0x144f0e80 (message: kXR_read (handle: 0x00000000, offset: 22368, size: 5240) ) with status: [SUCCESS] .
[2024-09-27 20:19:14.259850 +0100][Debug  ][ExDbgMsg          ][ 1288] [hooipolder-04.nikhef.nl:33279] Destroying MsgHandler: 0x144f0e80.
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=8192 keeping as is
Retagger: tagging trace set: raw with 10240 traces, 0 summary
wclsFrameSaver: saving raw::RawDigits tagged "raw"
wclsFrameSaver: found empty channel masks for "bad"
27-Sep-2024 20:19:27 BST  Opened output file with pattern "np04_hd_run023169_0001_dataflow0_datawriter_0_20231113T082725_reco_2024-09-27T_191836Z.root"
27-Sep-2024 20:19:36 BST  Closed output file "np04_hd_run023169_0001_dataflow0_datawriter_0_20231113T082725_reco_2024-09-27T_191836Z.root"
[2024-09-27 20:19:36.544342 +0100][Debug  ][File              ][ 1288] [0x11460960@root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/hd-protodune/47/9c/np04_hd_run023169_0001_dataflow0_datawriter_0_20231113T082725.hdf5?xrdcl.requuid=61b10b62-87b1-43b1-944c-abc2593b7686] Sending a close command for handle 0x0 to hooipolder-04.nikhef.nl:33279
[2024-09-27 20:19:36.544428 +0100][Debug  ][ExDbgMsg          ][ 1288] [hooipolder-04.nikhef.nl:33279] MsgHandler created: 0x2b3c3230 (message: kXR_close (handle: 0x00000000) ).
[2024-09-27 20:19:36.544578 +0100][Debug  ][ExDbgMsg          ][ 1288] [hooipolder-04.nikhef.nl:33279] Moving MsgHandler: 0x2b3c3230 (message: kXR_close (handle: 0x00000000) ) from out-queu to in-queue.
[2024-09-27 20:19:36.568630 +0100][Debug  ][ExDbgMsg          ][ 1288] [msg: 0x2b3b12d0] Assigned MsgHandler: 0x2b3c3230.
[2024-09-27 20:19:36.568662 +0100][Debug  ][ExDbgMsg          ][ 1288] [handler: 0x2b3c3230] Removed MsgHandler: 0x2b3c3230 from the in-queue.
[2024-09-27 20:19:36.568710 +0100][Debug  ][ExDbgMsg          ][ 1288] [hooipolder-04.nikhef.nl:33279] Calling MsgHandler: 0x2b3c3230 (message: kXR_close (handle: 0x00000000) ) with status: [SUCCESS] .
[2024-09-27 20:19:36.568750 +0100][Debug  ][File              ][ 1288] [0x11460960@root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/hd-protodune/47/9c/np04_hd_run023169_0001_dataflow0_datawriter_0_20231113T082725.hdf5?xrdcl.requuid=61b10b62-87b1-43b1-944c-abc2593b7686] Close returned from hooipolder-04.nikhef.nl:33279 with: [SUCCESS] 
[2024-09-27 20:19:36.568788 +0100][Debug  ][ExDbgMsg          ][ 1288] [hooipolder-04.nikhef.nl:33279] Destroying MsgHandler: 0x2b3c3230.

===================================================================================================================================
TimeTracker printout (sec)                           Min           Avg           Max         Median          RMS         nEvts   
===================================================================================================================================
Full event                                         32.7845       32.7845       32.7845       32.7845          0            1     
-----------------------------------------------------------------------------------------------------------------------------------
source:HDF5RawInput3(read)                       0.000150706   0.000150706   0.000150706   0.000150706        0            1     
produce:tpcrawdecoder:PDHDTPCReader                12.4762       12.4762       12.4762       12.4762          0            1     
produce:triggerrawdecoder:PDHDTriggerReader3      0.0368378     0.0368378     0.0368378     0.0368378         0            1     
produce:timingrawdecoder:PDHDTimingRawDecoder     0.0186176     0.0186176     0.0186176     0.0186176         0            1     
produce:pdhddaphne:DAPHNEReaderPDHD              0.00116861    0.00116861    0.00116861    0.00116861         0            1     
produce:fembfilter:PDHDFEMBFilter                0.000511761   0.000511761   0.000511761   0.000511761        0            1     
produce:wclsdatahdfilter:WireCellToolkit           13.2667       13.2667       13.2667       13.2667          0            1     
[art]:TriggerResults:TriggerResultInserter       7.4144e-05    7.4144e-05    7.4144e-05    7.4144e-05         0            1     
end_path:out1:RootOutput                         1.6508e-05    1.6508e-05    1.6508e-05    1.6508e-05         0            1     
end_path:out1:RootOutput(write)                    6.98303       6.98303       6.98303       6.98303          0            1     
===================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 3423.16 MB
  Peak resident set size usage (VmHWM): 2091.02 MB
  Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
[2024-09-27 20:19:37.448010 +0100][Debug  ][JobMgr            ][ 1288] Stopping the job manager...
[2024-09-27 20:19:37.448888 +0100][Debug  ][JobMgr            ][ 1288] Job manager stopped
[2024-09-27 20:19:37.449055 +0100][Debug  ][TaskMgr           ][ 1288] Stopping the task manager...
[2024-09-27 20:19:37.449133 +0100][Debug  ][TaskMgr           ][ 1288] Task manager stopped
[2024-09-27 20:19:37.449165 +0100][Debug  ][Poller            ][ 1288] Stopping the poller...
[2024-09-27 20:19:37.449579 +0100][Debug  ][AsyncSock         ][ 1288] [dune.dcache.nikhef.nl:1094.0] Closing the socket
[2024-09-27 20:19:37.449892 +0100][Debug  ][Poller            ][ 1288] <[::ffff:192.41.104.55]:58856><--><[::ffff:194.171.96.149]:1094> Removing socket from the poller
[2024-09-27 20:19:37.449968 +0100][Debug  ][PostMaster        ][ 1288] [dune.dcache.nikhef.nl:1094] Destroying stream
[2024-09-27 20:19:37.450000 +0100][Debug  ][AsyncSock         ][ 1288] [dune.dcache.nikhef.nl:1094.0] Closing the socket
[2024-09-27 20:19:37.450021 +0100][Debug  ][AsyncSock         ][ 1288] [hooipolder-04.nikhef.nl:33279.0] Closing the socket
[2024-09-27 20:19:37.450045 +0100][Debug  ][Poller            ][ 1288] <[::ffff:192.41.104.55]:44990><--><[::ffff:194.171.96.152]:33279> Removing socket from the poller
[2024-09-27 20:19:37.450114 +0100][Debug  ][PostMaster        ][ 1288] [hooipolder-04.nikhef.nl:33279] Destroying stream
[2024-09-27 20:19:37.450151 +0100][Debug  ][AsyncSock         ][ 1288] [hooipolder-04.nikhef.nl:33279.0] Closing the socket
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2024-09-29 09:32:35 UTC       justIN version: 01.01.08