justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.

Jobsub ID 229377.123@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID229377.123@justin-prod-sched02.dune.hep.ac.uk
Workflow ID7926
Stage ID1
User namehsouza@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes3145728000 (3000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-06-25 16:01:20
SiteUK_RAL-PPD
EntryCMSHTPC_T2_UK_SGrid_RALPP_hep207
Last heartbeat2025-06-25 16:39:59
From worker nodeHostnameheplnc171.pp.rl.ac.uk
cpuinfoAMD EPYC 9534 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes3145728000 (3000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statestalled
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-06-25 16:24:17
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1793_102_20231027T174138Z_gen_g4_detsim_hitreco__20240503T041536Z_reco2.root
Outputting started2025-06-25 16:39:59
Output files
Finished2025-06-25 17:13:54
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

128th record. run: 1793 subRun: 1 event: 20528 at 25-Jun-2025 17:33:12 BST
Begin processing the 129th record. run: 1793 subRun: 1 event: 20529 at 25-Jun-2025 17:33:14 BST
Begin processing the 130th record. run: 1793 subRun: 1 event: 20530 at 25-Jun-2025 17:33:16 BST
Begin processing the 131st record. run: 1793 subRun: 1 event: 20531 at 25-Jun-2025 17:33:18 BST
Begin processing the 132nd record. run: 1793 subRun: 1 event: 20532 at 25-Jun-2025 17:33:19 BST
Begin processing the 133rd record. run: 1793 subRun: 1 event: 20533 at 25-Jun-2025 17:33:21 BST
Begin processing the 134th record. run: 1793 subRun: 1 event: 20534 at 25-Jun-2025 17:33:23 BST
Begin processing the 135th record. run: 1793 subRun: 1 event: 20535 at 25-Jun-2025 17:33:24 BST
Begin processing the 136th record. run: 1793 subRun: 1 event: 20536 at 25-Jun-2025 17:33:26 BST
Begin processing the 137th record. run: 1793 subRun: 1 event: 20537 at 25-Jun-2025 17:33:27 BST
Begin processing the 138th record. run: 1793 subRun: 1 event: 20538 at 25-Jun-2025 17:33:29 BST
Begin processing the 139th record. run: 1793 subRun: 1 event: 20539 at 25-Jun-2025 17:33:30 BST
Begin processing the 140th record. run: 1793 subRun: 1 event: 20540 at 25-Jun-2025 17:33:32 BST
Begin processing the 141st record. run: 1793 subRun: 1 event: 20541 at 25-Jun-2025 17:33:35 BST
Begin processing the 142nd record. run: 1793 subRun: 1 event: 20542 at 25-Jun-2025 17:33:36 BST
Begin processing the 143rd record. run: 1793 subRun: 1 event: 20543 at 25-Jun-2025 17:33:38 BST
Begin processing the 144th record. run: 1793 subRun: 1 event: 20544 at 25-Jun-2025 17:33:40 BST
Begin processing the 145th record. run: 1793 subRun: 1 event: 20545 at 25-Jun-2025 17:33:42 BST
Begin processing the 146th record. run: 1793 subRun: 1 event: 20546 at 25-Jun-2025 17:33:44 BST
Begin processing the 147th record. run: 1793 subRun: 1 event: 20547 at 25-Jun-2025 17:33:48 BST
Begin processing the 148th record. run: 1793 subRun: 1 event: 20548 at 25-Jun-2025 17:33:50 BST
Begin processing the 149th record. run: 1793 subRun: 1 event: 20549 at 25-Jun-2025 17:33:52 BST
Begin processing the 150th record. run: 1793 subRun: 1 event: 20550 at 25-Jun-2025 17:33:55 BST
Begin processing the 151st record. run: 1793 subRun: 1 event: 20551 at 25-Jun-2025 17:33:55 BST
Begin processing the 152nd record. run: 1793 subRun: 1 event: 20552 at 25-Jun-2025 17:33:57 BST
Begin processing the 153rd record. run: 1793 subRun: 1 event: 20553 at 25-Jun-2025 17:33:59 BST
Begin processing the 154th record. run: 1793 subRun: 1 event: 20554 at 25-Jun-2025 17:34:00 BST
Begin processing the 155th record. run: 1793 subRun: 1 event: 20555 at 25-Jun-2025 17:34:02 BST
Begin processing the 156th record. run: 1793 subRun: 1 event: 20556 at 25-Jun-2025 17:34:04 BST
Begin processing the 157th record. run: 1793 subRun: 1 event: 20557 at 25-Jun-2025 17:34:05 BST
Begin processing the 158th record. run: 1793 subRun: 1 event: 20558 at 25-Jun-2025 17:34:07 BST
Begin processing the 159th record. run: 1793 subRun: 1 event: 20559 at 25-Jun-2025 17:34:08 BST
Begin processing the 160th record. run: 1793 subRun: 1 event: 20560 at 25-Jun-2025 17:34:10 BST
Begin processing the 161st record. run: 1793 subRun: 1 event: 20561 at 25-Jun-2025 17:34:12 BST
Begin processing the 162nd record. run: 1793 subRun: 1 event: 20562 at 25-Jun-2025 17:34:13 BST
Begin processing the 163rd record. run: 1793 subRun: 1 event: 20563 at 25-Jun-2025 17:34:14 BST
Begin processing the 164th record. run: 1793 subRun: 1 event: 20564 at 25-Jun-2025 17:34:16 BST
Begin processing the 165th record. run: 1793 subRun: 1 event: 20565 at 25-Jun-2025 17:34:18 BST
Begin processing the 166th record. run: 1793 subRun: 1 event: 20566 at 25-Jun-2025 17:34:19 BST
Begin processing the 167th record. run: 1793 subRun: 1 event: 20567 at 25-Jun-2025 17:34:21 BST
Begin processing the 168th record. run: 1793 subRun: 1 event: 20568 at 25-Jun-2025 17:34:22 BST
Begin processing the 169th record. run: 1793 subRun: 1 event: 20569 at 25-Jun-2025 17:34:24 BST
Begin processing the 170th record. run: 1793 subRun: 1 event: 20570 at 25-Jun-2025 17:34:26 BST
Begin processing the 171st record. run: 1793 subRun: 1 event: 20571 at 25-Jun-2025 17:34:28 BST
Begin processing the 172nd record. run: 1793 subRun: 1 event: 20572 at 25-Jun-2025 17:34:29 BST
Begin processing the 173rd record. run: 1793 subRun: 1 event: 20573 at 25-Jun-2025 17:34:31 BST
Begin processing the 174th record. run: 1793 subRun: 1 event: 20574 at 25-Jun-2025 17:34:33 BST
Begin processing the 175th record. run: 1793 subRun: 1 event: 20575 at 25-Jun-2025 17:34:33 BST
Begin processing the 176th record. run: 1793 subRun: 1 event: 20576 at 25-Jun-2025 17:34:35 BST
Begin processing the 177th record. run: 1793 subRun: 1 event: 20577 at 25-Jun-2025 17:34:37 BST
Begin processing the 178th record. run: 1793 subRun: 1 event: 20578 at 25-Jun-2025 17:34:38 BST
Begin processing the 179th record. run: 1793 subRun: 1 event: 20579 at 25-Jun-2025 17:34:40 BST
Begin processing the 180th record. run: 1793 subRun: 1 event: 20580 at 25-Jun-2025 17:34:42 BST
Begin processing the 181st record. run: 1793 subRun: 1 event: 20581 at 25-Jun-2025 17:34:43 BST
Begin processing the 182nd record. run: 1793 subRun: 1 event: 20582 at 25-Jun-2025 17:34:45 BST
Begin processing the 183rd record. run: 1793 subRun: 1 event: 20583 at 25-Jun-2025 17:34:46 BST
Begin processing the 184th record. run: 1793 subRun: 1 event: 20584 at 25-Jun-2025 17:34:46 BST
Begin processing the 185th record. run: 1793 subRun: 1 event: 20585 at 25-Jun-2025 17:34:47 BST
Begin processing the 186th record. run: 1793 subRun: 1 event: 20586 at 25-Jun-2025 17:34:49 BST
Begin processing the 187th record. run: 1793 subRun: 1 event: 20587 at 25-Jun-2025 17:34:49 BST
Begin processing the 188th record. run: 1793 subRun: 1 event: 20588 at 25-Jun-2025 17:34:51 BST
Begin processing the 189th record. run: 1793 subRun: 1 event: 20589 at 25-Jun-2025 17:34:52 BST
Begin processing the 190th record. run: 1793 subRun: 1 event: 20590 at 25-Jun-2025 17:34:54 BST
Begin processing the 191st record. run: 1793 subRun: 1 event: 20591 at 25-Jun-2025 17:34:56 BST
Begin processing the 192nd record. run: 1793 subRun: 1 event: 20592 at 25-Jun-2025 17:34:57 BST
Begin processing the 193rd record. run: 1793 subRun: 1 event: 20593 at 25-Jun-2025 17:34:59 BST
Begin processing the 194th record. run: 1793 subRun: 1 event: 20594 at 25-Jun-2025 17:35:01 BST
Begin processing the 195th record. run: 1793 subRun: 1 event: 20595 at 25-Jun-2025 17:35:03 BST
Begin processing the 196th record. run: 1793 subRun: 1 event: 20596 at 25-Jun-2025 17:35:04 BST
Begin processing the 197th record. run: 1793 subRun: 1 event: 20597 at 25-Jun-2025 17:35:06 BST
Begin processing the 198th record. run: 1793 subRun: 1 event: 20598 at 25-Jun-2025 17:35:09 BST
Begin processing the 199th record. run: 1793 subRun: 1 event: 20599 at 25-Jun-2025 17:35:11 BST
Begin processing the 200th record. run: 1793 subRun: 1 event: 20600 at 25-Jun-2025 17:35:16 BST
25-Jun-2025 17:35:18 BST  Closed input file "root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/45/2d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1793_102_20231027T174138Z_gen_g4_detsim_hitreco__20240503T041536Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.399422       1.40864       3.76463       1.40443      0.419591        200    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.022405      0.0386335     0.0793405     0.0442678     0.0130914       200    
prod:pandorapid:Chi2ParticleID                 0.131498      0.197308      0.272329      0.201659      0.031026        200    
prod:energyreconue:EnergyReco                  0.231703      0.493282      0.873735      0.473634      0.113164        200    
prod:energyreconumu:EnergyReco                0.00114652     0.118931       1.69311      0.0996642     0.154632        200    
prod:energyreconc:EnergyReco                   0.0026408     0.0277493     0.236282      0.0327788     0.031223        200    
[art]:TriggerResults:TriggerResultInserter     1.119e-05    1.73814e-05   0.000110162   1.65505e-05   7.33225e-06      200    
end_path:ana:PIDASelectionAna                 0.00318165      0.53211       1.71179      0.527214      0.206493        200    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 36875.4 MB
  Peak resident set size usage (VmHWM): 1256.82 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
.:
total 72
-rw-r--r-- 1 pltdune01 pltdune 28229 Jun 25 17:35 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1793_102_20231027T174138Z_gen_g4_detsim_hitreco__20240503T041536Z_reco2_pida_2025-06-25T_162914Z.root
-rw-r--r-- 1 pltdune01 pltdune 23406 Jun 25 17:35 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1793_102_20231027T174138Z_gen_g4_detsim_hitreco__20240503T041536Z_reco2_reco_2025-06-25T_162914Z.log
-rw-r--r-- 1 pltdune01 pltdune 11733 Jun 25 17:35 jobscript.log
-rw-r--r-- 1 pltdune01 pltdune   183 Jun 25 17:35 justin-processed-pfns.txt
-rw-r--r-- 1 pltdune01 pltdune   134 Jun 25 17:29 all-input-dids.txt
-rw-r--r-- 1 pltdune01 pltdune     0 Jun 25 17:29 debugprod.log
justIN time: 2025-08-14 21:52:29 UTC       justIN version: 01.03.02