justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.

Jobsub ID 421771.191@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID421771.191@justin-prod-sched01.dune.hep.ac.uk
Workflow ID7925
Stage ID1
User namehsouza@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes3145728000 (3000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-06-25 15:54:39
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce02
Last heartbeat2025-06-25 16:33:32
From worker nodeHostnamewn1207250.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7702P 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes3145728000 (3000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statestalled
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-06-25 16:02:24
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1498_119_20231027T230240Z_gen_g4_detsim_hitreco__20240503T043155Z_reco2.root
Outputting started2025-06-25 16:10:10
Output files
Finished2025-06-25 17:13:54
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

ST
Begin processing the 128th record. run: 1498 subRun: 1 event: 23928 at 25-Jun-2025 17:07:12 BST
Begin processing the 129th record. run: 1498 subRun: 1 event: 23929 at 25-Jun-2025 17:07:14 BST
Begin processing the 130th record. run: 1498 subRun: 1 event: 23930 at 25-Jun-2025 17:07:15 BST
Begin processing the 131st record. run: 1498 subRun: 1 event: 23931 at 25-Jun-2025 17:07:17 BST
Begin processing the 132nd record. run: 1498 subRun: 1 event: 23932 at 25-Jun-2025 17:07:19 BST
Begin processing the 133rd record. run: 1498 subRun: 1 event: 23933 at 25-Jun-2025 17:07:21 BST
Begin processing the 134th record. run: 1498 subRun: 1 event: 23934 at 25-Jun-2025 17:07:23 BST
Begin processing the 135th record. run: 1498 subRun: 1 event: 23935 at 25-Jun-2025 17:07:25 BST
Begin processing the 136th record. run: 1498 subRun: 1 event: 23936 at 25-Jun-2025 17:07:27 BST
Begin processing the 137th record. run: 1498 subRun: 1 event: 23937 at 25-Jun-2025 17:07:29 BST
Begin processing the 138th record. run: 1498 subRun: 1 event: 23938 at 25-Jun-2025 17:07:31 BST
Begin processing the 139th record. run: 1498 subRun: 1 event: 23939 at 25-Jun-2025 17:07:33 BST
Begin processing the 140th record. run: 1498 subRun: 1 event: 23940 at 25-Jun-2025 17:07:35 BST
Begin processing the 141st record. run: 1498 subRun: 1 event: 23941 at 25-Jun-2025 17:07:37 BST
Begin processing the 142nd record. run: 1498 subRun: 1 event: 23942 at 25-Jun-2025 17:07:39 BST
Begin processing the 143rd record. run: 1498 subRun: 1 event: 23943 at 25-Jun-2025 17:07:42 BST
Begin processing the 144th record. run: 1498 subRun: 1 event: 23944 at 25-Jun-2025 17:07:44 BST
Begin processing the 145th record. run: 1498 subRun: 1 event: 23945 at 25-Jun-2025 17:07:46 BST
Begin processing the 146th record. run: 1498 subRun: 1 event: 23946 at 25-Jun-2025 17:07:48 BST
Begin processing the 147th record. run: 1498 subRun: 1 event: 23947 at 25-Jun-2025 17:07:50 BST
Begin processing the 148th record. run: 1498 subRun: 1 event: 23948 at 25-Jun-2025 17:07:52 BST
Begin processing the 149th record. run: 1498 subRun: 1 event: 23949 at 25-Jun-2025 17:07:54 BST
Begin processing the 150th record. run: 1498 subRun: 1 event: 23950 at 25-Jun-2025 17:07:55 BST
Begin processing the 151st record. run: 1498 subRun: 1 event: 23951 at 25-Jun-2025 17:07:59 BST
Begin processing the 152nd record. run: 1498 subRun: 1 event: 23952 at 25-Jun-2025 17:08:01 BST
Begin processing the 153rd record. run: 1498 subRun: 1 event: 23953 at 25-Jun-2025 17:08:03 BST
Begin processing the 154th record. run: 1498 subRun: 1 event: 23954 at 25-Jun-2025 17:08:05 BST
Begin processing the 155th record. run: 1498 subRun: 1 event: 23955 at 25-Jun-2025 17:08:07 BST
Begin processing the 156th record. run: 1498 subRun: 1 event: 23956 at 25-Jun-2025 17:08:08 BST
Begin processing the 157th record. run: 1498 subRun: 1 event: 23957 at 25-Jun-2025 17:08:10 BST
Begin processing the 158th record. run: 1498 subRun: 1 event: 23958 at 25-Jun-2025 17:08:12 BST
Begin processing the 159th record. run: 1498 subRun: 1 event: 23959 at 25-Jun-2025 17:08:14 BST
Begin processing the 160th record. run: 1498 subRun: 1 event: 23960 at 25-Jun-2025 17:08:16 BST
Begin processing the 161st record. run: 1498 subRun: 1 event: 23961 at 25-Jun-2025 17:08:17 BST
Begin processing the 162nd record. run: 1498 subRun: 1 event: 23962 at 25-Jun-2025 17:08:19 BST
Begin processing the 163rd record. run: 1498 subRun: 1 event: 23963 at 25-Jun-2025 17:08:21 BST
Begin processing the 164th record. run: 1498 subRun: 1 event: 23964 at 25-Jun-2025 17:08:23 BST
Begin processing the 165th record. run: 1498 subRun: 1 event: 23965 at 25-Jun-2025 17:08:26 BST
Begin processing the 166th record. run: 1498 subRun: 1 event: 23966 at 25-Jun-2025 17:08:27 BST
Begin processing the 167th record. run: 1498 subRun: 1 event: 23967 at 25-Jun-2025 17:08:29 BST
Begin processing the 168th record. run: 1498 subRun: 1 event: 23968 at 25-Jun-2025 17:08:31 BST
Begin processing the 169th record. run: 1498 subRun: 1 event: 23969 at 25-Jun-2025 17:08:31 BST
Begin processing the 170th record. run: 1498 subRun: 1 event: 23970 at 25-Jun-2025 17:08:33 BST
Begin processing the 171st record. run: 1498 subRun: 1 event: 23971 at 25-Jun-2025 17:08:35 BST
Begin processing the 172nd record. run: 1498 subRun: 1 event: 23972 at 25-Jun-2025 17:08:37 BST
Begin processing the 173rd record. run: 1498 subRun: 1 event: 23973 at 25-Jun-2025 17:08:39 BST
Begin processing the 174th record. run: 1498 subRun: 1 event: 23974 at 25-Jun-2025 17:08:41 BST
Begin processing the 175th record. run: 1498 subRun: 1 event: 23975 at 25-Jun-2025 17:08:43 BST
Begin processing the 176th record. run: 1498 subRun: 1 event: 23976 at 25-Jun-2025 17:08:45 BST
Begin processing the 177th record. run: 1498 subRun: 1 event: 23977 at 25-Jun-2025 17:08:47 BST
Begin processing the 178th record. run: 1498 subRun: 1 event: 23978 at 25-Jun-2025 17:08:49 BST
Begin processing the 179th record. run: 1498 subRun: 1 event: 23979 at 25-Jun-2025 17:08:51 BST
Begin processing the 180th record. run: 1498 subRun: 1 event: 23980 at 25-Jun-2025 17:08:53 BST
Begin processing the 181st record. run: 1498 subRun: 1 event: 23981 at 25-Jun-2025 17:08:55 BST
Begin processing the 182nd record. run: 1498 subRun: 1 event: 23982 at 25-Jun-2025 17:08:57 BST
Begin processing the 183rd record. run: 1498 subRun: 1 event: 23983 at 25-Jun-2025 17:08:59 BST
Begin processing the 184th record. run: 1498 subRun: 1 event: 23984 at 25-Jun-2025 17:09:02 BST
Begin processing the 185th record. run: 1498 subRun: 1 event: 23985 at 25-Jun-2025 17:09:04 BST
Begin processing the 186th record. run: 1498 subRun: 1 event: 23986 at 25-Jun-2025 17:09:06 BST
Begin processing the 187th record. run: 1498 subRun: 1 event: 23987 at 25-Jun-2025 17:09:08 BST
Begin processing the 188th record. run: 1498 subRun: 1 event: 23988 at 25-Jun-2025 17:09:10 BST
Begin processing the 189th record. run: 1498 subRun: 1 event: 23989 at 25-Jun-2025 17:09:12 BST
Begin processing the 190th record. run: 1498 subRun: 1 event: 23990 at 25-Jun-2025 17:09:14 BST
Begin processing the 191st record. run: 1498 subRun: 1 event: 23991 at 25-Jun-2025 17:09:16 BST
Begin processing the 192nd record. run: 1498 subRun: 1 event: 23992 at 25-Jun-2025 17:09:18 BST
Begin processing the 193rd record. run: 1498 subRun: 1 event: 23993 at 25-Jun-2025 17:09:20 BST
Begin processing the 194th record. run: 1498 subRun: 1 event: 23994 at 25-Jun-2025 17:09:22 BST
Begin processing the 195th record. run: 1498 subRun: 1 event: 23995 at 25-Jun-2025 17:09:24 BST
Begin processing the 196th record. run: 1498 subRun: 1 event: 23996 at 25-Jun-2025 17:09:25 BST
Begin processing the 197th record. run: 1498 subRun: 1 event: 23997 at 25-Jun-2025 17:09:27 BST
Begin processing the 198th record. run: 1498 subRun: 1 event: 23998 at 25-Jun-2025 17:09:28 BST
Begin processing the 199th record. run: 1498 subRun: 1 event: 23999 at 25-Jun-2025 17:09:30 BST
Begin processing the 200th record. run: 1498 subRun: 1 event: 24000 at 25-Jun-2025 17:09:32 BST
25-Jun-2025 17:09:34 BST  Closed input file "root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/bf/f0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1498_119_20231027T230240Z_gen_g4_detsim_hitreco__20240503T043155Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.506314       1.6692        4.27906       1.67915      0.398188        200    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.0270287     0.0453954      0.11283      0.0533388     0.015465        200    
prod:pandorapid:Chi2ParticleID                 0.157716      0.234484      0.287638      0.240768      0.0313971       200    
prod:energyreconue:EnergyReco                  0.276452      0.601894       1.0672       0.572449       0.12283        200    
prod:energyreconumu:EnergyReco                0.00156498     0.117169       1.1282       0.0932229     0.110205        200    
prod:energyreconc:EnergyReco                   0.0034979     0.0349314     0.331103      0.0397738     0.0355179       200    
[art]:TriggerResults:TriggerResultInserter    1.4127e-05    2.38376e-05   9.0862e-05    2.1816e-05    9.00218e-06      200    
end_path:ana:PIDASelectionAna                  0.0303438      0.63447       2.37597       0.62592       0.21343        200    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19750.8 MB
  Peak resident set size usage (VmHWM): 1379.72 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
.:
total 80
-rw-r--r-- 1 dune01 dune01 28784 Jun 25 17:09 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1498_119_20231027T230240Z_gen_g4_detsim_hitreco__20240503T043155Z_reco2_pida_2025-06-25T_160229Z.root
-rw-r--r-- 1 dune01 dune01 23406 Jun 25 17:09 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1498_119_20231027T230240Z_gen_g4_detsim_hitreco__20240503T043155Z_reco2_reco_2025-06-25T_160229Z.log
-rw-r--r-- 1 dune01 dune01 11733 Jun 25 17:09 jobscript.log
-rw-r--r-- 1 dune01 dune01   183 Jun 25 17:09 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01   134 Jun 25 17:02 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01     0 Jun 25 17:02 debugprod.log
justIN time: 2025-08-14 21:50:47 UTC       justIN version: 01.03.02