justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.

Jobsub ID 421777.43@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID421777.43@justin-prod-sched01.dune.hep.ac.uk
Workflow ID7926
Stage ID1
User namehsouza@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes3145728000 (3000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-06-25 15:58:40
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce01
Last heartbeat2025-06-25 16:33:51
From worker nodeHostnamewn1905250.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7513 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes3145728000 (3000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statestalled
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-06-25 16:22:41
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65840069_473_20231122T063334Z_gen_g4_detsim_hitreco__20240507T171006Z_reco2.root
Outputting started2025-06-25 16:32:48
Output files
Finished2025-06-25 17:13:54
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

Begin processing the 131st record. run: 65840069 subRun: 1 event: 94731 at 25-Jun-2025 17:28:49 BST
Begin processing the 132nd record. run: 65840069 subRun: 1 event: 94732 at 25-Jun-2025 17:28:51 BST
Begin processing the 133rd record. run: 65840069 subRun: 1 event: 94733 at 25-Jun-2025 17:28:53 BST
Begin processing the 134th record. run: 65840069 subRun: 1 event: 94734 at 25-Jun-2025 17:28:54 BST
Begin processing the 135th record. run: 65840069 subRun: 1 event: 94735 at 25-Jun-2025 17:28:56 BST
Begin processing the 136th record. run: 65840069 subRun: 1 event: 94736 at 25-Jun-2025 17:28:58 BST
Begin processing the 137th record. run: 65840069 subRun: 1 event: 94737 at 25-Jun-2025 17:29:00 BST
Begin processing the 138th record. run: 65840069 subRun: 1 event: 94738 at 25-Jun-2025 17:29:03 BST
Begin processing the 139th record. run: 65840069 subRun: 1 event: 94739 at 25-Jun-2025 17:29:05 BST
Begin processing the 140th record. run: 65840069 subRun: 1 event: 94740 at 25-Jun-2025 17:29:07 BST
Begin processing the 141st record. run: 65840069 subRun: 1 event: 94741 at 25-Jun-2025 17:29:09 BST
Begin processing the 142nd record. run: 65840069 subRun: 1 event: 94742 at 25-Jun-2025 17:29:11 BST
Begin processing the 143rd record. run: 65840069 subRun: 1 event: 94743 at 25-Jun-2025 17:29:13 BST
Begin processing the 144th record. run: 65840069 subRun: 1 event: 94744 at 25-Jun-2025 17:29:15 BST
Begin processing the 145th record. run: 65840069 subRun: 1 event: 94745 at 25-Jun-2025 17:29:17 BST
Begin processing the 146th record. run: 65840069 subRun: 1 event: 94746 at 25-Jun-2025 17:29:19 BST
Begin processing the 147th record. run: 65840069 subRun: 1 event: 94747 at 25-Jun-2025 17:29:21 BST
Begin processing the 148th record. run: 65840069 subRun: 1 event: 94748 at 25-Jun-2025 17:29:23 BST
Begin processing the 149th record. run: 65840069 subRun: 1 event: 94749 at 25-Jun-2025 17:29:25 BST
Begin processing the 150th record. run: 65840069 subRun: 1 event: 94750 at 25-Jun-2025 17:29:27 BST
Begin processing the 151st record. run: 65840069 subRun: 1 event: 94751 at 25-Jun-2025 17:29:29 BST
Begin processing the 152nd record. run: 65840069 subRun: 1 event: 94752 at 25-Jun-2025 17:29:31 BST
Begin processing the 153rd record. run: 65840069 subRun: 1 event: 94753 at 25-Jun-2025 17:29:34 BST
Begin processing the 154th record. run: 65840069 subRun: 1 event: 94754 at 25-Jun-2025 17:29:36 BST
Begin processing the 155th record. run: 65840069 subRun: 1 event: 94755 at 25-Jun-2025 17:29:38 BST
Begin processing the 156th record. run: 65840069 subRun: 1 event: 94756 at 25-Jun-2025 17:29:40 BST
Begin processing the 157th record. run: 65840069 subRun: 1 event: 94757 at 25-Jun-2025 17:29:42 BST
Begin processing the 158th record. run: 65840069 subRun: 1 event: 94758 at 25-Jun-2025 17:29:44 BST
Begin processing the 159th record. run: 65840069 subRun: 1 event: 94759 at 25-Jun-2025 17:29:46 BST
Begin processing the 160th record. run: 65840069 subRun: 1 event: 94760 at 25-Jun-2025 17:29:48 BST
Begin processing the 161st record. run: 65840069 subRun: 1 event: 94761 at 25-Jun-2025 17:29:49 BST
Begin processing the 162nd record. run: 65840069 subRun: 1 event: 94762 at 25-Jun-2025 17:29:49 BST
Begin processing the 163rd record. run: 65840069 subRun: 1 event: 94763 at 25-Jun-2025 17:29:52 BST
Begin processing the 164th record. run: 65840069 subRun: 1 event: 94764 at 25-Jun-2025 17:29:54 BST
Begin processing the 165th record. run: 65840069 subRun: 1 event: 94765 at 25-Jun-2025 17:29:56 BST
Begin processing the 166th record. run: 65840069 subRun: 1 event: 94766 at 25-Jun-2025 17:29:58 BST
Begin processing the 167th record. run: 65840069 subRun: 1 event: 94767 at 25-Jun-2025 17:30:00 BST
Begin processing the 168th record. run: 65840069 subRun: 1 event: 94768 at 25-Jun-2025 17:30:02 BST
Begin processing the 169th record. run: 65840069 subRun: 1 event: 94769 at 25-Jun-2025 17:30:04 BST
Begin processing the 170th record. run: 65840069 subRun: 1 event: 94770 at 25-Jun-2025 17:30:06 BST
Begin processing the 171st record. run: 65840069 subRun: 1 event: 94771 at 25-Jun-2025 17:30:07 BST
Begin processing the 172nd record. run: 65840069 subRun: 1 event: 94772 at 25-Jun-2025 17:30:10 BST
Begin processing the 173rd record. run: 65840069 subRun: 1 event: 94773 at 25-Jun-2025 17:30:11 BST
Begin processing the 174th record. run: 65840069 subRun: 1 event: 94774 at 25-Jun-2025 17:30:14 BST
Begin processing the 175th record. run: 65840069 subRun: 1 event: 94775 at 25-Jun-2025 17:30:16 BST
Begin processing the 176th record. run: 65840069 subRun: 1 event: 94776 at 25-Jun-2025 17:30:20 BST
Begin processing the 177th record. run: 65840069 subRun: 1 event: 94777 at 25-Jun-2025 17:30:22 BST
Begin processing the 178th record. run: 65840069 subRun: 1 event: 94778 at 25-Jun-2025 17:30:24 BST
Begin processing the 179th record. run: 65840069 subRun: 1 event: 94779 at 25-Jun-2025 17:30:26 BST
Begin processing the 180th record. run: 65840069 subRun: 1 event: 94780 at 25-Jun-2025 17:30:28 BST
Begin processing the 181st record. run: 65840069 subRun: 1 event: 94781 at 25-Jun-2025 17:30:29 BST
Begin processing the 182nd record. run: 65840069 subRun: 1 event: 94782 at 25-Jun-2025 17:30:31 BST
Begin processing the 183rd record. run: 65840069 subRun: 1 event: 94783 at 25-Jun-2025 17:30:33 BST
Begin processing the 184th record. run: 65840069 subRun: 1 event: 94784 at 25-Jun-2025 17:30:36 BST
Begin processing the 185th record. run: 65840069 subRun: 1 event: 94785 at 25-Jun-2025 17:30:37 BST
Begin processing the 186th record. run: 65840069 subRun: 1 event: 94786 at 25-Jun-2025 17:30:40 BST
Begin processing the 187th record. run: 65840069 subRun: 1 event: 94787 at 25-Jun-2025 17:30:42 BST
Begin processing the 188th record. run: 65840069 subRun: 1 event: 94788 at 25-Jun-2025 17:30:44 BST
Begin processing the 189th record. run: 65840069 subRun: 1 event: 94789 at 25-Jun-2025 17:30:46 BST
Begin processing the 190th record. run: 65840069 subRun: 1 event: 94790 at 25-Jun-2025 17:30:47 BST
Begin processing the 191st record. run: 65840069 subRun: 1 event: 94791 at 25-Jun-2025 17:30:49 BST
Begin processing the 192nd record. run: 65840069 subRun: 1 event: 94792 at 25-Jun-2025 17:30:51 BST
Begin processing the 193rd record. run: 65840069 subRun: 1 event: 94793 at 25-Jun-2025 17:30:53 BST
Begin processing the 194th record. run: 65840069 subRun: 1 event: 94794 at 25-Jun-2025 17:30:55 BST
Begin processing the 195th record. run: 65840069 subRun: 1 event: 94795 at 25-Jun-2025 17:30:59 BST
Begin processing the 196th record. run: 65840069 subRun: 1 event: 94796 at 25-Jun-2025 17:31:01 BST
Begin processing the 197th record. run: 65840069 subRun: 1 event: 94797 at 25-Jun-2025 17:31:01 BST
Begin processing the 198th record. run: 65840069 subRun: 1 event: 94798 at 25-Jun-2025 17:31:04 BST
Begin processing the 199th record. run: 65840069 subRun: 1 event: 94799 at 25-Jun-2025 17:31:06 BST
Begin processing the 200th record. run: 65840069 subRun: 1 event: 94800 at 25-Jun-2025 17:31:08 BST
25-Jun-2025 17:31:10 BST  Closed input file "root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/27/13/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65840069_473_20231122T063334Z_gen_g4_detsim_hitreco__20240507T171006Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.486772       1.7426         5.473        1.73115      0.464641        200    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.0276895     0.0563131     0.0960712     0.0551446     0.0159813       200    
prod:pandorapid:Chi2ParticleID                 0.158604      0.248411      0.505295      0.256172      0.0399237       200    
prod:energyreconue:EnergyReco                  0.278746      0.602881       1.14379      0.574693      0.122465        200    
prod:energyreconumu:EnergyReco                0.00186854     0.138876       1.80437      0.144036      0.170842        200    
prod:energyreconc:EnergyReco                  0.00329036     0.0392769     0.494061      0.0421565     0.042853        200    
[art]:TriggerResults:TriggerResultInserter    1.5569e-05    3.93499e-05   0.000192222   3.6434e-05    1.74391e-05      200    
end_path:ana:PIDASelectionAna                 0.00423459     0.655836       1.6677       0.631361      0.210786        200    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19436 MB
  Peak resident set size usage (VmHWM): 1063.35 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
.:
total 80
-rw-r--r-- 1 dune01 dune01 28727 Jun 25 17:31 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65840069_473_20231122T063334Z_gen_g4_detsim_hitreco__20240507T171006Z_reco2_pida_2025-06-25T_162249Z.root
-rw-r--r-- 1 dune01 dune01 24216 Jun 25 17:31 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65840069_473_20231122T063334Z_gen_g4_detsim_hitreco__20240507T171006Z_reco2_reco_2025-06-25T_162249Z.log
-rw-r--r-- 1 dune01 dune01 12047 Jun 25 17:31 jobscript.log
-rw-r--r-- 1 dune01 dune01   187 Jun 25 17:31 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01   138 Jun 25 17:22 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01     0 Jun 25 17:22 debugprod.log
justIN time: 2025-08-14 21:49:25 UTC       justIN version: 01.03.02