justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.

Jobsub ID 421771.177@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID421771.177@justin-prod-sched01.dune.hep.ac.uk
Workflow ID7925
Stage ID1
User namehsouza@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes3145728000 (3000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-06-25 15:54:39
SiteUK_Manchester
EntryUBoone_T2_UK_Manchester_ce01
Last heartbeat2025-06-25 16:33:50
From worker nodeHostnamewn1905250.tier2.hep.manchester.ac.uk
cpuinfoAMD EPYC 7513 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes3145728000 (3000 MiB)
Wall seconds limit257400 (71 hours)
GPU
Inner Apptainer?True
Job statestalled
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-06-25 16:02:40
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74574126_226_20231208T001520Z_gen_g4_detsim_hitreco__20240510T035146Z_reco2.root
Outputting started2025-06-25 16:06:51
Output files
Finished2025-06-25 17:13:54
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

d. run: 74574126 subRun: 1 event: 22630 at 25-Jun-2025 17:04:14 BST
Begin processing the 31st record. run: 74574126 subRun: 1 event: 22631 at 25-Jun-2025 17:04:16 BST
Begin processing the 32nd record. run: 74574126 subRun: 1 event: 22632 at 25-Jun-2025 17:04:18 BST
Begin processing the 33rd record. run: 74574126 subRun: 1 event: 22633 at 25-Jun-2025 17:04:20 BST
Begin processing the 34th record. run: 74574126 subRun: 1 event: 22634 at 25-Jun-2025 17:04:22 BST
Begin processing the 35th record. run: 74574126 subRun: 1 event: 22635 at 25-Jun-2025 17:04:26 BST
Begin processing the 36th record. run: 74574126 subRun: 1 event: 22636 at 25-Jun-2025 17:04:29 BST
Begin processing the 37th record. run: 74574126 subRun: 1 event: 22637 at 25-Jun-2025 17:04:32 BST
Begin processing the 38th record. run: 74574126 subRun: 1 event: 22638 at 25-Jun-2025 17:04:34 BST
Begin processing the 39th record. run: 74574126 subRun: 1 event: 22639 at 25-Jun-2025 17:04:36 BST
Begin processing the 40th record. run: 74574126 subRun: 1 event: 22640 at 25-Jun-2025 17:04:36 BST
Begin processing the 41st record. run: 74574126 subRun: 1 event: 22641 at 25-Jun-2025 17:04:38 BST
Begin processing the 42nd record. run: 74574126 subRun: 1 event: 22642 at 25-Jun-2025 17:04:40 BST
Begin processing the 43rd record. run: 74574126 subRun: 1 event: 22643 at 25-Jun-2025 17:04:42 BST
Begin processing the 44th record. run: 74574126 subRun: 1 event: 22644 at 25-Jun-2025 17:04:45 BST
Begin processing the 45th record. run: 74574126 subRun: 1 event: 22645 at 25-Jun-2025 17:04:48 BST
Begin processing the 46th record. run: 74574126 subRun: 1 event: 22646 at 25-Jun-2025 17:04:50 BST
Begin processing the 47th record. run: 74574126 subRun: 1 event: 22647 at 25-Jun-2025 17:04:52 BST
Begin processing the 48th record. run: 74574126 subRun: 1 event: 22648 at 25-Jun-2025 17:04:54 BST
Begin processing the 49th record. run: 74574126 subRun: 1 event: 22649 at 25-Jun-2025 17:04:56 BST
Begin processing the 50th record. run: 74574126 subRun: 1 event: 22650 at 25-Jun-2025 17:04:58 BST
Begin processing the 51st record. run: 74574126 subRun: 1 event: 22651 at 25-Jun-2025 17:05:00 BST
Begin processing the 52nd record. run: 74574126 subRun: 1 event: 22652 at 25-Jun-2025 17:05:02 BST
Begin processing the 53rd record. run: 74574126 subRun: 1 event: 22653 at 25-Jun-2025 17:05:08 BST
Begin processing the 54th record. run: 74574126 subRun: 1 event: 22654 at 25-Jun-2025 17:05:09 BST
Begin processing the 55th record. run: 74574126 subRun: 1 event: 22655 at 25-Jun-2025 17:05:11 BST
Begin processing the 56th record. run: 74574126 subRun: 1 event: 22656 at 25-Jun-2025 17:05:13 BST
Begin processing the 57th record. run: 74574126 subRun: 1 event: 22657 at 25-Jun-2025 17:05:15 BST
Begin processing the 58th record. run: 74574126 subRun: 1 event: 22658 at 25-Jun-2025 17:05:17 BST
Begin processing the 59th record. run: 74574126 subRun: 1 event: 22659 at 25-Jun-2025 17:05:20 BST
Begin processing the 60th record. run: 74574126 subRun: 1 event: 22660 at 25-Jun-2025 17:05:21 BST
Begin processing the 61st record. run: 74574126 subRun: 1 event: 22661 at 25-Jun-2025 17:05:23 BST
Begin processing the 62nd record. run: 74574126 subRun: 1 event: 22662 at 25-Jun-2025 17:05:26 BST
Begin processing the 63rd record. run: 74574126 subRun: 1 event: 22663 at 25-Jun-2025 17:05:28 BST
Begin processing the 64th record. run: 74574126 subRun: 1 event: 22664 at 25-Jun-2025 17:05:31 BST
Begin processing the 65th record. run: 74574126 subRun: 1 event: 22665 at 25-Jun-2025 17:05:33 BST
Begin processing the 66th record. run: 74574126 subRun: 1 event: 22666 at 25-Jun-2025 17:05:39 BST
Begin processing the 67th record. run: 74574126 subRun: 1 event: 22667 at 25-Jun-2025 17:05:41 BST
Begin processing the 68th record. run: 74574126 subRun: 1 event: 22668 at 25-Jun-2025 17:05:43 BST
Begin processing the 69th record. run: 74574126 subRun: 1 event: 22669 at 25-Jun-2025 17:05:45 BST
Begin processing the 70th record. run: 74574126 subRun: 1 event: 22670 at 25-Jun-2025 17:05:47 BST
Begin processing the 71st record. run: 74574126 subRun: 1 event: 22671 at 25-Jun-2025 17:05:49 BST
Begin processing the 72nd record. run: 74574126 subRun: 1 event: 22672 at 25-Jun-2025 17:05:51 BST
Begin processing the 73rd record. run: 74574126 subRun: 1 event: 22673 at 25-Jun-2025 17:05:53 BST
Begin processing the 74th record. run: 74574126 subRun: 1 event: 22674 at 25-Jun-2025 17:05:54 BST
Begin processing the 75th record. run: 74574126 subRun: 1 event: 22675 at 25-Jun-2025 17:05:56 BST
Begin processing the 76th record. run: 74574126 subRun: 1 event: 22676 at 25-Jun-2025 17:05:58 BST
Begin processing the 77th record. run: 74574126 subRun: 1 event: 22677 at 25-Jun-2025 17:06:00 BST
Begin processing the 78th record. run: 74574126 subRun: 1 event: 22678 at 25-Jun-2025 17:06:02 BST
Begin processing the 79th record. run: 74574126 subRun: 1 event: 22679 at 25-Jun-2025 17:06:05 BST
Begin processing the 80th record. run: 74574126 subRun: 1 event: 22680 at 25-Jun-2025 17:06:07 BST
Begin processing the 81st record. run: 74574126 subRun: 1 event: 22681 at 25-Jun-2025 17:06:09 BST
Begin processing the 82nd record. run: 74574126 subRun: 1 event: 22682 at 25-Jun-2025 17:06:10 BST
Begin processing the 83rd record. run: 74574126 subRun: 1 event: 22683 at 25-Jun-2025 17:06:12 BST
Begin processing the 84th record. run: 74574126 subRun: 1 event: 22684 at 25-Jun-2025 17:06:14 BST
Begin processing the 85th record. run: 74574126 subRun: 1 event: 22685 at 25-Jun-2025 17:06:16 BST
Begin processing the 86th record. run: 74574126 subRun: 1 event: 22686 at 25-Jun-2025 17:06:19 BST
Begin processing the 87th record. run: 74574126 subRun: 1 event: 22687 at 25-Jun-2025 17:06:22 BST
Begin processing the 88th record. run: 74574126 subRun: 1 event: 22688 at 25-Jun-2025 17:06:24 BST
Begin processing the 89th record. run: 74574126 subRun: 1 event: 22689 at 25-Jun-2025 17:06:25 BST
Begin processing the 90th record. run: 74574126 subRun: 1 event: 22690 at 25-Jun-2025 17:06:27 BST
Begin processing the 91st record. run: 74574126 subRun: 1 event: 22691 at 25-Jun-2025 17:06:30 BST
Begin processing the 92nd record. run: 74574126 subRun: 1 event: 22692 at 25-Jun-2025 17:06:32 BST
Begin processing the 93rd record. run: 74574126 subRun: 1 event: 22693 at 25-Jun-2025 17:06:34 BST
Begin processing the 94th record. run: 74574126 subRun: 1 event: 22694 at 25-Jun-2025 17:06:36 BST
Begin processing the 95th record. run: 74574126 subRun: 1 event: 22695 at 25-Jun-2025 17:06:38 BST
Begin processing the 96th record. run: 74574126 subRun: 1 event: 22696 at 25-Jun-2025 17:06:40 BST
Begin processing the 97th record. run: 74574126 subRun: 1 event: 22697 at 25-Jun-2025 17:06:42 BST
Begin processing the 98th record. run: 74574126 subRun: 1 event: 22698 at 25-Jun-2025 17:06:44 BST
Begin processing the 99th record. run: 74574126 subRun: 1 event: 22699 at 25-Jun-2025 17:06:46 BST
Begin processing the 100th record. run: 74574126 subRun: 1 event: 22700 at 25-Jun-2025 17:06:48 BST
25-Jun-2025 17:06:50 BST  Closed input file "root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/ef/e3/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74574126_226_20231208T001520Z_gen_g4_detsim_hitreco__20240510T035146Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.537186       1.77925       3.67424       1.71216      0.400076        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.0264696     0.0507699     0.0846965     0.0541081     0.0171944       100    
prod:pandorapid:Chi2ParticleID                 0.157314      0.243141      0.544669      0.244539      0.0436424       100    
prod:energyreconue:EnergyReco                   0.2726       0.609107      0.982129      0.574386      0.111867        100    
prod:energyreconumu:EnergyReco                0.00161203     0.130859      0.772574      0.152627      0.106525        100    
prod:energyreconc:EnergyReco                   0.0022261     0.0399068     0.230507      0.0395415     0.0356205       100    
[art]:TriggerResults:TriggerResultInserter    1.3325e-05    2.92728e-05   0.000129344   2.4326e-05    1.76875e-05      100    
end_path:ana:PIDASelectionAna                 0.00435592     0.704547       2.34956      0.631928      0.286022        100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19648.8 MB
  Peak resident set size usage (VmHWM): 1257.64 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
.:
total 60
-rw-r--r-- 1 dune01 dune01 20621 Jun 25 17:06 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74574126_226_20231208T001520Z_gen_g4_detsim_hitreco__20240510T035146Z_reco2_pida_2025-06-25T_160246Z.root
-rw-r--r-- 1 dune01 dune01 14218 Jun 25 17:06 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74574126_226_20231208T001520Z_gen_g4_detsim_hitreco__20240510T035146Z_reco2_reco_2025-06-25T_160246Z.log
-rw-r--r-- 1 dune01 dune01 11973 Jun 25 17:06 jobscript.log
-rw-r--r-- 1 dune01 dune01   187 Jun 25 17:06 justin-processed-pfns.txt
-rw-r--r-- 1 dune01 dune01   138 Jun 25 17:02 all-input-dids.txt
-rw-r--r-- 1 dune01 dune01     0 Jun 25 17:02 debugprod.log
justIN time: 2025-08-14 21:52:28 UTC       justIN version: 01.03.02