justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.

Jobsub ID 418767.109@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID418767.109@justin-prod-sched01.dune.hep.ac.uk
Workflow ID7800
Stage ID1
User namehsouza@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
GPUNo
RSS bytes3145728000 (3000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2025-06-20 12:48:00
SiteCH_UNIBE-LHEP
EntryOSG_CH_UNIBE_LHEP_ce02
Last heartbeat2025-06-20 12:56:53
From worker nodeHostnamewn-1-35.local
cpuinfoIntel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes3145728000 (3000 MiB)
Wall seconds limit86400 (24 hours)
GPU
Inner Apptainer?True
Job statestalled
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2025-06-20 12:51:20
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6396715_759_20231202T065324Z_gen_g4_detsim_hitreco__20240507T202722Z_reco2.root
Outputting started2025-06-20 12:56:53
Output files
Finished2025-06-20 14:03:24
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

th record. run: 6396715 subRun: 1 event: 75930 at 20-Jun-2025 14:53:25 CEST
Begin processing the 31st record. run: 6396715 subRun: 1 event: 75931 at 20-Jun-2025 14:53:42 CEST
Begin processing the 32nd record. run: 6396715 subRun: 1 event: 75932 at 20-Jun-2025 14:53:45 CEST
Begin processing the 33rd record. run: 6396715 subRun: 1 event: 75933 at 20-Jun-2025 14:53:46 CEST
Begin processing the 34th record. run: 6396715 subRun: 1 event: 75934 at 20-Jun-2025 14:53:47 CEST
Begin processing the 35th record. run: 6396715 subRun: 1 event: 75935 at 20-Jun-2025 14:53:50 CEST
Begin processing the 36th record. run: 6396715 subRun: 1 event: 75936 at 20-Jun-2025 14:53:51 CEST
Begin processing the 37th record. run: 6396715 subRun: 1 event: 75937 at 20-Jun-2025 14:53:52 CEST
Begin processing the 38th record. run: 6396715 subRun: 1 event: 75938 at 20-Jun-2025 14:53:59 CEST
Begin processing the 39th record. run: 6396715 subRun: 1 event: 75939 at 20-Jun-2025 14:54:01 CEST
Begin processing the 40th record. run: 6396715 subRun: 1 event: 75940 at 20-Jun-2025 14:54:03 CEST
Begin processing the 41st record. run: 6396715 subRun: 1 event: 75941 at 20-Jun-2025 14:54:07 CEST
Begin processing the 42nd record. run: 6396715 subRun: 1 event: 75942 at 20-Jun-2025 14:54:09 CEST
Begin processing the 43rd record. run: 6396715 subRun: 1 event: 75943 at 20-Jun-2025 14:54:10 CEST
Begin processing the 44th record. run: 6396715 subRun: 1 event: 75944 at 20-Jun-2025 14:54:11 CEST
Begin processing the 45th record. run: 6396715 subRun: 1 event: 75945 at 20-Jun-2025 14:54:13 CEST
Begin processing the 46th record. run: 6396715 subRun: 1 event: 75946 at 20-Jun-2025 14:54:13 CEST
Begin processing the 47th record. run: 6396715 subRun: 1 event: 75947 at 20-Jun-2025 14:54:16 CEST
Begin processing the 48th record. run: 6396715 subRun: 1 event: 75948 at 20-Jun-2025 14:54:19 CEST
Begin processing the 49th record. run: 6396715 subRun: 1 event: 75949 at 20-Jun-2025 14:54:20 CEST
Begin processing the 50th record. run: 6396715 subRun: 1 event: 75950 at 20-Jun-2025 14:54:20 CEST
Begin processing the 51st record. run: 6396715 subRun: 1 event: 75951 at 20-Jun-2025 14:54:21 CEST
Begin processing the 52nd record. run: 6396715 subRun: 1 event: 75952 at 20-Jun-2025 14:54:24 CEST
Begin processing the 53rd record. run: 6396715 subRun: 1 event: 75953 at 20-Jun-2025 14:54:41 CEST
Begin processing the 54th record. run: 6396715 subRun: 1 event: 75954 at 20-Jun-2025 14:54:43 CEST
Begin processing the 55th record. run: 6396715 subRun: 1 event: 75955 at 20-Jun-2025 14:54:45 CEST
Begin processing the 56th record. run: 6396715 subRun: 1 event: 75956 at 20-Jun-2025 14:54:47 CEST
Begin processing the 57th record. run: 6396715 subRun: 1 event: 75957 at 20-Jun-2025 14:54:48 CEST
Begin processing the 58th record. run: 6396715 subRun: 1 event: 75958 at 20-Jun-2025 14:54:50 CEST
Begin processing the 59th record. run: 6396715 subRun: 1 event: 75959 at 20-Jun-2025 14:55:22 CEST
Begin processing the 60th record. run: 6396715 subRun: 1 event: 75960 at 20-Jun-2025 14:55:24 CEST
Begin processing the 61st record. run: 6396715 subRun: 1 event: 75961 at 20-Jun-2025 14:55:28 CEST
Begin processing the 62nd record. run: 6396715 subRun: 1 event: 75962 at 20-Jun-2025 14:55:28 CEST
Begin processing the 63rd record. run: 6396715 subRun: 1 event: 75963 at 20-Jun-2025 14:55:30 CEST
Begin processing the 64th record. run: 6396715 subRun: 1 event: 75964 at 20-Jun-2025 14:55:31 CEST
Begin processing the 65th record. run: 6396715 subRun: 1 event: 75965 at 20-Jun-2025 14:55:34 CEST
Begin processing the 66th record. run: 6396715 subRun: 1 event: 75966 at 20-Jun-2025 14:55:35 CEST
Begin processing the 67th record. run: 6396715 subRun: 1 event: 75967 at 20-Jun-2025 14:55:36 CEST
Begin processing the 68th record. run: 6396715 subRun: 1 event: 75968 at 20-Jun-2025 14:55:38 CEST
Begin processing the 69th record. run: 6396715 subRun: 1 event: 75969 at 20-Jun-2025 14:55:39 CEST
Begin processing the 70th record. run: 6396715 subRun: 1 event: 75970 at 20-Jun-2025 14:55:41 CEST
Begin processing the 71st record. run: 6396715 subRun: 1 event: 75971 at 20-Jun-2025 14:55:42 CEST
Begin processing the 72nd record. run: 6396715 subRun: 1 event: 75972 at 20-Jun-2025 14:55:43 CEST
Begin processing the 73rd record. run: 6396715 subRun: 1 event: 75973 at 20-Jun-2025 14:55:44 CEST
Begin processing the 74th record. run: 6396715 subRun: 1 event: 75974 at 20-Jun-2025 14:55:45 CEST
Begin processing the 75th record. run: 6396715 subRun: 1 event: 75975 at 20-Jun-2025 14:55:49 CEST
Begin processing the 76th record. run: 6396715 subRun: 1 event: 75976 at 20-Jun-2025 14:55:51 CEST
Begin processing the 77th record. run: 6396715 subRun: 1 event: 75977 at 20-Jun-2025 14:55:52 CEST
Begin processing the 78th record. run: 6396715 subRun: 1 event: 75978 at 20-Jun-2025 14:55:54 CEST
Begin processing the 79th record. run: 6396715 subRun: 1 event: 75979 at 20-Jun-2025 14:55:55 CEST
Begin processing the 80th record. run: 6396715 subRun: 1 event: 75980 at 20-Jun-2025 14:55:56 CEST
Begin processing the 81st record. run: 6396715 subRun: 1 event: 75981 at 20-Jun-2025 14:55:58 CEST
Begin processing the 82nd record. run: 6396715 subRun: 1 event: 75982 at 20-Jun-2025 14:55:59 CEST
Begin processing the 83rd record. run: 6396715 subRun: 1 event: 75983 at 20-Jun-2025 14:56:00 CEST
Begin processing the 84th record. run: 6396715 subRun: 1 event: 75984 at 20-Jun-2025 14:56:01 CEST
Begin processing the 85th record. run: 6396715 subRun: 1 event: 75985 at 20-Jun-2025 14:56:02 CEST
Begin processing the 86th record. run: 6396715 subRun: 1 event: 75986 at 20-Jun-2025 14:56:03 CEST
Begin processing the 87th record. run: 6396715 subRun: 1 event: 75987 at 20-Jun-2025 14:56:06 CEST
Begin processing the 88th record. run: 6396715 subRun: 1 event: 75988 at 20-Jun-2025 14:56:08 CEST
Begin processing the 89th record. run: 6396715 subRun: 1 event: 75989 at 20-Jun-2025 14:56:18 CEST
Begin processing the 90th record. run: 6396715 subRun: 1 event: 75990 at 20-Jun-2025 14:56:19 CEST
Begin processing the 91st record. run: 6396715 subRun: 1 event: 75991 at 20-Jun-2025 14:56:19 CEST
Begin processing the 92nd record. run: 6396715 subRun: 1 event: 75992 at 20-Jun-2025 14:56:21 CEST
Begin processing the 93rd record. run: 6396715 subRun: 1 event: 75993 at 20-Jun-2025 14:56:32 CEST
Begin processing the 94th record. run: 6396715 subRun: 1 event: 75994 at 20-Jun-2025 14:56:34 CEST
Begin processing the 95th record. run: 6396715 subRun: 1 event: 75995 at 20-Jun-2025 14:56:35 CEST
Begin processing the 96th record. run: 6396715 subRun: 1 event: 75996 at 20-Jun-2025 14:56:37 CEST
Begin processing the 97th record. run: 6396715 subRun: 1 event: 75997 at 20-Jun-2025 14:56:41 CEST
Begin processing the 98th record. run: 6396715 subRun: 1 event: 75998 at 20-Jun-2025 14:56:46 CEST
Begin processing the 99th record. run: 6396715 subRun: 1 event: 75999 at 20-Jun-2025 14:56:47 CEST
Begin processing the 100th record. run: 6396715 subRun: 1 event: 76000 at 20-Jun-2025 14:56:48 CEST
20-Jun-2025 14:56:52 CEST  Closed input file "root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/1a/ff/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6396715_759_20231202T065324Z_gen_g4_detsim_hitreco__20240507T202722Z_reco2.root"

================================================================================================================================
TimeTracker printout (sec)                        Min           Avg           Max         Median          RMS         nEvts   
================================================================================================================================
Full event                                     0.398796       1.76855       15.8594       1.17122       2.29825        100    
--------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                         0.0181372     0.0344109     0.0952616     0.0361619     0.0146851       100    
prod:pandorapid:Chi2ParticleID                 0.106856      0.194639      0.616478      0.171917      0.0841455       100    
prod:energyreconue:EnergyReco                  0.218615      0.500038       1.37781      0.445723      0.187931        100    
prod:energyreconumu:EnergyReco                0.00275626     0.141067       1.81752      0.104934       0.2146         100    
prod:energyreconc:EnergyReco                   0.0040306     0.114597      0.797133      0.0694446     0.157129        100    
[art]:TriggerResults:TriggerResultInserter    1.2159e-05    2.08442e-05   6.3974e-05    2.0056e-05    6.58221e-06      100    
end_path:ana:PIDASelectionAna                 0.00376159     0.783286       13.2872       0.2379        1.94107        100    
================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 7633.18 MB
  Peak resident set size usage (VmHWM): 891.539 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
.:
total 49
-rw-r--r-- 1 dune002 dune 24674 Jun 20 14:56 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6396715_759_20231202T065324Z_gen_g4_detsim_hitreco__20240507T202722Z_reco2_pida_2025-06-20T_125125Z.root
-rw-r--r-- 1 dune002 dune 14218 Jun 20 14:56 atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6396715_759_20231202T065324Z_gen_g4_detsim_hitreco__20240507T202722Z_reco2_reco_2025-06-20T_125125Z.log
-rw-r--r-- 1 dune002 dune 11972 Jun 20 14:56 jobscript.log
-rw-r--r-- 1 dune002 dune   186 Jun 20 14:56 justin-processed-pfns.txt
-rw-r--r-- 1 dune002 dune   137 Jun 20 14:51 all-input-dids.txt
-rw-r--r-- 1 dune002 dune     0 Jun 20 14:51 debugprod.log
justIN time: 2025-08-14 19:32:05 UTC       justIN version: 01.03.02