21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 228270.18@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 228270.18@justin-prod-sched02.dune.hep.ac.uk | |
| Workflow ID | 7875 | |
| Stage ID | 1 | |
| User name | higuera@fnal.gov | |
| HTCondor Group | group_dune | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-06-24 08:08:48 | |
| Site | UK_Manchester | |
| Entry | UBoone_T2_UK_Manchester_ce02 | |
| Last heartbeat | 2025-06-24 19:24:39 | |
| From worker node | Hostname | wn2209250.tier2.hep.manchester.ac.uk |
| cpuinfo | AMD EPYC 7513 32-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 257400 (71 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | stalled | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-06-24 09:42:33 | |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505925_251_20231202T162314Z_gen_g4_detsim_hitreco.root | |
| Outputting started | 2025-06-24 09:54:40 | |
| Output files | ||
| Finished | 2025-06-24 20:26:26 | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
e26_prof/' is not a directory WARNING: '/_reco/localProducts_larsoft_v09_89_01_e26_prof/' is not a directory WARNING: '/_reco/localProducts_larsoft_v09_89_01_e26_prof/' is not a directory WARNING: '/_reco/localProducts_larsoft_v09_89_01_e26_prof/' is not a directory WARNING: '/_reco/localProducts_larsoft_v09_89_01_e26_prof/' is not a directory WARNING: '/_reco/localProducts_larsoft_v09_89_01_e26_prof/' is not a directory WARNING: '/_reco/localProducts_larsoft_v09_89_01_e26_prof/' is not a directory WARNING: '/_reco/localProducts_larsoft_v09_89_01_e26_prof/' is not a directory WARNING: '/_reco/localProducts_larsoft_v09_89_01_e26_prof/' is not a directory WARNING: '/_reco/localProducts_larsoft_v09_89_01_e26_prof/' is not a directory WARNING: '/_reco/localProducts_larsoft_v09_89_01_e26_prof/' is not a directory WARNING: '/_reco/localProducts_larsoft_v09_89_01_e26_prof/' is not a directory WARNING: '/_reco/localProducts_larsoft_v09_89_01_e26_prof/' is not a directory WARNING: '/_reco/localProducts_larsoft_v09_89_01_e26_prof/' is not a directory WARNING: '/_reco/localProducts_larsoft_v09_89_01_e26_prof/' is not a directory WARNING: '/_reco/localProducts_larsoft_v09_89_01_e26_prof/' is not a directory WARNING: '/_reco/localProducts_larsoft_v09_89_01_e26_prof/' is not a directory WARNING: '/_reco/localProducts_larsoft_v09_89_01_e26_prof/' is not a directory WARNING: '/_reco/localProducts_larsoft_v09_89_01_e26_prof/' is not a directory WARNING: '/_reco/localProducts_larsoft_v09_89_01_e26_prof/' is not a directory Justin processors: 1 /cvmfs/fifeuser2.opensciencegrid.org/sw/dune/cdba6e7604dd8b049ed4282aee20b939a4b77fad/atmo_reco2_lbl_model.fcl Input PFN = root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/45/48/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505925_251_20231202T162314Z_gen_g4_detsim_hitreco.root /cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so === Start last 100 lines of lar log file === Output 1: 0.988513, 0.0110245, 0.000393155, 6.96427e-05, Output 2: 0.852524, 0.145135, 0.00222384, 0.000117864, %MSG-e DUNEAna: EnergyReco:energyrecnumurange@BeginModule 24-Jun-2025 10:54:05 BST run: 74505925 subRun: 1 event: 25198 Failed to find product with label wclsmcnfsp:gauss ... returning empty vector %MSG %MSG-e DUNEAna: EnergyReco:energyrecnumumcs@BeginModule 24-Jun-2025 10:54:05 BST run: 74505925 subRun: 1 event: 25198 Failed to find product with label wclsmcnfsp:gauss ... returning empty vector %MSG 358 ***Wiener Filter*******ApplyPostfilter*** Running Ophitfinder with InputDigiType = 'recob' Found hits: 1054! Begin processing the 99th record. run: 74505925 subRun: 1 event: 25199 at 24-Jun-2025 10:54:06 BST %MSG-e ShowerProducedPtrsHolder: LArPandoraModularShowerCreation:pandoraShower@BeginModule 24-Jun-2025 10:54:08 BST run: 74505925 subRun: 1 event: 25199 Trying to add data product: InitialTrack. This element does not exist in the element holder %MSG Boundary wire vector sizes: 108, 123, 110 minwire 0: 2171 minwire 1: 654 minwire 2: 2175 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Classifier summary: Output 0: 0.000342645, 0.997343, 0.00231402, Output 1: 0.866414, 0.132824, 0.000715246, 4.6334e-05, Output 2: 0.998796, 0.00117839, 2.41838e-05, 1.67666e-06, %MSG-e DUNEAna: EnergyReco:energyrecnumumcs@BeginModule 24-Jun-2025 10:54:08 BST run: 74505925 subRun: 1 event: 25199 Failed to find product with label wclsmcnfsp:gauss ... returning empty vector %MSG 449 ***Wiener Filter*******ApplyPostfilter*** Running Ophitfinder with InputDigiType = 'recob' Found hits: 1499! Begin processing the 100th record. run: 74505925 subRun: 1 event: 25200 at 24-Jun-2025 10:54:10 BST %MSG-e ShowerProducedPtrsHolder: LArPandoraModularShowerCreation:pandoraShower@BeginModule 24-Jun-2025 10:54:11 BST run: 74505925 subRun: 1 event: 25200 Trying to add data product: InitialTrack. This element does not exist in the element holder %MSG Boundary wire vector sizes: 97, 43, 66 minwire 0: 1603 minwire 1: 1358 minwire 2: 1506 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Classifier summary: Output 0: 0.981516, 0.000617259, 0.017867, Output 1: 0.138652, 0.843916, 0.0170504, 0.000381165, Output 2: 0.98813, 0.0116349, 0.000229453, 5.33447e-06, %MSG-e DUNEAna: EnergyReco:energyrecnumumcs@BeginModule 24-Jun-2025 10:54:12 BST run: 74505925 subRun: 1 event: 25200 Failed to find product with label wclsmcnfsp:gauss ... returning empty vector %MSG 493 ***Wiener Filter*******ApplyPostfilter*** Running Ophitfinder with InputDigiType = 'recob' Found hits: 1573! 24-Jun-2025 10:54:14 BST Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505925_251_20231202T162314Z_gen_g4_detsim_hitreco_reco2.root" 24-Jun-2025 10:54:14 BST Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/45/48/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74505925_251_20231202T162314Z_gen_g4_detsim_hitreco.root" ======================================================================================================================================== TimeTracker printout (sec) Min Avg Max Median RMS nEvts ======================================================================================================================================== Full event 0.638163 4.12076 22.4726 3.60598 2.73373 100 ---------------------------------------------------------------------------------------------------------------------------------------- source:RootInput(read) 0.0222541 0.027579 0.0615126 0.0233494 0.00913714 100 reco:pandora:StandardPandora 0.140494 1.5255 16.956 1.02103 1.89899 100 reco:pandoraTrack:LArPandoraTrackCreation 0.000110569 0.00465159 0.0466756 0.00145793 0.00917183 100 reco:pandoraShower:LArPandoraModularShowerCreation 0.000152939 0.00976013 0.151614 0.00283892 0.022241 100 reco:pandoracalo:Calorimetry 8.3227e-05 0.0114608 0.235195 0.00106673 0.0323715 100 reco:pandorapid:Chi2ParticleID 3.0097e-05 0.000143426 0.00095305 0.000102638 0.000126226 100 reco:cvnmap:CVNMapper 2.0569e-05 0.00889049 0.0525047 0.00810925 0.00807258 100 reco:cvneva:CVNEvaluator 1.3035e-05 0.252547 1.51311 0.303503 0.183278 100 reco:energyrecnumu:EnergyReco 0.0665307 0.147563 1.5246 0.109542 0.169002 100 reco:energyrecnue:EnergyReco 0.000163038 0.0127655 0.0825942 0.000787383 0.024833 100 reco:energyrecnc:EnergyReco 0.00139086 0.100179 0.232019 0.118128 0.0519544 100 reco:energyrecnumurange:EnergyReco 0.000158299 0.0334109 1.10231 0.00114418 0.125719 100 reco:energyrecnumumcs:EnergyReco 9.024e-05 0.031866 1.08952 0.00095572 0.122718 100 reco:opdec:Deconvolution 0.0810619 0.21486 0.399251 0.215804 0.0548907 100 reco:ophitspe:OpHitFinderDeco 0.303529 1.54417 2.76528 1.57038 0.565344 100 reco:opflash:OpFlashFinder 0.000323301 0.00125385 0.00321623 0.00113265 0.000571986 100 reco:rns:RandomNumberSaver 1.8144e-05 6.6501e-05 0.000882387 3.32785e-05 0.000128821 100 reco:anglereconue:NuAngularReco 0.000146467 0.00118213 0.0109904 0.000357576 0.00210771 100 reco:anglereconumu:NuAngularReco 6.2127e-05 0.000905303 0.0108049 0.000202257 0.00198401 100 reco:anglereconuepfps:NuAngularReco 0.000101782 0.00104717 0.0109193 0.000329668 0.00199804 100 reco:anglereconumupfps:NuAngularReco 7.7666e-05 0.000951577 0.0109874 0.000242454 0.00202191 100 [art]:TriggerResults:TriggerResultInserter 7.905e-06 1.2322e-05 4.4374e-05 1.1051e-05 5.05592e-06 100 end_path:out1:RootOutput 2.485e-06 5.67425e-06 2.3825e-05 4.849e-06 3.22449e-06 100 end_path:out1:RootOutput(write) 0.0106711 0.188803 0.358676 0.207567 0.09151 100 ======================================================================================================================================== ==================================================================================================== MemoryTracker summary (base-10 MB units used) Peak virtual memory usage (VmPeak) : 3580.08 MB Peak resident set size usage (VmHWM): 1817.48 MB ==================================================================================================== Art has completed and will exit with status 0. === End last 100 lines of lar log file === lar exit code 0