21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.
Jobsub ID 228097.88@justin-prod-sched02.dune.hep.ac.uk
| Jobsub ID | 228097.88@justin-prod-sched02.dune.hep.ac.uk | |
| Workflow ID | 7875 | |
| Stage ID | 1 | |
| User name | higuera@fnal.gov | |
| HTCondor Group | group_dune | |
| Requested | Processors | 1 |
| GPU | No | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 80000 (22 hours) | |
| Submitted time | 2025-06-23 22:38:34 | |
| Site | UK_Manchester | |
| Entry | UBoone_T2_UK_Manchester_ce02 | |
| Last heartbeat | 2025-06-24 19:25:27 | |
| From worker node | Hostname | wn1904240.tier2.hep.manchester.ac.uk |
| cpuinfo | AMD EPYC 7513 32-Core Processor | |
| OS release | Scientific Linux release 7.9 (Nitrogen) | |
| Processors | 1 | |
| RSS bytes | 4194304000 (4000 MiB) | |
| Wall seconds limit | 257400 (71 hours) | |
| GPU | ||
| Inner Apptainer? | True | |
| Job state | stalled | |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk | |
| Started | 2025-06-24 09:39:30 | |
| Input files | fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65996932_922_20231202T073525Z_gen_g4_detsim_hitreco.root | |
| Outputting started | 2025-06-24 09:56:36 | |
| Output files | ||
| Finished | 2025-06-24 20:27:42 | |
| List job events Wrapper job log | ||
Jobscript log (last 10,000 characters)
e26_prof/' is not a directory WARNING: '/_reco/localProducts_larsoft_v09_89_01_e26_prof/' is not a directory WARNING: '/_reco/localProducts_larsoft_v09_89_01_e26_prof/' is not a directory WARNING: '/_reco/localProducts_larsoft_v09_89_01_e26_prof/' is not a directory WARNING: '/_reco/localProducts_larsoft_v09_89_01_e26_prof/' is not a directory WARNING: '/_reco/localProducts_larsoft_v09_89_01_e26_prof/' is not a directory WARNING: '/_reco/localProducts_larsoft_v09_89_01_e26_prof/' is not a directory WARNING: '/_reco/localProducts_larsoft_v09_89_01_e26_prof/' is not a directory WARNING: '/_reco/localProducts_larsoft_v09_89_01_e26_prof/' is not a directory WARNING: '/_reco/localProducts_larsoft_v09_89_01_e26_prof/' is not a directory WARNING: '/_reco/localProducts_larsoft_v09_89_01_e26_prof/' is not a directory WARNING: '/_reco/localProducts_larsoft_v09_89_01_e26_prof/' is not a directory WARNING: '/_reco/localProducts_larsoft_v09_89_01_e26_prof/' is not a directory WARNING: '/_reco/localProducts_larsoft_v09_89_01_e26_prof/' is not a directory WARNING: '/_reco/localProducts_larsoft_v09_89_01_e26_prof/' is not a directory WARNING: '/_reco/localProducts_larsoft_v09_89_01_e26_prof/' is not a directory WARNING: '/_reco/localProducts_larsoft_v09_89_01_e26_prof/' is not a directory WARNING: '/_reco/localProducts_larsoft_v09_89_01_e26_prof/' is not a directory WARNING: '/_reco/localProducts_larsoft_v09_89_01_e26_prof/' is not a directory WARNING: '/_reco/localProducts_larsoft_v09_89_01_e26_prof/' is not a directory Justin processors: 1 /cvmfs/fifeuser2.opensciencegrid.org/sw/dune/cdba6e7604dd8b049ed4282aee20b939a4b77fad/atmo_reco2_lbl_model.fcl Input PFN = root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/1f/5d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65996932_922_20231202T073525Z_gen_g4_detsim_hitreco.root /cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so === Start last 100 lines of lar log file === Running Ophitfinder with InputDigiType = 'recob' Found hits: 1088! Begin processing the 98th record. run: 65996932 subRun: 1 event: 92298 at 24-Jun-2025 10:56:23 BST %MSG-e ShowerProducedPtrsHolder: LArPandoraModularShowerCreation:pandoraShower@BeginModule 24-Jun-2025 10:56:25 BST run: 65996932 subRun: 1 event: 92298 Trying to add data product: InitialTrack. This element does not exist in the element holder %MSG Boundary wire vector sizes: 220, 226, 195 minwire 0: 1866 minwire 1: 814 minwire 2: 1774 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Classifier summary: Output 0: 0.0862943, 0.0193326, 0.894373, Output 1: 0.0141009, 0.0447988, 0.206566, 0.734534, Output 2: 0.780043, 0.205977, 0.0129464, 0.0010334, %MSG-e DUNEAna: EnergyReco:energyrecnumumcs@BeginModule 24-Jun-2025 10:56:25 BST run: 65996932 subRun: 1 event: 92298 Failed to find product with label wclsmcnfsp:gauss ... returning empty vector %MSG 541 ***Wiener Filter*******ApplyPostfilter*** Running Ophitfinder with InputDigiType = 'recob' Found hits: 1953! Begin processing the 99th record. run: 65996932 subRun: 1 event: 92299 at 24-Jun-2025 10:56:28 BST Boundary wire vector sizes: 243, 260, 195 minwire 0: 1156 minwire 1: 1024 minwire 2: 1365 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Used alternate method to get min and max tdcs due to vertex determination failure: 0, 299 Classifier summary: Output 0: 0.000327873, 0.997796, 0.00187625, Output 1: 0.926797, 0.0727839, 0.000362845, 5.65636e-05, Output 2: 0.999235, 0.000743627, 1.85907e-05, 2.4558e-06, %MSG-e DUNEAna: EnergyReco:energyrecnumurange@BeginModule 24-Jun-2025 10:56:30 BST run: 65996932 subRun: 1 event: 92299 Failed to find product with label wclsmcnfsp:gauss ... returning empty vector %MSG %MSG-e DUNEAna: EnergyReco:energyrecnumumcs@BeginModule 24-Jun-2025 10:56:30 BST run: 65996932 subRun: 1 event: 92299 Failed to find product with label wclsmcnfsp:gauss ... returning empty vector %MSG 575 ***Wiener Filter*******ApplyPostfilter*** Running Ophitfinder with InputDigiType = 'recob' Found hits: 1949! Begin processing the 100th record. run: 65996932 subRun: 1 event: 92300 at 24-Jun-2025 10:56:32 BST %MSG-e DUNEAna: EnergyReco:energyrecnumurange@BeginModule 24-Jun-2025 10:56:34 BST run: 65996932 subRun: 1 event: 92300 Failed to find product with label wclsmcnfsp:gauss ... returning empty vector %MSG %MSG-e DUNEAna: EnergyReco:energyrecnumumcs@BeginModule 24-Jun-2025 10:56:34 BST run: 65996932 subRun: 1 event: 92300 Failed to find product with label wclsmcnfsp:gauss ... returning empty vector %MSG 128 ***Wiener Filter*******ApplyPostfilter*** Running Ophitfinder with InputDigiType = 'recob' Found hits: 234! 24-Jun-2025 10:56:35 BST Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65996932_922_20231202T073525Z_gen_g4_detsim_hitreco_reco2.root" 24-Jun-2025 10:56:35 BST Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/1f/5d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65996932_922_20231202T073525Z_gen_g4_detsim_hitreco.root" ======================================================================================================================================== TimeTracker printout (sec) Min Avg Max Median RMS nEvts ======================================================================================================================================== Full event 0.543504 5.47908 106.371 3.80566 10.6236 100 ---------------------------------------------------------------------------------------------------------------------------------------- source:RootInput(read) 0.0225806 0.0271672 0.064214 0.0234447 0.00879063 100 reco:pandora:StandardPandora 0.14133 2.64532 99.6137 1.07388 10.0541 100 reco:pandoraTrack:LArPandoraTrackCreation 8.7324e-05 0.00491177 0.118098 0.00109699 0.0144428 100 reco:pandoraShower:LArPandoraModularShowerCreation 0.000126077 0.00737859 0.0705075 0.00239674 0.0136376 100 reco:pandoracalo:Calorimetry 7.1675e-05 0.0178063 0.509282 0.000913192 0.0734961 100 reco:pandorapid:Chi2ParticleID 2.8323e-05 0.000131796 0.00116188 9.9307e-05 0.000137622 100 reco:cvnmap:CVNMapper 1.8655e-05 0.00882847 0.0337005 0.00833901 0.00718698 100 reco:cvneva:CVNEvaluator 1.2503e-05 0.284839 1.7986 0.322001 0.208449 100 reco:energyrecnumu:EnergyReco 0.0666685 0.135069 0.852929 0.0960657 0.11981 100 reco:energyrecnue:EnergyReco 0.000145263 0.0121087 0.0699893 0.000686521 0.0241567 100 reco:energyrecnc:EnergyReco 0.00142975 0.104402 0.255048 0.122435 0.0542263 100 reco:energyrecnumurange:EnergyReco 0.00013287 0.0242859 0.461261 0.00133175 0.0750408 100 reco:energyrecnumumcs:EnergyReco 8.7674e-05 0.0232146 0.444471 0.00131814 0.0713708 100 reco:opdec:Deconvolution 0.0703203 0.237888 0.384489 0.235617 0.0564358 100 reco:ophitspe:OpHitFinderDeco 0.225929 1.74101 4.22504 1.65562 0.706344 100 reco:opflash:OpFlashFinder 0.00014359 0.00107219 0.00400849 0.00100706 0.000521184 100 reco:rns:RandomNumberSaver 1.7653e-05 4.05081e-05 0.000339668 3.31825e-05 4.20231e-05 100 reco:anglereconue:NuAngularReco 0.000121839 0.00119615 0.0309013 0.000317682 0.00369036 100 reco:anglereconumu:NuAngularReco 5.5835e-05 0.00100195 0.0301651 0.000176712 0.00355686 100 reco:anglereconuepfps:NuAngularReco 8.3637e-05 0.00112924 0.0306847 0.000286023 0.00361157 100 reco:anglereconumupfps:NuAngularReco 7.3678e-05 0.00104359 0.0303107 0.000213612 0.00357407 100 [art]:TriggerResults:TriggerResultInserter 8.606e-06 1.24562e-05 7.3929e-05 1.1306e-05 6.61624e-06 100 end_path:out1:RootOutput 3.075e-06 5.74301e-06 2.154e-05 5.621e-06 2.20513e-06 100 end_path:out1:RootOutput(write) 0.00516692 0.198011 0.433259 0.216884 0.0944735 100 ======================================================================================================================================== ==================================================================================================== MemoryTracker summary (base-10 MB units used) Peak virtual memory usage (VmPeak) : 3582.27 MB Peak resident set size usage (VmHWM): 1850.6 MB ==================================================================================================== Art has completed and will exit with status 0. === End last 100 lines of lar log file === lar exit code 0