21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.

Jobsub ID 209655.31@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	209655.31@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	7131
Stage ID	1
User name	smanthey@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	12582912000 (12000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-05-26 14:31:38
Site	US_FNAL-T1
Entry	CMSHTPC_T1_US_FNAL_condce_opp1_whole
Last heartbeat	2025-05-27 12:50:19
From worker node	Hostname	dunegli-42167-0-cmswn2411.fnal.gov
	cpuinfo	AMD EPYC 7551 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	12582912000 (12000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2025-05-27 09:58:10
Input files	usertests:fdvd_lowBg_reco2_2025-05-11T_212016Z.root
Jobscript	Exit code	0
	Real time	2h (10316s)
	CPU time	2h (10231s = 99%)
	Max RSS bytes	1384366080 (1320 MiB)
Outputting started	2025-05-27 12:50:08
Output files
Finished	2025-05-27 12:50:19
List job events Wrapper job log

Jobscript log (last 10,000 characters)

larnuana@BeginModule 27-May-2025 12:18:47 UTC  run: 20000047 subRun: 0 event: 9
[35m
TPC Frequency in [MHz]: 2.00
TPC Tick in [us]: 0.50
Event Flag: 290830552
Succesfull reset of variables for evt 9
#########################################[0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 27-May-2025 12:18:48 UTC  run: 20000047 subRun: 0 event: 9
[91m
There are a total of 76059 Particles in the event

# of particles 9	from gen 1 marley
# of particles 30954	from gen 2 Ar39GenInLAr
# of particles 3446	from gen 3 Kr85GenInLAr
# of particles 3	from gen 4 Ar42GenInLAr
# of particles 0	from gen 5 K42From42ArGenInLAr
# of particles 37	from gen 6 Rn222ChainRn222GenInLAr
# of particles 18	from gen 7 Rn222ChainPo218GenInLAr
# of particles 56	from gen 8 Rn222ChainPb214GenInLAr
# of particles 35	from gen 9 Rn222ChainBi214GenInLAr
# of particles 36	from gen 10 Rn222ChainPb210GenInLAr
# of particles 32	from gen 11 Rn220ChainPb212GenInLAr
# of particles 231	from gen 12 K40GenInCathode
# of particles 3017	from gen 13 U238ChainGenInCathode
# of particles 3713	from gen 14 Th232ChainGenInCathode
# of particles 105	from gen 15 K40GenInAnode
# of particles 1320	from gen 16 U238ChainGenInAnode
# of particles 1467	from gen 17 Th232ChainGenInAnode
# of particles 7	from gen 18 Rn222ChainGenInPDS
# of particles 0	from gen 19 K42From42ArGenInUpperMesh
# of particles 2	from gen 20 Rn222ChainFromPo218GenInUpperMesh
# of particles 17	from gen 21 Rn222ChainFromPb214GenInUpperMesh
# of particles 48	from gen 22 Rn222ChainFromBi214GenInUpperMesh
# of particles 47	from gen 23 Rn222ChainFromPb210GenInUpperMesh
# of particles 35	from gen 24 Rn222ChainFromBi210GenInUpperMesh
# of particles 50	from gen 25 Rn220ChainFromPb212GenInUpperMesh
# of particles 27779	from gen 26 CavernwallGammasAtLAr
# of particles 3595	from gen 27 foamGammasAtLAr
# of particles 0	from gen 28 CavernwallNeutronsAtLAr
# of particles 0	from gen 29 CryostatNGammasAtLAr
# of particles 0	from gen 30 CavernNGammasAtLAr[0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 27-May-2025 12:18:48 UTC  run: 20000047 subRun: 0 event: 9
[33m
Number of Signal Particles: 8
Neutrino Interaction: -1
Neutrino Energy: 53.18 MeV
Position (-62.89, 731.58, 43.22) cm
	Gen.	PdgCode		Energy		EndPosition		Mother
------------------------------------------------------------------------
marley	11		0.78	 (-62.89, 731.59, 43.20)	76052
marley	-12		1.04	 (-2712.95, -5033.92, 4276.56)	76052
marley	1000200400	37214.71 (-62.89, 731.58, 43.22)	76052
marley	22		0.03	 (-62.97, 731.61, 43.41)	0
marley	22		0.77	 (-37.03, 729.71, 14.37)	0
marley	22		1.49	 (-57.30, 721.15, 42.25)	0
marley	22		2.10	 (-70.05, 737.97, 41.84)	0
marley	1000190400	37216.55 (-62.89, 731.58, 43.22)	0
marley	11		47.78	 (-63.56, 735.36, 27.78)	0
Signal Track IDs: 76051; 76052; 76053; 76054; 76055; 76056; 76106; 76107; 76108; [0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 27-May-2025 12:18:48 UTC  run: 20000047 subRun: 0 event: 9
[34m[0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 27-May-2025 12:18:48 UTC  run: 20000047 subRun: 0 event: 9
[36m
# OpHits (ophit10ppm) in full geometry: 24871
# OpFlashes (solarflash) in full geometry: 0
# Hits (gaushit) in each view: 3346, 3298, 4120, 0
# Cluster from the hits: 1515, 1504, 3516, 0
# Tracks (pmtracktc) in full geometry: 178[0m
%MSG
Begin processing the 10th record. run: 20000047 subRun: 0 event: 10 at 27-May-2025 12:34:27 UTC
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 27-May-2025 12:34:29 UTC  run: 20000047 subRun: 0 event: 10
[35m
TPC Frequency in [MHz]: 2.00
TPC Tick in [us]: 0.50
Event Flag: 1908726724
Succesfull reset of variables for evt 10
#########################################[0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 27-May-2025 12:34:30 UTC  run: 20000047 subRun: 0 event: 10
[91m
There are a total of 76352 Particles in the event

# of particles 7	from gen 1 marley
# of particles 31204	from gen 2 Ar39GenInLAr
# of particles 3487	from gen 3 Kr85GenInLAr
# of particles 2	from gen 4 Ar42GenInLAr
# of particles 1	from gen 5 K42From42ArGenInLAr
# of particles 36	from gen 6 Rn222ChainRn222GenInLAr
# of particles 27	from gen 7 Rn222ChainPo218GenInLAr
# of particles 46	from gen 8 Rn222ChainPb214GenInLAr
# of particles 41	from gen 9 Rn222ChainBi214GenInLAr
# of particles 16	from gen 10 Rn222ChainPb210GenInLAr
# of particles 28	from gen 11 Rn220ChainPb212GenInLAr
# of particles 249	from gen 12 K40GenInCathode
# of particles 2894	from gen 13 U238ChainGenInCathode
# of particles 3610	from gen 14 Th232ChainGenInCathode
# of particles 97	from gen 15 K40GenInAnode
# of particles 1359	from gen 16 U238ChainGenInAnode
# of particles 1468	from gen 17 Th232ChainGenInAnode
# of particles 0	from gen 18 Rn222ChainGenInPDS
# of particles 2	from gen 19 K42From42ArGenInUpperMesh
# of particles 1	from gen 20 Rn222ChainFromPo218GenInUpperMesh
# of particles 17	from gen 21 Rn222ChainFromPb214GenInUpperMesh
# of particles 63	from gen 22 Rn222ChainFromBi214GenInUpperMesh
# of particles 69	from gen 23 Rn222ChainFromPb210GenInUpperMesh
# of particles 32	from gen 24 Rn222ChainFromBi210GenInUpperMesh
# of particles 42	from gen 25 Rn220ChainFromPb212GenInUpperMesh
# of particles 28057	from gen 26 CavernwallGammasAtLAr
# of particles 3497	from gen 27 foamGammasAtLAr
# of particles 0	from gen 28 CavernwallNeutronsAtLAr
# of particles 0	from gen 29 CryostatNGammasAtLAr
# of particles 0	from gen 30 CavernNGammasAtLAr[0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 27-May-2025 12:34:30 UTC  run: 20000047 subRun: 0 event: 10
[33m
Number of Signal Particles: 7
Neutrino Interaction: -1
Neutrino Energy: 25.82 MeV
Position (10.40, -566.14, 1133.47) cm
	Gen.	PdgCode		Energy		EndPosition		Mother
------------------------------------------------------------------------
marley	11		0.86	 (10.39, -566.16, 1133.49)	76347
marley	-12		0.22	 (603.82, 5033.92, 3434.82)	76347
marley	22		2.09	 (23.73, -546.24, 1139.03)	0
marley	22		0.54	 (-16.03, -581.10, 1133.74)	0
marley	2212		942.38	 (10.38, -566.11, 1133.47)	0
marley	1000180390	36286.01 (10.40, -566.14, 1133.47)	0
marley	11		10.34	 (11.27, -562.33, 1134.55)	0
Signal Track IDs: 76346; 76347; 76348; 76349; 76350; 76385; 76386; [0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 27-May-2025 12:34:30 UTC  run: 20000047 subRun: 0 event: 10
[34m[0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 27-May-2025 12:34:30 UTC  run: 20000047 subRun: 0 event: 10
[36m
# OpHits (ophit10ppm) in full geometry: 25073
# OpFlashes (solarflash) in full geometry: 0
# Hits (gaushit) in each view: 3156, 3111, 3975, 0
# Cluster from the hits: 1429, 1437, 3421, 0
# Tracks (pmtracktc) in full geometry: 190[0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 27-May-2025 12:50:07 UTC  run: 20000047 subRun: 0 event: 10
[32m*** Matched preselection cluster: 
 - MainTrackID 76350
 - Purity 1.00 Hits 2
 - #AdjCl 2 ( 0 signal )
 - Gen 1 -> marley
 - Reco Time,Y,Z (3661.28, -562.82, 1126.10)
[0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 27-May-2025 12:50:07 UTC  run: 20000047 subRun: 0 event: 10
[32m*** Matched preselection cluster: 
 - MainTrackID 76350
 - Purity 1.00 Hits 2
 - #AdjCl 2 ( 0 signal )
 - Gen 1 -> marley
 - Reco Time,Y,Z (3576.81, -561.97, 1128.57)
[0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 27-May-2025 12:50:07 UTC  run: 20000047 subRun: 0 event: 10
[32m*** Matched preselection cluster: 
 - MainTrackID 76346
 - Purity 1.00 Hits 5
 - #AdjCl 2 ( 0 signal )
 - Gen 1 -> marley
 - Reco Time,Y,Z (3914.06, -563.89, 1133.68)
[0m
%MSG
27-May-2025 12:50:07 UTC  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/f2/a1/fdvd_lowBg_reco2_2025-05-11T_212016Z.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                          937.58        1026.56       1215.16       978.546       89.3597        10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000629367    0.0104703     0.0431598    0.00572279     0.0127476       10     
end_path:solarnuana:SolarNuAna      937.579       1026.55       1215.16       978.545       89.3619        10     
====================================================================================================================
%MSG-i NuRandomService:  SolarNuAna:solarnuana@EndJob 27-May-2025 12:50:07 UTC  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1

%MSG

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2011.02 MB
  Peak resident set size usage (VmHWM): 1384.37 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 10 passed = 10 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         10         10          0 solarnuana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 10211.367183 Real = 10291.563545

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 2011.02 VmHWM = 1384.37

Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Output file: fdhd_ana_2025-05-27T_095821Z.root
Output hist file: fdhd_ana_2025-05-27T_095821Z_hist.root
lar exit code 0