21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.

Jobsub ID 209655.40@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	209655.40@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	7131
Stage ID	1
User name	smanthey@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	12582912000 (12000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-05-26 14:31:38
Site	US_FNAL-T1
Entry	CMSHTPC_T1_US_FNAL_condce_opp1_whole
Last heartbeat	2025-05-27 12:56:25
From worker node	Hostname	dunegli-42167-0-cmswn2411.fnal.gov
	cpuinfo	AMD EPYC 7551 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	12582912000 (12000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2025-05-27 10:13:55
Input files	usertests:fdvd_lowBg_reco2_2025-05-12T_061254Z.root
Jobscript	Exit code	0
	Real time	2h (9737s)
	CPU time	2h (9652s = 99%)
	Max RSS bytes	1369092096 (1305 MiB)
Outputting started	2025-05-27 12:56:14
Output files
Finished	2025-05-27 12:56:25
List job events Wrapper job log

Jobscript log (last 10,000 characters)

2ChainGenInPDS
# of particles 1	from gen 19 K42From42ArGenInUpperMesh
# of particles 0	from gen 20 Rn222ChainFromPo218GenInUpperMesh
# of particles 20	from gen 21 Rn222ChainFromPb214GenInUpperMesh
# of particles 72	from gen 22 Rn222ChainFromBi214GenInUpperMesh
# of particles 43	from gen 23 Rn222ChainFromPb210GenInUpperMesh
# of particles 36	from gen 24 Rn222ChainFromBi210GenInUpperMesh
# of particles 48	from gen 25 Rn220ChainFromPb212GenInUpperMesh
# of particles 27779	from gen 26 CavernwallGammasAtLAr
# of particles 3595	from gen 27 foamGammasAtLAr
# of particles 0	from gen 28 CavernwallNeutronsAtLAr
# of particles 0	from gen 29 CryostatNGammasAtLAr
# of particles 0	from gen 30 CavernNGammasAtLAr[0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 27-May-2025 12:31:31 UTC  run: 20000047 subRun: 0 event: 9
[33m
Number of Signal Particles: 7
Neutrino Interaction: -1
Neutrino Energy: 4.98 MeV
Position (6.56, -285.04, 1204.25) cm
	Gen.	PdgCode		Energy		EndPosition		Mother
------------------------------------------------------------------------
marley	11		0.77	 (6.57, -285.02, 1204.24)	76339
marley	-12		1.06	 (-1617.05, 5033.92, 2541.92)	76339
marley	1000200400	37214.71 (6.56, -285.04, 1204.25)	76339
marley	22		0.03	 (6.68, -284.69, 1204.14)	0
marley	22		1.61	 (6.37, -306.30, 1210.02)	0
marley	22		0.65	 (-0.18, -314.72, 1181.67)	0
marley	1000190400	37216.53 (6.56, -285.04, 1204.25)	0
marley	11		1.70	 (6.67, -285.10, 1204.30)	0
Signal Track IDs: 76338; 76339; 76340; 76341; 76342; 76375; 76376; 76377; [0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 27-May-2025 12:31:31 UTC  run: 20000047 subRun: 0 event: 9
[34m[0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 27-May-2025 12:31:31 UTC  run: 20000047 subRun: 0 event: 9
[36m
# OpHits (ophit10ppm) in full geometry: 25215
# OpFlashes (solarflash) in full geometry: 0
# Hits (gaushit) in each view: 3369, 3259, 4180, 0
# Cluster from the hits: 1507, 1517, 3555, 0
# Tracks (pmtracktc) in full geometry: 182[0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 27-May-2025 12:47:23 UTC  run: 20000047 subRun: 0 event: 9
[32m*** Matched preselection cluster: 
 - MainTrackID 76341
 - Purity 1.00 Hits 2
 - #AdjCl 7 ( 0 signal )
 - Gen 1 -> marley
 - Reco Time,Y,Z (3939.09, -315.76, 1198.44)
[0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 27-May-2025 12:47:23 UTC  run: 20000047 subRun: 0 event: 9
[32m*** Matched preselection cluster: 
 - MainTrackID 76342
 - Purity 0.68 Hits 3
 - #AdjCl 5 ( 0 signal )
 - Gen 1 -> marley
 - Reco Time,Y,Z (3962.78, 13.12, 1204.39)
[0m
%MSG
Begin processing the 10th record. run: 20000047 subRun: 0 event: 10 at 27-May-2025 12:47:23 UTC
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 27-May-2025 12:47:25 UTC  run: 20000047 subRun: 0 event: 10
[35m
TPC Frequency in [MHz]: 2.00
TPC Tick in [us]: 0.50
Event Flag: 1010468935
Succesfull reset of variables for evt 10
#########################################[0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 27-May-2025 12:47:26 UTC  run: 20000047 subRun: 0 event: 10
[91m
There are a total of 76811 Particles in the event

# of particles 10	from gen 1 marley
# of particles 31112	from gen 2 Ar39GenInLAr
# of particles 3556	from gen 3 Kr85GenInLAr
# of particles 6	from gen 4 Ar42GenInLAr
# of particles 0	from gen 5 K42From42ArGenInLAr
# of particles 39	from gen 6 Rn222ChainRn222GenInLAr
# of particles 21	from gen 7 Rn222ChainPo218GenInLAr
# of particles 46	from gen 8 Rn222ChainPb214GenInLAr
# of particles 56	from gen 9 Rn222ChainBi214GenInLAr
# of particles 14	from gen 10 Rn222ChainPb210GenInLAr
# of particles 30	from gen 11 Rn220ChainPb212GenInLAr
# of particles 229	from gen 12 K40GenInCathode
# of particles 3110	from gen 13 U238ChainGenInCathode
# of particles 3602	from gen 14 Th232ChainGenInCathode
# of particles 108	from gen 15 K40GenInAnode
# of particles 1422	from gen 16 U238ChainGenInAnode
# of particles 1631	from gen 17 Th232ChainGenInAnode
# of particles 1	from gen 18 Rn222ChainGenInPDS
# of particles 1	from gen 19 K42From42ArGenInUpperMesh
# of particles 1	from gen 20 Rn222ChainFromPo218GenInUpperMesh
# of particles 10	from gen 21 Rn222ChainFromPb214GenInUpperMesh
# of particles 90	from gen 22 Rn222ChainFromBi214GenInUpperMesh
# of particles 58	from gen 23 Rn222ChainFromPb210GenInUpperMesh
# of particles 57	from gen 24 Rn222ChainFromBi210GenInUpperMesh
# of particles 47	from gen 25 Rn220ChainFromPb212GenInUpperMesh
# of particles 28057	from gen 26 CavernwallGammasAtLAr
# of particles 3497	from gen 27 foamGammasAtLAr
# of particles 0	from gen 28 CavernwallNeutronsAtLAr
# of particles 0	from gen 29 CryostatNGammasAtLAr
# of particles 0	from gen 30 CavernNGammasAtLAr[0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 27-May-2025 12:47:26 UTC  run: 20000047 subRun: 0 event: 10
[33m
Number of Signal Particles: 9
Neutrino Interaction: -1
Neutrino Energy: 61.77 MeV
Position (218.50, 217.87, 1821.26) cm
	Gen.	PdgCode		Energy		EndPosition		Mother
------------------------------------------------------------------------
marley	11		1.39	 (218.46, 217.64, 1821.34)	76803
marley	-12		0.43	 (1713.50, 5033.92, 114.42)	76803
marley	1000200400	37214.71 (218.50, 217.87, 1821.26)	76803
marley	22		0.03	 (218.37, 218.04, 1821.28)	0
marley	22		0.77	 (210.86, 225.24, 1825.17)	0
marley	22		0.84	 (140.60, 257.85, 1755.61)	0
marley	22		0.65	 (210.25, 214.86, 1832.34)	0
marley	22		2.09	 (202.60, 210.89, 1817.95)	0
marley	1000190400	37216.55 (218.50, 217.87, 1821.26)	0
marley	11		56.37	 (231.05, 213.63, 1820.29)	0
Signal Track IDs: 76802; 76803; 76804; 76805; 76806; 76807; 76808; 76903; 76904; 76905; [0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 27-May-2025 12:47:26 UTC  run: 20000047 subRun: 0 event: 10
[34m[0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 27-May-2025 12:47:26 UTC  run: 20000047 subRun: 0 event: 10
[36m
# OpHits (ophit10ppm) in full geometry: 26041
# OpFlashes (solarflash) in full geometry: 0
# Hits (gaushit) in each view: 3424, 3385, 4277, 0
# Cluster from the hits: 1537, 1572, 3642, 0
# Tracks (pmtracktc) in full geometry: 208[0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 27-May-2025 12:56:12 UTC  run: 20000047 subRun: 0 event: 10
[32m*** Matched preselection cluster: 
 - MainTrackID 76802
 - Purity 1.00 Hits 2
 - #AdjCl 9 ( 0 signal )
 - Gen 1 -> marley
 - Reco Time,Y,Z (921.42, 205.21, 1814.25)
[0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 27-May-2025 12:56:12 UTC  run: 20000047 subRun: 0 event: 10
[32m*** Matched preselection cluster: 
 - MainTrackID 76802
 - Purity 1.00 Hits 1
 - #AdjCl 8 ( 0 signal )
 - Gen 1 -> marley
 - Reco Time,Y,Z (853.71, 202.26, 1816.74)
[0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 27-May-2025 12:56:12 UTC  run: 20000047 subRun: 0 event: 10
[32m*** Matched preselection cluster: 
 - MainTrackID 76804
 - Purity 1.00 Hits 2
 - #AdjCl 8 ( 0 signal )
 - Gen 1 -> marley
 - Reco Time,Y,Z (1526.63, 210.98, 1817.87)
[0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 27-May-2025 12:56:12 UTC  run: 20000047 subRun: 0 event: 10
[32m*** Matched preselection cluster: 
 - MainTrackID 76802
 - Purity 1.00 Hits 3
 - #AdjCl 10 ( 0 signal )
 - Gen 1 -> marley
 - Reco Time,Y,Z (1215.43, 216.17, 1820.34)
[0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 27-May-2025 12:56:12 UTC  run: 20000047 subRun: 0 event: 10
[32m*** Matched preselection cluster: 
 - MainTrackID 76802
 - Purity 1.00 Hits 4
 - #AdjCl 9 ( 0 signal )
 - Gen 1 -> marley
 - Reco Time,Y,Z (1083.29, 216.83, 1820.96)
[0m
%MSG
27-May-2025 12:56:13 UTC  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/92/42/fdvd_lowBg_reco2_2025-05-12T_061254Z.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                          527.294       968.479        1137         994.276       159.488        10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000667978   0.00946347     0.0374352    0.00569117     0.0111382       10     
end_path:solarnuana:SolarNuAna      527.293       968.469       1136.98       994.263       159.483        10     
====================================================================================================================
%MSG-i NuRandomService:  SolarNuAna:solarnuana@EndJob 27-May-2025 12:56:13 UTC  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1

%MSG

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2007.3 MB
  Peak resident set size usage (VmHWM): 1369.09 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 10 passed = 10 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         10         10          0 solarnuana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 9631.220838 Real = 9712.044102

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 2007.3 VmHWM = 1369.09

Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Output file: fdhd_ana_2025-05-27T_101407Z.root
Output hist file: fdhd_ana_2025-05-27T_101407Z_hist.root
lar exit code 0