21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.

Jobsub ID 212554.168@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	212554.168@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	7257
Stage ID	1
User name	smanthey@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-05-31 16:43:25
Site	US_FNAL-T1
Entry	CMSHTPC_T1_US_FNAL_condce_opp1_whole
Last heartbeat	2025-06-01 15:50:27
From worker node	Hostname	dunegli-42342-0-cmswn2385.fnal.gov
	cpuinfo	Intel(R) Xeon(R) CPU E5-2670 v3 @ 2.30GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2025-06-01 04:36:10
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250205T002157Z_gen_001839_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	11h (40446s)
	CPU time	11h (39742s = 98%)
	Max RSS bytes	1763880960 (1682 MiB)
Outputting started	2025-06-01 15:50:17
Output files
Finished	2025-06-01 15:50:27
List job events Wrapper job log

Jobscript log (last 10,000 characters)

rley	11		0.97	 (-152.44, 448.06, 1924.27)	132311
marley	-12		0.85	 (1080.79, 5033.92, 4652.59)	132311
marley	22		0.03	 (-152.76, 448.16, 1924.36)	0
marley	22		1.61	 (-161.28, 475.25, 1930.33)	0
marley	22		0.65	 (-157.53, 445.81, 1921.80)	0
marley	22		2.09	 (-143.38, 416.39, 1887.27)	0
marley	1000190400	37216.54 (-152.43, 448.05, 1924.32)	0
marley	11		14.86	 (-157.67, 446.69, 1925.76)	0
Signal Track IDs: 132310; 132311; 132312; 132313; 132314; 132315; 132374; 132375; [0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 01-Jun-2025 13:54:53 UTC  run: 5055 subRun: 0 event: 18389
[34mOpFlash PE 400.12 with purity 0.53 time 1.51 plane 0
 - Vertex (-327.50, 427.68, 1902.50)
	*** 1st Sanity check: Ratio 0.18 <= 1.00
	*** 2nd Sanity check: #OpHits 11 >= 11
OpFlash PE 67.25 with purity 0.14 time 2.22 plane 0
 - Vertex (-327.50, 511.16, 1409.34)
	*** 1st Sanity check: Ratio 0.25 <= 1.00
	*** 2nd Sanity check: #OpHits 6 >= 6
OpFlash PE 151.19 with purity 0.77 time 2.09 plane 0
 - Vertex (-327.50, 393.49, 1894.69)
	*** 1st Sanity check: Ratio 0.42 <= 1.00
	*** 2nd Sanity check: #OpHits 3 >= 3
OpFlash PE 22.55 with purity 0.00 time 2.49 plane 1
 - Vertex (60.07, 743.30, 1722.86)
	*** 1st Sanity check: Ratio 0.36 <= 1.00
	*** 2nd Sanity check: #OpHits 3 >= 3
OpFlash PE 92.33 with purity 0.57 time 2.59 plane 0
 - Vertex (-327.50, 373.40, 1973.46)
	*** 1st Sanity check: Ratio 0.57 <= 1.00
	*** 2nd Sanity check: #OpHits 4 >= 4
OpFlash PE 23.24 with purity 0.51 time 2.83 plane 0
 - Vertex (-327.50, -34.99, 2003.82)
	*** 1st Sanity check: Ratio 0.43 <= 1.00
	*** 2nd Sanity check: #OpHits 3 >= 3
[0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 01-Jun-2025 13:54:54 UTC  run: 5055 subRun: 0 event: 18389
[36m
# OpHits (ophit10ppm) in full geometry: 53124
# OpFlashes (solarflash) in full geometry: 4223
# Hits (gaushit) in each view: 6436, 6440, 6554, 0
# Cluster from the hits: 2673, 2736, 5544, 0
# Tracks (pmtracktc) in full geometry: 0[0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 01-Jun-2025 14:53:13 UTC  run: 5055 subRun: 0 event: 18389
[32m*** Matched preselection cluster: 
 - MainTrackID 132312
 - Purity 1.00 Hits 4
 - Gen 1 -> marley
 - TPC 102
 - #AdjCl 14 ( 0 signal )
 - Reco Time,Y,Z ( 2954.25, 185.41, 1923.41 )
*** Matched flash: 
 - Purity 0.00 Plane 3 #Hits 6 PE 2689.56 MaxPE 708.76
 - Time 1997.95 Fast 0.99 Residual 148962.19
 - Reco Time,Y,Z ( 1997.95, 220.00, 2188.38 )
*** Reconstructed Interaction Vertex: 
 - True X,Y,Z ( -152.43, 448.05, 1924.32 )
 - Reco X,Y,Z ( 171.49, 185.41, 1923.41 )[0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 01-Jun-2025 14:53:13 UTC  run: 5055 subRun: 0 event: 18389
[32m*** Matched preselection cluster: 
 - MainTrackID 132310
 - Purity 1.00 Hits 7
 - Gen 1 -> marley
 - TPC 102
 - #AdjCl 13 ( 1 signal )
 - Reco Time,Y,Z ( 2992.85, 447.48, 1924.73 )
*** Matched flash: 
 - Purity 0.00 Plane 1 #Hits 4 PE 1180.56 MaxPE 504.01
 - Time 921.59 Fast 0.99 Residual 87254.28
 - Reco Time,Y,Z ( 921.59, 743.30, 1939.46 )
*** Reconstructed Interaction Vertex: 
 - True X,Y,Z ( -152.43, 448.05, 1924.32 )
 - Reco X,Y,Z ( -7.53, 447.48, 1924.73 )[0m
%MSG
Begin processing the 10th record. run: 5055 subRun: 0 event: 18390 at 01-Jun-2025 14:53:14 UTC
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 01-Jun-2025 14:53:18 UTC  run: 5055 subRun: 0 event: 18390
[35m
#########################################
Event: 18390 Flag: 1093121210
TPC Frequency in [MHz]: 2.00
TPC Tick in [us]: 0.50
TPC DriftLength in [cm]: 650.07
TPC DriftTime in [us]: 4048.69
#########################################[0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 01-Jun-2025 14:53:19 UTC  run: 5055 subRun: 0 event: 18390
[91m
There are a total of 132683 Particles in the event

# of particles 8	from gen 1 marley
# of particles 31074	from gen 2 Ar39GenInLAr
# of particles 3586	from gen 3 Kr85GenInLAr
# of particles 0	from gen 4 Ar42GenInLAr
# of particles 0	from gen 5 K42From42ArGenInLAr
# of particles 38	from gen 6 Rn222ChainRn222GenInLAr
# of particles 29	from gen 7 Rn222ChainPo218GenInLAr
# of particles 49	from gen 8 Rn222ChainPb214GenInLAr
# of particles 43	from gen 9 Rn222ChainBi214GenInLAr
# of particles 15	from gen 10 Rn222ChainPb210GenInLAr
# of particles 32	from gen 11 Rn220ChainPb212GenInLAr
# of particles 229	from gen 12 K40GenInCathode
# of particles 2942	from gen 13 U238ChainGenInCathode
# of particles 3488	from gen 14 Th232ChainGenInCathode
# of particles 103	from gen 15 K40GenInAnode
# of particles 1334	from gen 16 U238ChainGenInAnode
# of particles 1636	from gen 17 Th232ChainGenInAnode
# of particles 5	from gen 18 Rn222ChainGenInPDS
# of particles 2	from gen 19 K42From42ArGenInUpperMesh1x8x14
# of particles 2	from gen 20 Rn222ChainFromPo218GenInUpperMesh1x8x14
# of particles 29	from gen 21 Rn222ChainFromPb214GenInUpperMesh1x8x14
# of particles 45	from gen 22 Rn222ChainFromBi214GenInUpperMesh1x8x14
# of particles 39	from gen 23 Rn222ChainFromPb210GenInUpperMesh1x8x14
# of particles 30	from gen 24 Rn222ChainFromBi210GenInUpperMesh1x8x14
# of particles 47	from gen 25 Rn220ChainFromPb212GenInUpperMesh1x8x14
# of particles 84313	from gen 26 CavernwallGammasAtLAr1x8x14
# of particles 3534	from gen 27 foamGammasAtLAr1x8x14
# of particles 30	from gen 28 CavernwallNeutronsAtLAr1x8x14
# of particles 1	from gen 29 CryostatNGammasAtLAr1x8x14
# of particles 0	from gen 30 CavernNGammasAtLAr1x8x14[0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 01-Jun-2025 14:53:19 UTC  run: 5055 subRun: 0 event: 18390
[33m
Number of Signal Particles: 7
Neutrino Interaction: -1
Neutrino Energy: 5.21 MeV
Position (314.19, -279.53, 804.10) cm
Gen.	PdgCode		Energy		EndPosition		Mother
------------------------------------------------------------------------
marley	11		1.12	 (314.25, -279.52, 804.10)	132647
marley	-12		0.70	 (4655.16, 1529.02, 1065.36)	132647
marley	1000200400	37214.71 (314.19, -279.53, 804.10)	132647
marley	22		0.03	 (314.19, -279.54, 804.11)	0
marley	22		1.61	 (291.34, -289.04, 788.59)	0
marley	22		0.65	 (297.03, -263.45, 830.24)	0
marley	1000190400	37216.53 (314.19, -279.53, 804.10)	0
marley	11		1.93	 (314.34, -279.50, 803.92)	0
Signal Track IDs: 132646; 132647; 132648; 132649; 132650; 132698; 132699; 132700; [0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 01-Jun-2025 14:53:32 UTC  run: 5055 subRun: 0 event: 18390
[34mOpFlash PE 226.07 with purity 0.00 time -2.68 plane 1
 - Vertex (60.07, 743.30, 1939.46)
	*** 1st Sanity check: Ratio 0.27 <= 1.00
	*** 2nd Sanity check: #OpHits 4 >= 4
[0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 01-Jun-2025 14:53:33 UTC  run: 5055 subRun: 0 event: 18390
[36m
# OpHits (ophit10ppm) in full geometry: 51887
# OpFlashes (solarflash) in full geometry: 4116
# Hits (gaushit) in each view: 6505, 6460, 6433, 0
# Cluster from the hits: 2709, 2731, 5448, 0
# Tracks (pmtracktc) in full geometry: 0[0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 01-Jun-2025 15:50:16 UTC  run: 5055 subRun: 0 event: 18390
[32m*** Matched preselection cluster: 
 - MainTrackID 132646
 - Purity 1.00 Hits 1
 - Gen 1 -> marley
 - TPC 42
 - #AdjCl 9 ( 1 signal )
 - Reco Time,Y,Z ( 66.77, -279.62, 803.96 )
*** Matched flash: 
 - Purity 0.00 Plane 3 #Hits 7 PE 1382.88 MaxPE 452.01
 - Time -609.37 Fast 0.36 Residual 109392.30
 - Reco Time,Y,Z ( -609.37, -220.00, 2188.38 )
*** Reconstructed Interaction Vertex: 
 - True X,Y,Z ( 314.19, -279.53, 804.10 )
 - Reco X,Y,Z ( 216.47, -279.62, 803.96 )[0m
%MSG
01-Jun-2025 15:50:16 UTC  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/dd/32/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250205T002157Z_gen_001839_supernova_g4stage1_g4stage2_detsim_reco.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                          3418.1        3987.68       4450.72       4026.33       347.042        10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000487957     6.8288        68.2797     0.000747932     20.4836        10     
end_path:solarnuana:SolarNuAna      3418.1        3980.85       4450.72       4026.33       345.258        10     
====================================================================================================================
%MSG-i NuRandomService:  SolarNuAna:solarnuana@EndJob 01-Jun-2025 15:50:16 UTC  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1

%MSG

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2490.36 MB
  Peak resident set size usage (VmHWM): 1763.88 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 10 passed = 10 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         10         10          0 solarnuana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 39709.638843 Real = 40404.336320

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 2490.36 VmHWM = 1763.88

Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Output file: fdhd_ana_2025-06-01T_043620Z.root
Output hist file: fdhd_ana_2025-06-01T_043620Z_hist.root
lar exit code 0