21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.

Jobsub ID 212420.176@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	212420.176@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	7257
Stage ID	1
User name	smanthey@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-05-31 11:25:51
Site	US_FNAL-T1
Entry	CMSHTPC_T1_US_FNAL_condce_opp1_whole
Last heartbeat	2025-06-01 14:49:06
From worker node	Hostname	dunegli-42342-0-cmswn2385.fnal.gov
	cpuinfo	Intel(R) Xeon(R) CPU E5-2670 v3 @ 2.30GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2025-06-01 02:14:47
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T180510Z_gen_000020_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	12h (45235s)
	CPU time	12h (44413s = 98%)
	Max RSS bytes	1773969408 (1691 MiB)
Outputting started	2025-06-01 14:48:46
Output files
Finished	2025-06-01 14:49:06
List job events Wrapper job log

Jobscript log (last 10,000 characters)

nity check: Ratio 0.39 <= 1.00
	*** 2nd Sanity check: #OpHits 4 >= 4
OpFlash PE 68.98 with purity 0.15 time 2.56 plane 0
 - Vertex (-327.50, 211.69, 844.68)
	*** 1st Sanity check: Ratio 0.44 <= 1.00
	*** 2nd Sanity check: #OpHits 3 >= 3
[0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 01-Jun-2025 12:41:10 UTC  run: 5038 subRun: 0 event: 199
[36m
# OpHits (ophit10ppm) in full geometry: 52693
# OpFlashes (solarflash) in full geometry: 4269
# Hits (gaushit) in each view: 6538, 6503, 6597, 0
# Cluster from the hits: 2724, 2767, 5560, 0
# Tracks (pmtracktc) in full geometry: 0[0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 01-Jun-2025 13:44:22 UTC  run: 5038 subRun: 0 event: 199
[32m*** Matched preselection cluster: 
 - MainTrackID 132830
 - Purity 1.00 Hits 28
 - Gen 1 -> marley
 - TPC 63
 - #AdjCl 12 ( 4 signal )
 - Reco Time,Y,Z ( 2898.55, 614.46, 1051.12 )
*** Matched flash: 
 - Purity 0.00 Plane 1 #Hits 6 PE 1921.17 MaxPE 679.51
 - Time 1294.85 Fast 0.69 Residual 220843.75
 - Reco Time,Y,Z ( 1294.85, 743.30, 1045.94 )
*** Reconstructed Interaction Vertex: 
 - True X,Y,Z ( -137.46, 620.18, 1058.17 )
 - Reco X,Y,Z ( 67.54, 614.46, 1051.12 )[0m
%MSG
Begin processing the 10th record. run: 5038 subRun: 0 event: 200 at 01-Jun-2025 13:44:22 UTC
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 01-Jun-2025 13:45:33 UTC  run: 5038 subRun: 0 event: 200
[35m
#########################################
Event: 200 Flag: 1774865042
TPC Frequency in [MHz]: 2.00
TPC Tick in [us]: 0.50
TPC DriftLength in [cm]: 650.07
TPC DriftTime in [us]: 4048.69
#########################################[0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 01-Jun-2025 13:45:35 UTC  run: 5038 subRun: 0 event: 200
[91m
There are a total of 132733 Particles in the event

# of particles 9	from gen 1 marley
# of particles 30804	from gen 2 Ar39GenInLAr
# of particles 3556	from gen 3 Kr85GenInLAr
# of particles 2	from gen 4 Ar42GenInLAr
# of particles 0	from gen 5 K42From42ArGenInLAr
# of particles 30	from gen 6 Rn222ChainRn222GenInLAr
# of particles 30	from gen 7 Rn222ChainPo218GenInLAr
# of particles 76	from gen 8 Rn222ChainPb214GenInLAr
# of particles 62	from gen 9 Rn222ChainBi214GenInLAr
# of particles 48	from gen 10 Rn222ChainPb210GenInLAr
# of particles 21	from gen 11 Rn220ChainPb212GenInLAr
# of particles 210	from gen 12 K40GenInCathode
# of particles 2915	from gen 13 U238ChainGenInCathode
# of particles 3679	from gen 14 Th232ChainGenInCathode
# of particles 102	from gen 15 K40GenInAnode
# of particles 1318	from gen 16 U238ChainGenInAnode
# of particles 1792	from gen 17 Th232ChainGenInAnode
# of particles 1	from gen 18 Rn222ChainGenInPDS
# of particles 3	from gen 19 K42From42ArGenInUpperMesh1x8x14
# of particles 4	from gen 20 Rn222ChainFromPo218GenInUpperMesh1x8x14
# of particles 23	from gen 21 Rn222ChainFromPb214GenInUpperMesh1x8x14
# of particles 57	from gen 22 Rn222ChainFromBi214GenInUpperMesh1x8x14
# of particles 22	from gen 23 Rn222ChainFromPb210GenInUpperMesh1x8x14
# of particles 35	from gen 24 Rn222ChainFromBi210GenInUpperMesh1x8x14
# of particles 62	from gen 25 Rn220ChainFromPb212GenInUpperMesh1x8x14
# of particles 84313	from gen 26 CavernwallGammasAtLAr1x8x14
# of particles 3534	from gen 27 foamGammasAtLAr1x8x14
# of particles 24	from gen 28 CavernwallNeutronsAtLAr1x8x14
# of particles 1	from gen 29 CryostatNGammasAtLAr1x8x14
# of particles 0	from gen 30 CavernNGammasAtLAr1x8x14[0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 01-Jun-2025 13:45:35 UTC  run: 5038 subRun: 0 event: 200
[33m
Number of Signal Particles: 8
Neutrino Interaction: -1
Neutrino Energy: 21.63 MeV
Position (350.53, -143.43, 681.93) cm
Gen.	PdgCode		Energy		EndPosition		Mother
------------------------------------------------------------------------
marley	11		1.70	 (425.09, -176.36, 725.05)	132702
marley	-12		0.13	 (-1786.56, -5033.92, -4659.84)	132702
marley	1000200400	37214.71 (350.53, -143.43, 681.93)	132702
marley	22		0.03	 (328.41, -93.03, 705.61)	0
marley	22		1.61	 (506.96, -401.71, 618.72)	0
marley	22		0.65	 (317.83, -153.04, 724.21)	0
marley	22		2.09	 (315.46, -135.11, 692.46)	0
marley	1000190400	37216.55 (350.53, -143.43, 681.93)	0
marley	11		16.23	 (323.58, -174.57, 754.15)	0
Signal Track IDs: 132701; 132702; 132703; 132704; 132705; 132706; 132782; 132783; 132784; [0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 01-Jun-2025 13:45:56 UTC  run: 5038 subRun: 0 event: 200
[34mOpFlash PE 41.55 with purity 0.00 time -1.56 plane 2
 - Vertex (111.39, -743.30, 152.42)
	*** 1st Sanity check: Ratio 0.34 <= 1.00
	*** 2nd Sanity check: #OpHits 4 >= 4
OpFlash PE 43.90 with purity 0.00 time -0.38 plane 1
 - Vertex (135.07, 743.30, 1045.94)
	*** 1st Sanity check: Ratio 0.46 <= 1.00
	*** 2nd Sanity check: #OpHits 4 >= 4
OpFlash PE 44.73 with purity 0.34 time 0.69 plane 0
 - Vertex (-327.50, -196.44, 746.40)
	*** 1st Sanity check: Ratio 0.22 <= 1.00
	*** 2nd Sanity check: #OpHits 6 >= 6
OpFlash PE 20.71 with purity 0.00 time 0.14 plane 2
 - Vertex (285.07, -743.30, 1128.92)
	*** 1st Sanity check: Ratio 0.36 <= 1.00
	*** 2nd Sanity check: #OpHits 3 >= 3
OpFlash PE 17.87 with purity 0.29 time 0.78 plane 0
 - Vertex (-327.50, -129.28, 601.22)
	*** 1st Sanity check: Ratio 0.43 <= 1.00
	*** 2nd Sanity check: #OpHits 3 >= 3
OpFlash PE 50.51 with purity 0.24 time 1.13 plane 2
 - Vertex (141.17, -743.30, 748.10)
	*** 1st Sanity check: Ratio 0.21 <= 1.00
	*** 2nd Sanity check: #OpHits 7 >= 7
OpFlash PE 18.81 with purity 0.46 time 0.82 plane 0
 - Vertex (-327.50, 58.72, 678.96)
	*** 1st Sanity check: Ratio 0.40 <= 1.00
	*** 2nd Sanity check: #OpHits 3 >= 3
OpFlash PE 62.62 with purity 0.00 time 1.01 plane 0
 - Vertex (-327.50, 128.70, 1529.62)
	*** 1st Sanity check: Ratio 0.43 <= 1.00
	*** 2nd Sanity check: #OpHits 5 >= 5
OpFlash PE 40.46 with purity 0.04 time 1.68 plane 0
 - Vertex (-327.50, 395.44, 751.74)
	*** 1st Sanity check: Ratio 0.30 <= 1.00
	*** 2nd Sanity check: #OpHits 4 >= 4
[0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 01-Jun-2025 13:45:57 UTC  run: 5038 subRun: 0 event: 200
[36m
# OpHits (ophit10ppm) in full geometry: 52481
# OpFlashes (solarflash) in full geometry: 4199
# Hits (gaushit) in each view: 6988, 6941, 6760, 0
# Cluster from the hits: 2919, 2932, 5706, 0
# Tracks (pmtracktc) in full geometry: 0[0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 01-Jun-2025 14:48:42 UTC  run: 5038 subRun: 0 event: 200
[32m*** Matched preselection cluster: 
 - MainTrackID 132701
 - Purity 1.00 Hits 2
 - Gen 1 -> marley
 - TPC 42
 - #AdjCl 7 ( 0 signal )
 - Reco Time,Y,Z ( 90.38, -168.32, 776.63 )
*** Matched flash: 
 - Purity 0.00 Plane 3 #Hits 4 PE 1178.79 MaxPE 571.51
 - Time -424.84 Fast 0.44 Residual 139963.47
 - Reco Time,Y,Z ( -424.84, -220.00, 2188.38 )
*** Reconstructed Interaction Vertex: 
 - True X,Y,Z ( 350.53, -143.43, 681.93 )
 - Reco X,Y,Z ( 242.31, -168.32, 776.63 )[0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 01-Jun-2025 14:48:42 UTC  run: 5038 subRun: 0 event: 200
[32m*** Matched preselection cluster: 
 - MainTrackID 132701
 - Purity 0.53 Hits 2
 - Gen 1 -> marley
 - TPC 43
 - #AdjCl 2 ( 0 signal )
 - Reco Time,Y,Z ( 90.07, 544.54, 776.70 )
*** Matched flash: 
 - Purity 0.00 Plane 1 #Hits 5 PE 1054.35 MaxPE 496.66
 - Time -605.20 Fast 0.48 Residual 128756.71
 - Reco Time,Y,Z ( -605.20, 743.30, 748.10 )
*** Reconstructed Interaction Vertex: 
 - True X,Y,Z ( 350.53, -143.43, 681.93 )
 - Reco X,Y,Z ( 213.40, 544.54, 776.70 )[0m
%MSG
01-Jun-2025 14:48:42 UTC  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/c5/ce/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250203T180510Z_gen_000020_supernova_g4stage1_g4stage2_detsim_reco.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                          3788.98       4458.7        5173.2        4401.77       530.15         10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.00059719       7.02         70.192      0.000937607     21.0573        10     
end_path:solarnuana:SolarNuAna      3788.98       4451.68       5173.2        4366.67       529.676        10     
====================================================================================================================
%MSG-i NuRandomService:  SolarNuAna:solarnuana@EndJob 01-Jun-2025 14:48:42 UTC  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1

%MSG

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2495.55 MB
  Peak resident set size usage (VmHWM): 1773.97 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 10 passed = 10 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         10         10          0 solarnuana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 44389.091649 Real = 45203.872297

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 2495.55 VmHWM = 1773.97

Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Output file: fdhd_ana_2025-06-01T_021455Z.root
Output hist file: fdhd_ana_2025-06-01T_021455Z_hist.root
lar exit code 0