21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.

Jobsub ID 404957.19@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	404957.19@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	7257
Stage ID	1
User name	smanthey@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-05-31 17:29:37
Site	US_FNAL-T1
Entry	CMSHTPC_T1_US_FNAL_condce_opp1_whole
Last heartbeat	2025-06-01 15:40:21
From worker node	Hostname	dunegli-42342-0-cmswn2385.fnal.gov
	cpuinfo	Intel(R) Xeon(R) CPU E5-2670 v3 @ 2.30GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2025-06-01 04:14:46
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250205T183519Z_gen_002693_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	11h (41098s)
	CPU time	11h (40534s = 98%)
	Max RSS bytes	1760657408 (1679 MiB)
Outputting started	2025-06-01 15:40:10
Output files
Finished	2025-06-01 15:40:21
List job events Wrapper job log

Jobscript log (last 10,000 characters)

32m*** Matched preselection cluster: 
 - MainTrackID 132340
 - Purity 1.00 Hits 9
 - Gen 1 -> marley
 - TPC 71
 - #AdjCl 13 ( 1 signal )
 - Reco Time,Y,Z ( 454.79, 589.67, 1202.21 )
*** Matched flash: 
 - Purity 0.00 Plane 0 #Hits 5 PE 415.43 MaxPE 137.79
 - Time 309.23 Fast 0.70 Residual 5368.91
 - Reco Time,Y,Z ( 309.23, 538.10, 1150.94 )
*** Reconstructed Interaction Vertex: 
 - True X,Y,Z ( 252.60, 587.69, 1199.54 )
 - Reco X,Y,Z ( 301.66, 589.67, 1202.21 )[0m
%MSG
Begin processing the 10th record. run: 5057 subRun: 0 event: 26930 at 01-Jun-2025 14:39:02 UTC
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 01-Jun-2025 14:39:06 UTC  run: 5057 subRun: 0 event: 26930
[35m
#########################################
Event: 26930 Flag: 395074737
TPC Frequency in [MHz]: 2.00
TPC Tick in [us]: 0.50
TPC DriftLength in [cm]: 650.07
TPC DriftTime in [us]: 4048.69
#########################################[0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 01-Jun-2025 14:39:07 UTC  run: 5057 subRun: 0 event: 26930
[91m
There are a total of 133302 Particles in the event

# of particles 14	from gen 1 marley
# of particles 31399	from gen 2 Ar39GenInLAr
# of particles 3576	from gen 3 Kr85GenInLAr
# of particles 5	from gen 4 Ar42GenInLAr
# of particles 1	from gen 5 K42From42ArGenInLAr
# of particles 22	from gen 6 Rn222ChainRn222GenInLAr
# of particles 27	from gen 7 Rn222ChainPo218GenInLAr
# of particles 68	from gen 8 Rn222ChainPb214GenInLAr
# of particles 47	from gen 9 Rn222ChainBi214GenInLAr
# of particles 20	from gen 10 Rn222ChainPb210GenInLAr
# of particles 12	from gen 11 Rn220ChainPb212GenInLAr
# of particles 220	from gen 12 K40GenInCathode
# of particles 2961	from gen 13 U238ChainGenInCathode
# of particles 3645	from gen 14 Th232ChainGenInCathode
# of particles 126	from gen 15 K40GenInAnode
# of particles 1427	from gen 16 U238ChainGenInAnode
# of particles 1695	from gen 17 Th232ChainGenInAnode
# of particles 0	from gen 18 Rn222ChainGenInPDS
# of particles 0	from gen 19 K42From42ArGenInUpperMesh1x8x14
# of particles 3	from gen 20 Rn222ChainFromPo218GenInUpperMesh1x8x14
# of particles 20	from gen 21 Rn222ChainFromPb214GenInUpperMesh1x8x14
# of particles 40	from gen 22 Rn222ChainFromBi214GenInUpperMesh1x8x14
# of particles 26	from gen 23 Rn222ChainFromPb210GenInUpperMesh1x8x14
# of particles 37	from gen 24 Rn222ChainFromBi210GenInUpperMesh1x8x14
# of particles 50	from gen 25 Rn220ChainFromPb212GenInUpperMesh1x8x14
# of particles 84313	from gen 26 CavernwallGammasAtLAr1x8x14
# of particles 3534	from gen 27 foamGammasAtLAr1x8x14
# of particles 13	from gen 28 CavernwallNeutronsAtLAr1x8x14
# of particles 1	from gen 29 CryostatNGammasAtLAr1x8x14
# of particles 0	from gen 30 CavernNGammasAtLAr1x8x14[0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 01-Jun-2025 14:39:07 UTC  run: 5057 subRun: 0 event: 26930
[33m
Number of Signal Particles: 7
Neutrino Interaction: -1
Neutrino Energy: 27.66 MeV
Position (62.06, 575.74, 1290.25) cm
Gen.	PdgCode		Energy		EndPosition		Mother
------------------------------------------------------------------------
marley	22		1.46	 (60.81, 565.84, 1287.31)	133278
marley	1000180400	37215.54 (62.06, 575.74, 1290.25)	133278
marley	12		0.04	 (-1990.57, 5033.92, -2016.16)	133278
marley	11		0.51	 (62.06, 575.74, 1290.25)	133278
marley	11		0.51	 (62.06, 575.74, 1290.25)	133278
marley	11		0.51	 (62.06, 575.74, 1290.25)	133278
marley	11		0.51	 (62.06, 575.74, 1290.25)	133278
marley	11		0.51	 (62.06, 575.74, 1290.25)	133278
marley	11		0.51	 (62.06, 575.74, 1290.25)	133278
marley	22		0.03	 (61.84, 575.83, 1290.28)	0
marley	22		1.61	 (70.08, 567.66, 1285.74)	0
marley	22		0.65	 (58.45, 565.67, 1295.97)	0
marley	1000190400	37216.55 (62.06, 575.74, 1290.25)	0
marley	11		24.36	 (60.89, 575.47, 1294.59)	0
Signal Track IDs: 133277; 133278; 133279; 133280; 133281; 133315; 133316; 133317; 133318; 133319; 133320; 133322; 133323; 133324; [0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 01-Jun-2025 14:39:21 UTC  run: 5057 subRun: 0 event: 26930
[34mOpFlash PE 26.94 with purity 0.00 time -1.98 plane 0
 - Vertex (-327.50, -433.26, 723.50)
	*** 1st Sanity check: Ratio 0.46 <= 1.00
	*** 2nd Sanity check: #OpHits 3 >= 3
OpFlash PE 20.16 with purity 0.00 time -1.16 plane 2
 - Vertex (211.77, -743.30, 152.42)
	*** 1st Sanity check: Ratio 0.36 <= 1.00
	*** 2nd Sanity check: #OpHits 3 >= 3
OpFlash PE 49.27 with purity 0.46 time 0.64 plane 1
 - Vertex (255.17, 743.30, 1076.84)
	*** 1st Sanity check: Ratio 0.26 <= 1.00
	*** 2nd Sanity check: #OpHits 5 >= 5
OpFlash PE 59.28 with purity 0.23 time 0.75 plane 0
 - Vertex (-327.50, 34.99, 1541.21)
	*** 1st Sanity check: Ratio 0.28 <= 1.00
	*** 2nd Sanity check: #OpHits 5 >= 5
OpFlash PE 63.11 with purity 0.70 time 1.35 plane 0
 - Vertex (-327.50, 496.07, 976.93)
	*** 1st Sanity check: Ratio 0.35 <= 1.00
	*** 2nd Sanity check: #OpHits 5 >= 5
OpFlash PE 51.28 with purity 0.25 time 1.52 plane 1
 - Vertex (110.04, 743.30, 1641.62)
	*** 1st Sanity check: Ratio 0.50 <= 1.00
	*** 2nd Sanity check: #OpHits 4 >= 4
OpFlash PE 1205.86 with purity 0.00 time 1.97 plane 2
 - Vertex (219.00, -743.30, 1939.46)
	*** 1st Sanity check: Ratio 0.21 <= 1.00
	*** 2nd Sanity check: #OpHits 9 >= 9
OpFlash PE 261.51 with purity 0.38 time 1.78 plane 0
 - Vertex (-327.50, 424.40, 1333.23)
	*** 1st Sanity check: Ratio 0.13 <= 1.00
	*** 2nd Sanity check: #OpHits 13 >= 13
OpFlash PE 217.33 with purity 0.25 time 1.41 plane 1
 - Vertex (60.07, 743.30, 1343.78)
	*** 1st Sanity check: Ratio 0.37 <= 1.00
	*** 2nd Sanity check: #OpHits 7 >= 7
OpFlash PE 30.14 with purity 0.00 time 1.85 plane 0
 - Vertex (-327.50, 563.47, 1625.12)
	*** 1st Sanity check: Ratio 0.50 <= 1.00
	*** 2nd Sanity check: #OpHits 3 >= 3
OpFlash PE 43.41 with purity 0.13 time 1.81 plane 0
 - Vertex (-327.50, 23.13, 1748.89)
	*** 1st Sanity check: Ratio 0.44 <= 1.00
	*** 2nd Sanity check: #OpHits 4 >= 4
OpFlash PE 16.06 with purity 0.38 time 2.28 plane 0
 - Vertex (-327.50, 236.76, 910.11)
	*** 1st Sanity check: Ratio 0.38 <= 1.00
	*** 2nd Sanity check: #OpHits 3 >= 3
OpFlash PE 91.22 with purity 0.22 time 2.21 plane 1
 - Vertex (117.26, 743.30, 1045.94)
	*** 1st Sanity check: Ratio 0.24 <= 1.00
	*** 2nd Sanity check: #OpHits 6 >= 6
OpFlash PE 529.49 with purity 0.00 time 2.47 plane 2
 - Vertex (249.33, -743.30, 1939.46)
	*** 1st Sanity check: Ratio 0.43 <= 1.00
	*** 2nd Sanity check: #OpHits 4 >= 4
OpFlash PE 143.96 with purity 0.00 time 2.64 plane 4
 - Vertex (285.07, -220.00, -96.50)
	*** 1st Sanity check: Ratio 0.49 <= 1.00
	*** 2nd Sanity check: #OpHits 3 >= 3
[0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 01-Jun-2025 14:39:22 UTC  run: 5057 subRun: 0 event: 26930
[36m
# OpHits (ophit10ppm) in full geometry: 52537
# OpFlashes (solarflash) in full geometry: 4152
# Hits (gaushit) in each view: 6819, 6761, 6688, 0
# Cluster from the hits: 2815, 2834, 5648, 0
# Tracks (pmtracktc) in full geometry: 0[0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 01-Jun-2025 15:39:43 UTC  run: 5057 subRun: 0 event: 26930
[32m*** Matched preselection cluster: 
 - MainTrackID 133277
 - Purity 0.97 Hits 12
 - Gen 1 -> marley
 - TPC 71
 - #AdjCl 12 ( 4 signal )
 - Reco Time,Y,Z ( 1645.71, 574.93, 1293.20 )
*** Matched flash: 
 - Purity 0.00 Plane 2 #Hits 6 PE 1198.85 MaxPE 399.31
 - Time -30.65 Fast 0.32 Residual 70646.48
 - Reco Time,Y,Z ( -30.65, -743.30, 1343.78 )
*** Reconstructed Interaction Vertex: 
 - True X,Y,Z ( 62.06, 575.74, 1290.25 )
 - Reco X,Y,Z ( 55.87, 574.93, 1293.20 )[0m
%MSG
01-Jun-2025 15:39:44 UTC  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/fardet-vd/0e/cc/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20250205T183519Z_gen_002693_supernova_g4stage1_g4stage2_detsim_reco.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                          3637.46       4060.55       4547.42       4064.49       298.845        10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000510557   0.00293949     0.0197198    0.000780352   0.00564085       10     
end_path:solarnuana:SolarNuAna      3637.46       4060.55       4547.42       4064.49       298.843        10     
====================================================================================================================
%MSG-i NuRandomService:  SolarNuAna:solarnuana@EndJob 01-Jun-2025 15:39:44 UTC  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1

%MSG

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2493.42 MB
  Peak resident set size usage (VmHWM): 1760.66 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 10 passed = 10 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         10         10          0 solarnuana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 40501.968188 Real = 40677.378379

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 2493.42 VmHWM = 1760.66

Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Output file: fdhd_ana_2025-06-01T_041456Z.root
Output hist file: fdhd_ana_2025-06-01T_041456Z_hist.root
lar exit code 0