21 July 2025: This instance at RAL is read-only. Please do not try submitting new workflows for now.

Jobsub ID 209654.159@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	209654.159@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	7131
Stage ID	1
User name	smanthey@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	12582912000 (12000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-05-26 09:06:17
Site	US_FNAL-T1
Entry	CMSHTPC_T1_US_FNAL_condce_opp1_whole
Last heartbeat	2025-05-27 10:51:08
From worker node	Hostname	dunegli-42167-0-cmswn2411.fnal.gov
	cpuinfo	AMD EPYC 7551 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	12582912000 (12000 MiB)
	Wall seconds limit	171000 (47 hours)
	GPU
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2025-05-27 08:00:28
Input files	usertests:fdvd_lowBg_reco2_2025-05-12T_044628Z.root
Jobscript	Exit code	0
	Real time	2h (10227s)
	CPU time	2h (10136s = 99%)
	Max RSS bytes	1376858112 (1313 MiB)
Outputting started	2025-05-27 10:50:55
Output files
Finished	2025-05-27 10:51:08
List job events Wrapper job log

Jobscript log (last 10,000 characters)

30822	from gen 2 Ar39GenInLAr
# of particles 3473	from gen 3 Kr85GenInLAr
# of particles 3	from gen 4 Ar42GenInLAr
# of particles 0	from gen 5 K42From42ArGenInLAr
# of particles 31	from gen 6 Rn222ChainRn222GenInLAr
# of particles 26	from gen 7 Rn222ChainPo218GenInLAr
# of particles 57	from gen 8 Rn222ChainPb214GenInLAr
# of particles 42	from gen 9 Rn222ChainBi214GenInLAr
# of particles 28	from gen 10 Rn222ChainPb210GenInLAr
# of particles 26	from gen 11 Rn220ChainPb212GenInLAr
# of particles 244	from gen 12 K40GenInCathode
# of particles 2978	from gen 13 U238ChainGenInCathode
# of particles 3615	from gen 14 Th232ChainGenInCathode
# of particles 82	from gen 15 K40GenInAnode
# of particles 1429	from gen 16 U238ChainGenInAnode
# of particles 1693	from gen 17 Th232ChainGenInAnode
# of particles 1	from gen 18 Rn222ChainGenInPDS
# of particles 3	from gen 19 K42From42ArGenInUpperMesh
# of particles 2	from gen 20 Rn222ChainFromPo218GenInUpperMesh
# of particles 25	from gen 21 Rn222ChainFromPb214GenInUpperMesh
# of particles 32	from gen 22 Rn222ChainFromBi214GenInUpperMesh
# of particles 26	from gen 23 Rn222ChainFromPb210GenInUpperMesh
# of particles 43	from gen 24 Rn222ChainFromBi210GenInUpperMesh
# of particles 36	from gen 25 Rn220ChainFromPb212GenInUpperMesh
# of particles 27779	from gen 26 CavernwallGammasAtLAr
# of particles 3595	from gen 27 foamGammasAtLAr
# of particles 0	from gen 28 CavernwallNeutronsAtLAr
# of particles 0	from gen 29 CryostatNGammasAtLAr
# of particles 0	from gen 30 CavernNGammasAtLAr[0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 27-May-2025 10:14:16 UTC  run: 20000047 subRun: 0 event: 9
[33m
Number of Signal Particles: 8
Neutrino Interaction: -1
Neutrino Energy: 69.36 MeV
Position (94.27, 116.55, 217.40) cm
	Gen.	PdgCode		Energy		EndPosition		Mother
------------------------------------------------------------------------
marley	11		1.53	 (94.30, 116.72, 217.51)	76093
marley	-12		0.29	 (-3751.36, 5033.92, -2990.90)	76093
marley	1000200400	37214.71 (94.27, 116.55, 217.40)	76093
marley	22		0.03	 (94.32, 116.80, 217.29)	0
marley	22		1.61	 (73.00, 103.90, 242.81)	0
marley	22		0.65	 (100.67, 145.62, 208.52)	0
marley	22		2.09	 (68.48, 139.63, 184.37)	0
marley	1000190400	37216.65 (94.27, 116.55, 217.40)	0
marley	11		63.86	 (104.66, 100.80, 215.66)	0
Signal Track IDs: 76092; 76093; 76094; 76095; 76096; 76097; 76163; 76164; 76165; [0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 27-May-2025 10:14:16 UTC  run: 20000047 subRun: 0 event: 9
[34m[0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 27-May-2025 10:14:16 UTC  run: 20000047 subRun: 0 event: 9
[36m
# OpHits (ophit10ppm) in full geometry: 25165
# OpFlashes (solarflash) in full geometry: 0
# Hits (gaushit) in each view: 3399, 3363, 4130, 0
# Cluster from the hits: 1504, 1560, 3558, 0
# Tracks (pmtracktc) in full geometry: 206[0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 27-May-2025 10:32:33 UTC  run: 20000047 subRun: 0 event: 9
[32m*** Matched preselection cluster: 
 - MainTrackID 76092
 - Purity 1.00 Hits 5
 - #AdjCl 13 ( 0 signal )
 - Gen 1 -> marley
 - Reco Time,Y,Z (2692.34, 101.68, 214.83)
[0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 27-May-2025 10:32:33 UTC  run: 20000047 subRun: 0 event: 9
[32m*** Matched preselection cluster: 
 - MainTrackID 76092
 - Purity 1.00 Hits 6
 - #AdjCl 12 ( 0 signal )
 - Gen 1 -> marley
 - Reco Time,Y,Z (2776.18, 108.31, 216.05)
[0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 27-May-2025 10:32:33 UTC  run: 20000047 subRun: 0 event: 9
[32m*** Matched preselection cluster: 
 - MainTrackID 76096
 - Purity 1.00 Hits 2
 - #AdjCl 7 ( 0 signal )
 - Gen 1 -> marley
 - Reco Time,Y,Z (3118.41, 115.17, 237.51)
[0m
%MSG
Begin processing the 10th record. run: 20000047 subRun: 0 event: 10 at 27-May-2025 10:32:33 UTC
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 27-May-2025 10:32:36 UTC  run: 20000047 subRun: 0 event: 10
[35m
TPC Frequency in [MHz]: 2.00
TPC Tick in [us]: 0.50
Event Flag: 890598207
Succesfull reset of variables for evt 10
#########################################[0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 27-May-2025 10:32:36 UTC  run: 20000047 subRun: 0 event: 10
[91m
There are a total of 76548 Particles in the event

# of particles 10	from gen 1 marley
# of particles 31051	from gen 2 Ar39GenInLAr
# of particles 3492	from gen 3 Kr85GenInLAr
# of particles 1	from gen 4 Ar42GenInLAr
# of particles 0	from gen 5 K42From42ArGenInLAr
# of particles 36	from gen 6 Rn222ChainRn222GenInLAr
# of particles 24	from gen 7 Rn222ChainPo218GenInLAr
# of particles 56	from gen 8 Rn222ChainPb214GenInLAr
# of particles 51	from gen 9 Rn222ChainBi214GenInLAr
# of particles 14	from gen 10 Rn222ChainPb210GenInLAr
# of particles 39	from gen 11 Rn220ChainPb212GenInLAr
# of particles 236	from gen 12 K40GenInCathode
# of particles 3013	from gen 13 U238ChainGenInCathode
# of particles 3642	from gen 14 Th232ChainGenInCathode
# of particles 122	from gen 15 K40GenInAnode
# of particles 1307	from gen 16 U238ChainGenInAnode
# of particles 1673	from gen 17 Th232ChainGenInAnode
# of particles 1	from gen 18 Rn222ChainGenInPDS
# of particles 1	from gen 19 K42From42ArGenInUpperMesh
# of particles 3	from gen 20 Rn222ChainFromPo218GenInUpperMesh
# of particles 39	from gen 21 Rn222ChainFromPb214GenInUpperMesh
# of particles 54	from gen 22 Rn222ChainFromBi214GenInUpperMesh
# of particles 55	from gen 23 Rn222ChainFromPb210GenInUpperMesh
# of particles 34	from gen 24 Rn222ChainFromBi210GenInUpperMesh
# of particles 40	from gen 25 Rn220ChainFromPb212GenInUpperMesh
# of particles 28057	from gen 26 CavernwallGammasAtLAr
# of particles 3497	from gen 27 foamGammasAtLAr
# of particles 0	from gen 28 CavernwallNeutronsAtLAr
# of particles 0	from gen 29 CryostatNGammasAtLAr
# of particles 0	from gen 30 CavernNGammasAtLAr[0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 27-May-2025 10:32:36 UTC  run: 20000047 subRun: 0 event: 10
[33m
Number of Signal Particles: 6
Neutrino Interaction: -1
Neutrino Energy: 19.82 MeV
Position (84.26, -356.96, 1994.60) cm
	Gen.	PdgCode		Energy		EndPosition		Mother
------------------------------------------------------------------------
marley	22		1.46	 (85.47, -394.08, 1963.87)	76540
marley	1000180400	37215.54 (84.26, -356.96, 1994.60)	76540
marley	12		0.04	 (-1435.93, 5033.92, -1099.53)	76540
marley	11		0.51	 (84.26, -356.96, 1994.60)	76540
marley	11		0.51	 (84.26, -356.96, 1994.60)	76540
marley	11		0.51	 (84.26, -356.96, 1994.60)	76540
marley	22		2.42	 (74.11, -358.47, 1999.41)	0
marley	22		0.31	 (88.63, -341.51, 1986.67)	0
marley	1000190400	37216.54 (84.26, -356.96, 1994.60)	0
marley	11		16.09	 (81.10, -359.70, 1997.82)	0
Signal Track IDs: 76539; 76540; 76541; 76542; 76576; 76577; 76578; 76580; 76581; 76582; [0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 27-May-2025 10:32:37 UTC  run: 20000047 subRun: 0 event: 10
[34m[0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 27-May-2025 10:32:37 UTC  run: 20000047 subRun: 0 event: 10
[36m
# OpHits (ophit10ppm) in full geometry: 25197
# OpFlashes (solarflash) in full geometry: 0
# Hits (gaushit) in each view: 3240, 3157, 4133, 0
# Cluster from the hits: 1474, 1456, 3516, 0
# Tracks (pmtracktc) in full geometry: 210[0m
%MSG
%MSG-i SolarNuAna:  SolarNuAna:solarnuana@BeginModule 27-May-2025 10:50:54 UTC  run: 20000047 subRun: 0 event: 10
[32m*** Matched preselection cluster: 
 - MainTrackID 76539
 - Purity 1.00 Hits 11
 - #AdjCl 6 ( 0 signal )
 - Gen 1 -> marley
 - Reco Time,Y,Z (3030.68, -358.74, 1996.42)
[0m
%MSG
27-May-2025 10:50:54 UTC  Closed input file "root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/dune/persistent/staging/usertests/5d/2b/fdvd_lowBg_reco2_2025-05-12T_044628Z.root"

====================================================================================================================
TimeTracker printout (sec)            Min           Avg           Max         Median          RMS         nEvts   
====================================================================================================================
Full event                          577.236       1017.18       1108.92       1071.62       151.899        10     
--------------------------------------------------------------------------------------------------------------------
source:RootInput(read)            0.000711688    0.0144775     0.0492812     0.011317      0.0154893       10     
end_path:solarnuana:SolarNuAna      577.234       1017.16       1108.87       1071.6        151.892        10     
====================================================================================================================
%MSG-i NuRandomService:  SolarNuAna:solarnuana@EndJob 27-May-2025 10:50:54 UTC  ModuleEndJob

Summary of seeds computed by the NuRandomService
Random policy: 'perEvent'
  algorithm version: EventTimestamp_v1

%MSG

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 2006.76 MB
  Peak resident set size usage (VmHWM): 1376.86 MB
====================================================================================================

TrigReport ---------- Event summary -------------
TrigReport Events total = 10 passed = 10 failed = 0

TrigReport ---------- Modules in End-path ----------
TrigReport        Run    Success      Error Name
TrigReport         10         10          0 solarnuana

TimeReport ---------- Time summary [sec] -------
TimeReport CPU = 10115.843147 Real = 10202.412364

MemReport  ---------- Memory summary [base-10 MB] ------
MemReport  VmPeak = 2006.76 VmHWM = 1376.86

Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Output file: fdhd_ana_2025-05-27T_080038Z.root
Output hist file: fdhd_ana_2025-05-27T_080038Z_hist.root
lar exit code 0