Jobsub ID 207352.1@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	207352.1@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	7089
Stage ID	1
User name	ichong@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	GPU	No
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2025-05-21 17:56:44
Site	UK_Oxford
Entry	DUNE_UK_SGrid_Oxford_arc01
Last heartbeat	2025-05-21 18:52:29
From worker node	Hostname	t2wn067.physics.ox.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	GPU
	Inner Apptainer?	True
Job state	finished
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2025-05-21 18:43:23
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492237_611_20231208T060355Z_gen_g4_detsim_hitreco__20240510T051449Z_reco2.root
Jobscript	Exit code	0
	Real time	8m (486s)
	CPU time	3m (215s = 44%)
	Max RSS bytes	1880846336 (1793 MiB)
Outputting started	2025-05-21 18:51:30
Output files	https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/07089/1/001/graph_output_2025-05-21T_184336Z_5_training1_CaloHitListU_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/07089/1/001/graph_output_2025-05-21T_184336Z_5_training1_CaloHitListV_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/07089/1/001/graph_output_2025-05-21T_184336Z_5_training1_CaloHitListW_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/07089/1/001/graph_output_2025-05-21T_184336Z_5_training2_CaloHitListU_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/07089/1/001/graph_output_2025-05-21T_184336Z_5_training2_CaloHitListV_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/07089/1/001/graph_output_2025-05-21T_184336Z_5_training2_CaloHitListW_graph.data https://fndcadoor.fnal.gov:2880/dune/scratch/users/ichong/07089/1/001/graph_output_2025-05-21T_184336Z_5_analysiseid.root
Finished	2025-05-21 18:52:29
Saved logs	justin-logs:207352.1-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events Wrapper job log

Jobscript log (last 10,000 characters)

se-10 MB] ------
MemReport  VmPeak = 36001.2 VmHWM = 576.573

Art has completed and will exit with status 0.
=== End last 100 lines of secondary lar log file ===
Renamed training1_CaloHitListU_graph.data -> graph_output_2025-05-21T_184336Z_5_training1_CaloHitListU_graph.data
Renamed training1_CaloHitListV_graph.data -> graph_output_2025-05-21T_184336Z_5_training1_CaloHitListV_graph.data
Renamed training1_CaloHitListW_graph.data -> graph_output_2025-05-21T_184336Z_5_training1_CaloHitListW_graph.data
Renamed training2_CaloHitListU_graph.data -> graph_output_2025-05-21T_184336Z_5_training2_CaloHitListU_graph.data
Renamed training2_CaloHitListV_graph.data -> graph_output_2025-05-21T_184336Z_5_training2_CaloHitListV_graph.data
Renamed training2_CaloHitListW_graph.data -> graph_output_2025-05-21T_184336Z_5_training2_CaloHitListW_graph.data
Renamed analysiseid.root -> graph_output_2025-05-21T_184336Z_5_analysiseid.root
=== Start last 100 lines of lar log file ===
--Primary 1, MCPDG -211, Energy 1.24485, Dist. 63.5978, nMCHits 320 (112, 87, 121)
MCPDG -211, Energy 1.24485, Dist. 63.5978, nMCHits 314 (110, 85, 119)
\_ MCPDG 1000010020, Energy 1.892, Dist. 0.192143, nMCHits 5 (1, 2, 2)
\_ MCPDG 2212, Energy 0.952271, Dist. 0.242222, nMCHits 1 (1, 0, 0)

--Primary 2, MCPDG 211, Energy 0.282263, Dist. 43.2719, nMCHits 277 (86, 92, 99)
MCPDG 211, Energy 0.282263, Dist. 43.2719, nMCHits 227 (66, 76, 85)
   \_ MCPDG -11, Energy 0.0322109, Dist. 9.03856, nMCHits 50 (20, 16, 14)

--Primary 3, MCPDG 2212, Energy 1.67864, Dist. 47.4354, nMCHits 176 (30, 80, 66)
MCPDG 2212, Energy 1.67864, Dist. 47.4354, nMCHits 162 (26, 75, 61)
\_ MCPDG 2212, Energy 0.996237, Dist. 2.9944, nMCHits 9 (3, 3, 3)
\_ MCPDG 2212, Energy 0.949411, Dist. 0.16087, nMCHits 4 (1, 1, 2)
\_ MCPDG 22, Energy 0.000123707, Dist. 0.387514, nMCHits 1 (0, 1, 0)

--Primary 4, MCPDG 2212, Energy 1.02874, Dist. 6.54874, nMCHits 32 (6, 13, 13)
MCPDG 2212, Energy 1.02874, Dist. 6.54874, nMCHits 32 (6, 13, 13)
------------------------------------------------------------------------------------------------
Operating in training mode.
The eid is 19
Graph saved to training1_CaloHitListW_graph.data
Size of file training1_CaloHitListW_graph.data is 154364 bytes.
The eid is 19
Graph saved to training1_CaloHitListU_graph.data
Size of file training1_CaloHitListU_graph.data is 194260 bytes.
The eid is 19
Graph saved to training1_CaloHitListV_graph.data
Size of file training1_CaloHitListV_graph.data is 181276 bytes.
Operating in inference mode.
Operating in training mode.
The eid is -1
Graph saved to training2_CaloHitListW_graph.data
Size of file training2_CaloHitListW_graph.data is 154364 bytes.
The eid is -1
Graph saved to training2_CaloHitListU_graph.data
Size of file training2_CaloHitListU_graph.data is 194260 bytes.
The eid is -1
Graph saved to training2_CaloHitListV_graph.data
Size of file training2_CaloHitListV_graph.data is 181276 bytes.
Boundary wire vector sizes: 676, 628, 672
minwire 0: 1878
minwire 1: 17
minwire 2: 2207
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2379, 2878
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.148684, 
Output 1: 0.00322284, 0.0330504, 0.166588, 0.797139, 
Output 2: 0.820679, 0.057933, 0.0876463, 0.0337421, 
Output 3: 0.951242, 0.0474494, 0.000542152, 0.000766624, 
Output 4: 0.213785, 0.714969, 0.0554939, 0.0157524, 
Output 5: 0.461404, 0.53498, 0.00312192, 0.000493531, 
Output 6: 0.0381791, 0.0888526, 0.0915503, 0.781418, 


Running Ophitfinder with InputDigiType = 'recob'
Found hits: 169!
21-May-2025 19:49:56 BST  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492237_611_20231208T060355Z_gen_g4_detsim_hitreco__20240510T051449Z_reco2_graph_2025-05-21T_184336Z.root"
21-May-2025 19:49:56 BST  Closed input file "root://se1.farm.particle.cz:1094//dune/RSE/fardet-hd/2d/40/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492237_611_20231208T060355Z_gen_g4_detsim_hitreco__20240510T051449Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              9.74085       13.359        16.7199       13.0731       1.87409        20     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0248267     0.0364152     0.0521103     0.0292727     0.0116404       20     
reco:gaushit:GausHitFinder                             0.0925517     0.147743      0.322185      0.118652      0.0607091       20     
reco:spsolve:SpacePointSolver                         0.000201948    0.0330068     0.188644     0.00842551     0.0525716       20     
reco:hitfd:DisambigFromSpacePoints                    0.000405902   0.00660052     0.0312674     0.0018321     0.0095864       20     
reco:rns:RandomNumberSaver                            2.9726e-05    7.53791e-05   0.000573547   4.88715e-05   0.000115047      20     
reco:pandora:StandardPandora                            1.13787       1.57309       2.27201       1.48531      0.333942        20     
reco:pandoraTrack:LArPandoraTrackCreation             0.000229241   0.000740527   0.00705605    0.000345088    0.0014638       20     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000272672   0.000777675   0.00796484    0.000395973    0.001651        20     
reco:pandoracalo:Calorimetry                          0.000161162   0.000430784   0.00348221    0.000274951   0.00070276       20     
reco:pandorapid:Chi2ParticleID                        5.7107e-05    0.000179292   0.00193327    8.24895e-05   0.000403105      20     
reco:cvnmap:CVNMapper                                 4.5826e-05     0.042599      0.0734436     0.047291      0.0204447       20     
reco:cvneva:CVNEvaluator                               3.167e-05      1.42811        4.168        1.42214      0.856259        20     
reco:energyrecnumu:EnergyReco                         0.00339344    0.00631304     0.0123673    0.00562358     0.0026327       20     
reco:energyrecnue:EnergyReco                          0.00288884    0.00390591    0.00588632    0.00366856    0.000727504      20     
reco:energyrecnc:EnergyReco                           0.00312207    0.00387495    0.00558672    0.00358304    0.000710561      20     
reco:energyrecnumurange:EnergyReco                    0.00302397    0.00384533    0.00562635    0.00367172    0.000623513      20     
reco:energyrecnumumcs:EnergyReco                       0.0029777    0.00380089    0.00539249    0.00361537    0.000593746      20     
reco:opdec:Deconvolution                               0.206747      0.349808      0.531792      0.358232      0.0811672       20     
reco:ophitspe:OpHitFinderDeco                          0.211547      0.219722      0.229313       0.21946     0.00422909       20     
reco:opflash:OpFlashFinder                            0.000443693   0.000934512   0.00607906    0.000622924   0.00118858       20     
reco:opslicer:OpSlicer                                0.000393619    0.018383      0.0730878     0.0117924     0.0190725       20     
[art]:TriggerResults:TriggerResultInserter            2.1231e-05    4.85507e-05   0.000210405   3.59925e-05   4.16804e-05      20     
end_path:out1:RootOutput                               4.348e-06    8.5002e-06     3.183e-05    6.2415e-06    6.0924e-06       20     
end_path:out1:RootOutput(write)                         7.94594       9.47664       11.7291        9.136        1.05686        20     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 37877.7 MB
  Peak resident set size usage (VmHWM): 1880.85 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
=== Generated output files ===
207352.1_justin-prod-sched02.dune.hep.ac.uk.logs.tgz
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492237_611_20231208T060355Z_gen_g4_detsim_hitreco__20240510T051449Z_reco2_graph_2025-05-21T_184336Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492237_611_20231208T060355Z_gen_g4_detsim_hitreco__20240510T051449Z_reco2_graph_2025-05-21T_184336Z.root
debugprod.log
graph_output_2025-05-21T_184336Z_5_analysiseid.root
graph_output_2025-05-21T_184336Z_5_training1_CaloHitListU_graph.data
graph_output_2025-05-21T_184336Z_5_training1_CaloHitListV_graph.data
graph_output_2025-05-21T_184336Z_5_training1_CaloHitListW_graph.data
graph_output_2025-05-21T_184336Z_5_training2_CaloHitListU_graph.data
graph_output_2025-05-21T_184336Z_5_training2_CaloHitListV_graph.data
graph_output_2025-05-21T_184336Z_5_training2_CaloHitListW_graph.data
jobscript.log
justin-processed-pfns.txt
reco_hist.root
secondary_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6492237_611_20231208T060355Z_gen_g4_detsim_hitreco__20240510T051449Z_reco2_graph_2025-05-21T_184336Z.log
training1_CaloHitListU.csv
training1_CaloHitListV.csv
training1_CaloHitListW.csv
training2_CaloHitListU.csv
training2_CaloHitListV.csv
training2_CaloHitListW.csv