Jobsub ID 105927.11@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	105927.11@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4198
Stage ID	1
User name	osbiston@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-21 13:26:22
Site	CZ_FZU
Entry	Nova_CZ_FZU_osgce2
Last heartbeat	2024-11-21 21:11:18
From worker node	Hostname	mff-magic07.farm.particle.cz
	cpuinfo	Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	172800 (48 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-21 13:27:34
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T203208Z_gen_001676_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	7h (27801s)
	CPU time	7h (26785s = 96%)
Outputting started	2024-11-21 21:10:55
Output files
Finished	2024-11-21 21:11:18
List job events Wrapper job log

Jobscript log (last 10,000 characters)

Input PFN = root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-vd/4c/28/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T203208Z_gen_001676_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
Boundary wire vector sizes: 6788, 6848, 6673
minwire 0: 54
minwire 1: 0
minwire 2: 2
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 978, 1477
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0755444, 
Output 1: 0.0747608, 0.109732, 0.174314, 0.641193, 
Output 2: 0.0480672, 0.205468, 0.721907, 0.0245574, 
Output 3: 0.931658, 0.0485237, 0.00966388, 0.0101544, 
Output 4: 0.631957, 0.29345, 0.0628982, 0.0116955, 
Output 5: 0.755075, 0.20837, 0.03006, 0.00649525, 
Output 6: 0.0258098, 0.0679244, 0.0703577, 0.835908, 

Begin processing the 8th record. run: 3799 subRun: 0 event: 16758 at 21-Nov-2024 19:49:24 CET
Boundary wire vector sizes: 7018, 6859, 6729
minwire 0: 5
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1173, 1672
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0975416, 
Output 1: 0.0713937, 0.0399158, 0.107636, 0.781054, 
Output 2: 0.0804545, 0.251721, 0.62795, 0.0398738, 
Output 3: 0.963061, 0.0213936, 0.00430632, 0.0112392, 
Output 4: 0.606203, 0.343451, 0.044234, 0.00611191, 
Output 5: 0.707119, 0.254434, 0.032004, 0.00644259, 
Output 6: 0.0127758, 0.034172, 0.0483084, 0.904744, 

Begin processing the 9th record. run: 3799 subRun: 0 event: 16759 at 21-Nov-2024 20:34:10 CET
Boundary wire vector sizes: 6689, 6689, 6582
minwire 0: 35
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.120556, 
Output 1: 0.106265, 0.0845411, 0.131694, 0.6775, 
Output 2: 0.0456606, 0.178465, 0.744108, 0.0317668, 
Output 3: 0.974139, 0.00972597, 0.00241673, 0.0137185, 
Output 4: 0.619349, 0.345806, 0.0307349, 0.00410985, 
Output 5: 0.460346, 0.3484, 0.142424, 0.04883, 
Output 6: 0.0284054, 0.0553398, 0.0645279, 0.851727, 

Begin processing the 10th record. run: 3799 subRun: 0 event: 16760 at 21-Nov-2024 21:20:11 CET
Boundary wire vector sizes: 6978, 6932, 6752
minwire 0: 28
minwire 1: 0
minwire 2: 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.12212, 
Output 1: 0.106892, 0.12571, 0.169763, 0.597634, 
Output 2: 0.0314743, 0.188969, 0.746268, 0.0332881, 
Output 3: 0.950295, 0.0164173, 0.00677181, 0.0265158, 
Output 4: 0.605418, 0.355014, 0.0354404, 0.00412798, 
Output 5: 0.449596, 0.320304, 0.173519, 0.0565809, 
Output 6: 0.0240691, 0.0466341, 0.0590353, 0.870262, 

21-Nov-2024 22:10:54 CET  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T203208Z_gen_001676_supernova_g4stage1_g4stage2_detsim_reco_matt_2024-11-21T_132739Z.root"
21-Nov-2024 22:10:54 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-vd/4c/28/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T203208Z_gen_001676_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                               2666         2752.7        3033.81       2723.02       106.337        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000883323    0.0364375     0.338955     0.00121959     0.100887        10     
reco:pandora:StandardPandora                            2662.76       2748.92       3029.93       2719.5        106.122        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000352886   0.00633247     0.0521211    0.000913691    0.0152973       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000380556   0.00102886    0.00536141    0.000408978   0.00149128       10     
reco:pandoracalo:Calorimetry                          0.00268961     0.039692      0.0533966      0.04252      0.0130956       10     
reco:pandorapid:Chi2ParticleID                        9.4343e-05    0.000252263   0.00143379    0.000113755   0.000394716      10     
reco:cvnmap:CVNMapper                                  0.0820381     0.116481       0.23495      0.0992843     0.0425842       10     
reco:cvneva:CVNEvaluator                                2.20557       2.63001       5.50517       2.26501      0.965762        10     
reco:energyrecnumu:EnergyReco                          0.151595      0.182793      0.239491      0.180255      0.0248129       10     
reco:energyrecnue:EnergyReco                           0.0138006     0.016015      0.0229858     0.0145404    0.00298201       10     
reco:energyrecnc:EnergyReco                            0.0138017     0.0161019     0.023678      0.0146108     0.0030283       10     
[art]:TriggerResults:TriggerResultInserter            2.1234e-05    3.98702e-05   0.000107282   3.12745e-05   2.42241e-05      10     
end_path:out1:RootOutput                               7.415e-06    1.24792e-05   3.1323e-05    9.7595e-06    6.88473e-06      10     
end_path:out1:RootOutput(write)                        0.625951      0.736721      0.858168      0.702028      0.0846683       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9757.25 MB
  Peak resident set size usage (VmHWM): 2843.21 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0