Jobsub ID 102577.0@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	102577.0@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4134
Stage ID	1
User name	osbiston@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-19 01:01:06
Site	FR_CCIN2P3
Entry	DUNE_FR_CCIN2P3_cccondorce01
Last heartbeat	2024-11-19 04:59:20
From worker node	Hostname	ccwcondor0600
	cpuinfo	AMD EPYC 7453 28-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	106200 (29 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-19 01:02:03
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T224712Z_gen_003544_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	3h (14204s)
	CPU time	3h (13551s = 95%)
Outputting started	2024-11-19 04:58:48
Output files
Finished	2024-11-19 04:59:20
List job events Wrapper job log

Jobscript log (last 10,000 characters)

Input PFN = root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-vd/62/5e/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T224712Z_gen_003544_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
minwire 1: 0
minwire 2: 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 692, 1191
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0273147, 
Output 1: 0.0572034, 0.103224, 0.0450345, 0.794539, 
Output 2: 0.0790539, 0.292273, 0.59815, 0.0305227, 
Output 3: 0.828875, 0.148247, 0.0141583, 0.00871901, 
Output 4: 0.913499, 0.0808727, 0.00471819, 0.00091024, 
Output 5: 0.257614, 0.594428, 0.129013, 0.0189458, 
Output 6: 0.0544055, 0.356594, 0.196471, 0.39253, 

Begin processing the 8th record. run: 3799 subRun: 0 event: 35438 at 19-Nov-2024 04:46:30 CET
Boundary wire vector sizes: 6901, 6838, 6723
minwire 0: 49
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1005, 1504
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.105936, 
Output 1: 0.0718508, 0.0624353, 0.120327, 0.745387, 
Output 2: 0.0416907, 0.159614, 0.771884, 0.026811, 
Output 3: 0.964554, 0.0215185, 0.00413568, 0.00979152, 
Output 4: 0.582137, 0.363109, 0.0474764, 0.00727721, 
Output 5: 0.347263, 0.388222, 0.205263, 0.0592525, 
Output 6: 0.0396025, 0.0928009, 0.0918144, 0.775782, 

Begin processing the 9th record. run: 3799 subRun: 0 event: 35439 at 19-Nov-2024 05:10:52 CET
Boundary wire vector sizes: 6594, 6525, 6464
minwire 0: 36
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 564, 1063
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0996896, 
Output 1: 0.138742, 0.090919, 0.189466, 0.580873, 
Output 2: 0.0448572, 0.0961934, 0.846795, 0.0121545, 
Output 3: 0.973739, 0.0230622, 0.00111535, 0.00208379, 
Output 4: 0.463157, 0.489327, 0.0416027, 0.00591299, 
Output 5: 0.302836, 0.445678, 0.184411, 0.0670748, 
Output 6: 0.0916188, 0.200023, 0.12725, 0.581108, 

Begin processing the 10th record. run: 3799 subRun: 0 event: 35440 at 19-Nov-2024 05:34:45 CET
Boundary wire vector sizes: 6722, 6687, 6578
minwire 0: 24
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1488, 1987
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0919529, 
Output 1: 0.046074, 0.0533355, 0.160871, 0.73972, 
Output 2: 0.38921, 0.20898, 0.341107, 0.0607032, 
Output 3: 0.97134, 0.0260982, 0.000729202, 0.0018328, 
Output 4: 0.0898618, 0.887646, 0.021883, 0.000609334, 
Output 5: 0.951016, 0.0466701, 0.00144005, 0.000873708, 
Output 6: 0.0212016, 0.0782512, 0.078553, 0.821994, 

19-Nov-2024 05:58:46 CET  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T224712Z_gen_003544_supernova_g4stage1_g4stage2_detsim_reco_matt_2024-11-19T_010209Z.root"
19-Nov-2024 05:58:46 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-vd/62/5e/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T224712Z_gen_003544_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1350.37       1408.41       1457.76       1412.05       29.5583        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000816089   0.00795584     0.026432     0.00185055    0.00930193       10     
reco:pandora:StandardPandora                            1345.48       1405.99       1455.78       1409.98       30.1667        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.00026833    0.00448701     0.0405746    0.000319662    0.0120388       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000243815   0.000634751   0.00373993    0.00029054    0.00103565       10     
reco:pandoracalo:Calorimetry                          0.00177047     0.0308253     0.0414189     0.0326948     0.0110142       10     
reco:pandorapid:Chi2ParticleID                        6.1949e-05    0.000163219   0.000926863   7.77335e-05   0.000254867      10     
reco:cvnmap:CVNMapper                                  0.060675      0.0692409     0.0829802     0.0689349    0.00662447       10     
reco:cvneva:CVNEvaluator                                1.28237       1.69689       3.88514       1.45422      0.738724        10     
reco:energyrecnumu:EnergyReco                          0.0659084     0.100064      0.151345      0.106219      0.023704        10     
reco:energyrecnue:EnergyReco                          0.00909325     0.0124681     0.0209364     0.0102153    0.00437694       10     
reco:energyrecnc:EnergyReco                           0.00920116     0.013227      0.022453     0.00996454    0.00527571       10     
[art]:TriggerResults:TriggerResultInserter            1.9207e-05    3.1722e-05    0.000104552   2.2873e-05    2.45034e-05      10     
end_path:out1:RootOutput                              1.1733e-05    1.92342e-05   4.6654e-05    1.4387e-05    1.14369e-05      10     
end_path:out1:RootOutput(write)                        0.393545      0.482991      0.712993      0.452286      0.0954706       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19419.7 MB
  Peak resident set size usage (VmHWM): 2813.45 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0