Jobsub ID 297429.138@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	297429.138@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4146
Stage ID	1
User name	osbiston@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-19 21:21:38
Site	FR_CCIN2P3
Entry	DUNE_FR_CCIN2P3_cccondorce04
Last heartbeat	2024-11-20 00:59:10
From worker node	Hostname	ccwcondor0593
	cpuinfo	AMD EPYC 7453 28-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	106200 (29 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-19 21:22:30
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T193306Z_gen_000220_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	3h (12976s)
	CPU time	3h (12508s = 96%)
Outputting started	2024-11-20 00:58:47
Output files
Finished	2024-11-20 00:59:10
List job events Wrapper job log

Jobscript log (last 10,000 characters)

Input PFN = root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-vd/4c/9c/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T193306Z_gen_000220_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
minwire 0: 13
minwire 1: 0
minwire 2: 2
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 941, 1440
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.230611, 
Output 1: 0.0845082, 0.13727, 0.151569, 0.626653, 
Output 2: 0.0522396, 0.239668, 0.671746, 0.0363465, 
Output 3: 0.983411, 0.00956612, 0.00175003, 0.00527294, 
Output 4: 0.492809, 0.428566, 0.0700688, 0.00855698, 
Output 5: 0.284639, 0.471895, 0.200854, 0.0426118, 
Output 6: 0.0418249, 0.0892723, 0.0998716, 0.769031, 

Begin processing the 8th record. run: 3799 subRun: 0 event: 2198 at 20-Nov-2024 00:56:03 CET
Boundary wire vector sizes: 6752, 6701, 6759
minwire 0: 25
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2374, 2873
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0530207, 
Output 1: 0.0379918, 0.0274044, 0.0430412, 0.891563, 
Output 2: 0.0993024, 0.23446, 0.600173, 0.0660651, 
Output 3: 0.943142, 0.0121344, 0.00510808, 0.0396154, 
Output 4: 0.92465, 0.0633221, 0.00843509, 0.00359248, 
Output 5: 0.872919, 0.101331, 0.0175391, 0.00820994, 
Output 6: 0.00621127, 0.017135, 0.0293446, 0.947309, 

Begin processing the 9th record. run: 3799 subRun: 0 event: 2199 at 20-Nov-2024 01:17:43 CET
Boundary wire vector sizes: 6681, 6620, 6645
minwire 0: 42
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.113778, 
Output 1: 0.0610653, 0.115692, 0.186912, 0.63633, 
Output 2: 0.0584017, 0.185177, 0.734989, 0.0214318, 
Output 3: 0.941343, 0.0498321, 0.00426326, 0.00456138, 
Output 4: 0.312597, 0.614099, 0.0674974, 0.00580717, 
Output 5: 0.361114, 0.498788, 0.117957, 0.0221406, 
Output 6: 0.0446674, 0.123142, 0.112393, 0.719798, 

Begin processing the 10th record. run: 3799 subRun: 0 event: 2200 at 20-Nov-2024 01:37:23 CET
Boundary wire vector sizes: 6491, 6509, 6612
minwire 0: 70
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 716, 1215
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.102012, 
Output 1: 0.240901, 0.0795249, 0.200396, 0.479178, 
Output 2: 0.047016, 0.108259, 0.832075, 0.0126499, 
Output 3: 0.954366, 0.0410426, 0.00206419, 0.00252691, 
Output 4: 0.474362, 0.461197, 0.0550538, 0.00938753, 
Output 5: 0.37675, 0.420975, 0.149483, 0.0527908, 
Output 6: 0.122483, 0.238701, 0.139392, 0.499424, 

20-Nov-2024 01:58:46 CET  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T193306Z_gen_000220_supernova_g4stage1_g4stage2_detsim_reco_matt_2024-11-19T_212242Z.root"
20-Nov-2024 01:58:46 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-vd/4c/9c/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T193306Z_gen_000220_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1176.78       1282.94       1329.27       1294.41       44.3316        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000922036    0.0044133     0.0195363    0.00138834    0.00570719       10     
reco:pandora:StandardPandora                            1175.09       1280.9        1327.43       1292.4        44.1446        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000228248   0.00431235     0.0387394    0.000302015    0.0114886       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000226624   0.000828571   0.00538449    0.00032068    0.00152063       10     
reco:pandoracalo:Calorimetry                          0.000274845    0.0232734     0.040542      0.0275707     0.0122543       10     
reco:pandorapid:Chi2ParticleID                        5.1546e-05    0.000209549   0.00136091    7.8687e-05    0.00038431       10     
reco:cvnmap:CVNMapper                                  0.0448736     0.0673442     0.164759      0.0589226     0.0328081       10     
reco:cvneva:CVNEvaluator                                1.04386       1.39028       3.1263        1.21515      0.585299        10     
reco:energyrecnumu:EnergyReco                          0.063304      0.0723894     0.0871613     0.0702029    0.00813484       10     
reco:energyrecnue:EnergyReco                          0.00895325    0.00955335     0.0113176    0.00941734    0.000652423      10     
reco:energyrecnc:EnergyReco                           0.00899026    0.00955083     0.0110899    0.00939217    0.000601355      10     
[art]:TriggerResults:TriggerResultInserter            1.8364e-05    3.39303e-05   0.000117339   2.1761e-05    2.91581e-05      10     
end_path:out1:RootOutput                               7.945e-06    1.45073e-05   3.8993e-05    1.2769e-05    8.61701e-06      10     
end_path:out1:RootOutput(write)                        0.414623      0.459452      0.568708      0.441081      0.0489853       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 19420.5 MB
  Peak resident set size usage (VmHWM): 2742.66 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0