Jobsub ID 104392.9@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	104392.9@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4134
Stage ID	1
User name	osbiston@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-19 16:18:25
Site	CZ_FZU
Entry	Nova_CZ_FZU_osgce1
Last heartbeat	2024-11-20 00:00:22
From worker node	Hostname	mikan35.farm.particle.cz
	cpuinfo	Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	2097152000 (2000 MiB)
	Wall seconds limit	172800 (48 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-19 16:39:34
Input files	fardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241025T185255Z_gen_000494_supernova_g4stage1_g4stage2_detsim_reco.root
Jobscript	Exit code	0
	Real time	7h (26426s)
	CPU time	7h (25894s = 97%)
Outputting started	2024-11-20 00:00:00
Output files
Finished	2024-11-20 00:00:22
List job events Wrapper job log

Jobscript log (last 10,000 characters)

Input PFN = root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-vd/da/5e/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241025T185255Z_gen_000494_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
Boundary wire vector sizes: 6587, 6591, 6613
minwire 0: 47
minwire 1: 0
minwire 2: 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.131873, 
Output 1: 0.0638239, 0.0403104, 0.144388, 0.751478, 
Output 2: 0.124831, 0.209044, 0.623004, 0.0431219, 
Output 3: 0.977865, 0.0141697, 0.00180443, 0.0061611, 
Output 4: 0.270275, 0.674182, 0.0514622, 0.00408056, 
Output 5: 0.656014, 0.301434, 0.0336997, 0.00885285, 
Output 6: 0.0213814, 0.0635286, 0.0728287, 0.842261, 

Begin processing the 8th record. run: 3844 subRun: 0 event: 4938 at 19-Nov-2024 22:59:45 CET
Boundary wire vector sizes: 6668, 6590, 6600
minwire 0: 17
minwire 1: 0
minwire 2: 1
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1922, 2421
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.11621, 
Output 1: 0.087398, 0.112822, 0.112726, 0.687054, 
Output 2: 0.1172, 0.274356, 0.57951, 0.0289337, 
Output 3: 0.977079, 0.0200621, 0.000790618, 0.00206821, 
Output 4: 0.327483, 0.619136, 0.0498462, 0.00353565, 
Output 5: 0.693743, 0.268529, 0.0293798, 0.00834754, 
Output 6: 0.039876, 0.100199, 0.102252, 0.757673, 

Begin processing the 9th record. run: 3844 subRun: 0 event: 4939 at 19-Nov-2024 23:40:32 CET
Boundary wire vector sizes: 6759, 6796, 6806
minwire 0: 40
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.139896, 
Output 1: 0.0545911, 0.139443, 0.177453, 0.628513, 
Output 2: 0.0409218, 0.223581, 0.709095, 0.0264022, 
Output 3: 0.960484, 0.0260163, 0.00492328, 0.00857664, 
Output 4: 0.371951, 0.54763, 0.0742595, 0.00615926, 
Output 5: 0.425229, 0.432615, 0.120689, 0.0214679, 
Output 6: 0.0229542, 0.0589912, 0.0734662, 0.844588, 

Begin processing the 10th record. run: 3844 subRun: 0 event: 4940 at 20-Nov-2024 00:20:03 CET
Boundary wire vector sizes: 6895, 6792, 6776
minwire 0: 22
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 1420, 1919
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.106557, 
Output 1: 0.0401266, 0.0862066, 0.21355, 0.660117, 
Output 2: 0.0891744, 0.183267, 0.689541, 0.0380172, 
Output 3: 0.937895, 0.0537898, 0.00470974, 0.00360537, 
Output 4: 0.375589, 0.57309, 0.0477347, 0.00358577, 
Output 5: 0.535851, 0.356417, 0.0862516, 0.0214806, 
Output 6: 0.0407487, 0.0942419, 0.0839253, 0.781084, 

20-Nov-2024 00:59:59 CET  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241025T185255Z_gen_000494_supernova_g4stage1_g4stage2_detsim_reco_matt_2024-11-19T_163940Z.root"
20-Nov-2024 00:59:59 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-vd/da/5e/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241025T185255Z_gen_000494_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2247.48       2624.53       2987.89       2533.16       273.907        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000784467   0.00133949    0.00240345    0.00117461    0.000470613      10     
reco:pandora:StandardPandora                            2244.3        2621.2        2985.6        2529.75       274.288        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000299434   0.00475354     0.0343067    0.000798523   0.00994831       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000287089   0.000940886   0.00555603    0.000422606   0.00154131       10     
reco:pandoracalo:Calorimetry                          0.00272571     0.0409942     0.0631999     0.0409382     0.0176005       10     
reco:pandorapid:Chi2ParticleID                        7.6185e-05    0.000241618   0.00139488    0.000106697   0.000385466      10     
reco:cvnmap:CVNMapper                                  0.0812095     0.122566      0.287675      0.0983563     0.0598062       10     
reco:cvneva:CVNEvaluator                                1.42803       2.33525       4.98881       2.13483      0.938436        10     
reco:energyrecnumu:EnergyReco                          0.104615      0.121736      0.137909      0.121054      0.0126002       10     
reco:energyrecnue:EnergyReco                           0.0130282     0.0138107     0.014554      0.0139309    0.00057537       10     
reco:energyrecnc:EnergyReco                            0.0129282     0.0137211     0.0142645     0.013904     0.000509072      10     
[art]:TriggerResults:TriggerResultInserter            2.5886e-05    4.00493e-05   0.000124601   2.77775e-05   2.89558e-05      10     
end_path:out1:RootOutput                               8.849e-06    1.48548e-05   3.8069e-05    1.11495e-05   8.48259e-06      10     
end_path:out1:RootOutput(write)                        0.438453      0.676941      0.742327      0.700781      0.0831787       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 10808.3 MB
  Peak resident set size usage (VmHWM): 2780.52 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0