justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 299042.16@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID299042.16@justin-prod-sched01.dune.hep.ac.uk
Workflow ID4199
Stage ID1
User nameosbiston@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-11-21 15:59:37
SiteUK_RAL-PPD
EntryCMSHTPC_T2_UK_SGrid_RALPP_hep206
Last heartbeat2024-11-21 21:29:41
From worker nodeHostnameheplnc161.pp.rl.ac.uk
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes2097152000 (2000 MiB)
Wall seconds limit257400 (71 hours)
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-11-21 16:57:44
Input filesfardet-vd:prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T205205Z_gen_000671_supernova_g4stage1_g4stage2_detsim_reco.root
JobscriptExit code0
Real time4h (16290s)
CPU time4h (15581s = 95%)
Outputting started2024-11-21 21:29:14
Output files
Finished2024-11-21 21:29:41
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

Input PFN = root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-vd/de/4a/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T205205Z_gen_000671_supernova_g4stage1_g4stage2_detsim_reco.root
Setting up larsoft UPS area... /cvmfs/larsoft.opensciencegrid.org
Setting up DUNE UPS area... /cvmfs/dune.opensciencegrid.org/products/dune/
/cvmfs/larsoft.opensciencegrid.org/products/xrootd/v5_5_5a/Linux64bit+3.10-2.17-e26-p3915-prof/lib/libXrdPosixPreload.so
=== Start last 100 lines of lar log file ===
Boundary wire vector sizes: 6862, 6824, 6680
minwire 0: 37
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.101671, 
Output 1: 0.0729824, 0.113006, 0.105619, 0.708393, 
Output 2: 0.0619904, 0.279655, 0.640894, 0.0174602, 
Output 3: 0.953025, 0.0419646, 0.00218558, 0.00282501, 
Output 4: 0.433796, 0.501701, 0.0593503, 0.00515273, 
Output 5: 0.375708, 0.546429, 0.0697372, 0.00812594, 
Output 6: 0.0277464, 0.0945772, 0.106014, 0.771663, 

Begin processing the 8th record. run: 3799 subRun: 0 event: 6708 at 21-Nov-2024 20:07:05 GMT
Boundary wire vector sizes: 6536, 6473, 6609
minwire 0: 45
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 230, 729
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.124398, 
Output 1: 0.0644266, 0.0433793, 0.109677, 0.782517, 
Output 2: 0.0937496, 0.250382, 0.619975, 0.035893, 
Output 3: 0.974885, 0.017203, 0.00176977, 0.00614187, 
Output 4: 0.264909, 0.681765, 0.0502541, 0.0030716, 
Output 5: 0.59723, 0.343321, 0.0485455, 0.0109037, 
Output 6: 0.0160009, 0.046685, 0.0645623, 0.872752, 

Begin processing the 9th record. run: 3799 subRun: 0 event: 6709 at 21-Nov-2024 20:33:25 GMT
Boundary wire vector sizes: 6570, 6484, 6617
minwire 0: 29
minwire 1: 0
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0636829, 
Output 1: 0.448425, 0.0850126, 0.148809, 0.317753, 
Output 2: 0.0775997, 0.0701288, 0.844219, 0.00805245, 
Output 3: 0.970531, 0.0281503, 0.000473051, 0.000845211, 
Output 4: 0.798657, 0.190016, 0.00870595, 0.00262039, 
Output 5: 0.224909, 0.577751, 0.140123, 0.0572158, 
Output 6: 0.291933, 0.406826, 0.115571, 0.18567, 

Begin processing the 10th record. run: 3799 subRun: 0 event: 6710 at 21-Nov-2024 21:01:19 GMT
Boundary wire vector sizes: 6624, 6520, 6593
minwire 0: 24
minwire 1: 0
minwire 2: 3
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Used alternate method to get min and max wires due to vertex determination failure: 2294, 2793
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 499
Classifier summary: 
Output 0: 0.0883511, 
Output 1: 0.0508441, 0.0696445, 0.125204, 0.754308, 
Output 2: 0.041473, 0.1767, 0.749887, 0.0319398, 
Output 3: 0.951541, 0.0289299, 0.0072976, 0.0122318, 
Output 4: 0.674418, 0.266906, 0.0494456, 0.00923014, 
Output 5: 0.595283, 0.260631, 0.110073, 0.0340133, 
Output 6: 0.0251474, 0.0583857, 0.0656695, 0.850797, 

21-Nov-2024 21:29:13 GMT  Closed output file "prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T205205Z_gen_000671_supernova_g4stage1_g4stage2_detsim_reco_matt_2024-11-21T_165752Z.root"
21-Nov-2024 21:29:13 GMT  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-vd/de/4a/prodmarley_nue_cc_flat_radiological_decay0_dunevd10kt_1x8x14_3view_30deg_20241023T205205Z_gen_000671_supernova_g4stage1_g4stage2_detsim_reco.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1548.96       1615.02        1669         1616.69       40.1605        10     
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000995407   0.00438685     0.0184075    0.00136461    0.00619389       10     
reco:pandora:StandardPandora                            1542.71       1612.52       1666.63       1614.51       40.8015        10     
reco:pandoraTrack:LArPandoraTrackCreation             0.000360796   0.00527499     0.0459112    0.000411361    0.0135762       10     
reco:pandoraShower:LArPandoraModularShowerCreation    0.000335479   0.00105829    0.00694913    0.000412794   0.00196401       10     
reco:pandoracalo:Calorimetry                          0.00311217     0.0438858     0.0721335     0.0438163     0.0175588       10     
reco:pandorapid:Chi2ParticleID                        8.9237e-05    0.00023674    0.00141239    9.7558e-05    0.000392378      10     
reco:cvnmap:CVNMapper                                  0.0458088     0.0605664     0.0886463     0.0592339     0.0129458       10     
reco:cvneva:CVNEvaluator                                1.15306       1.72281       5.26008       1.35698       1.18515        10     
reco:energyrecnumu:EnergyReco                          0.111433      0.177027      0.253674      0.166581      0.0454539       10     
reco:energyrecnue:EnergyReco                           0.0107343     0.0117035     0.012595      0.0116733    0.000472941      10     
reco:energyrecnc:EnergyReco                            0.0110365     0.0115877     0.0121651     0.0116102    0.000331623      10     
[art]:TriggerResults:TriggerResultInserter            2.1541e-05    4.39558e-05   0.000190397   2.6901e-05    4.90634e-05      10     
end_path:out1:RootOutput                              1.2493e-05    1.82212e-05   5.5974e-05    1.40415e-05   1.26268e-05      10     
end_path:out1:RootOutput(write)                        0.321076      0.463378      0.549547      0.466814      0.0539689       10     
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 38738.3 MB
  Peak resident set size usage (VmHWM): 2698.18 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
lar exit code 0
justIN time: 2024-11-21 22:28:40 UTC       justIN version: 01.01.09