justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 301666.63@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID301666.63@justin-prod-sched01.dune.hep.ac.uk
Workflow ID4213
Stage ID1
User nameismerio@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-11-23 03:09:01
SiteUK_Durham
EntryDUNE_UK_SGridDurham_ce4
Last heartbeat2024-11-23 04:17:40
From worker nodeHostnamen255.dur.scotgrid.ac.uk
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-11-23 03:12:07
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_967_20231203T154456Z_gen_g4_detsim_hitreco__20240508T030108Z_reco2.root
JobscriptExit code0
Real time1h (3913s)
CPU time57m (3446s = 88%)
Outputting started2024-11-23 04:17:20
Output files
Finished2024-11-23 04:17:40
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

-Nov-2024 04:17:10 GMT run: 74511302 subRun: 1 event: 96800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 04:17:10 GMT run: 74511302 subRun: 1 event: 96800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 04:17:10 GMT run: 74511302 subRun: 1 event: 96800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e pma::Track3D:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 04:17:10 GMT run: 74511302 subRun: 1 event: 96800
Flip, endpoint closer to vStart.
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:17:10 GMT run: 74511302 subRun: 1 event: 96800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:17:10 GMT run: 74511302 subRun: 1 event: 96800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:17:11 GMT run: 74511302 subRun: 1 event: 96800
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 167, 67, 89
minwire 0: 2143
minwire 1: 1241
minwire 2: 1989
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.00579305, 0.000117728, 0.994089, 
Output 1: 0.00542294, 0.993468, 0.00109578, 1.35088e-05, 
Output 2: 0.997108, 0.00289176, 

464
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1370!
23-Nov-2024 04:17:19 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_967_20231203T154456Z_gen_g4_detsim_hitreco__20240508T030108Z_reco2_reco_data_2024-11-23T_031213Z.root"
23-Nov-2024 04:17:19 GMT  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/13/76/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_967_20231203T154456Z_gen_g4_detsim_hitreco__20240508T030108Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.64706       34.3553       684.024       10.0685       92.1174        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0183141     0.0423794     0.078384      0.0418032     0.0135064       100    
reco:linecluster:LineCluster                           0.178416      0.554247       21.0245      0.194809       2.14648        100    
reco:trajcluster:TrajCluster                           0.0516567      1.33241       56.2314      0.0599194      6.45828        100    
reco:pandora:StandardPandora                            1.06929       3.1083        92.7092       1.37584       9.71291        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000186804   0.00746571     0.166859     0.00144344     0.0227689       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000248932    0.0111978     0.155059     0.00371955     0.0245668       100    
reco:pandoracalo:Calorimetry                          0.000135597    0.0336945      1.27509     0.00103656     0.158403        100    
reco:pandorapid:Chi2ParticleID                        5.2229e-05    0.000200133   0.000916868   0.000166762   0.000142295      100    
reco:trkshowersplit:TrackShowerHits                   0.000172667    0.0374059      1.2714      0.000768602    0.165433        100    
reco:pmtrack:PMAlgTrackMaker                          0.00151273      2.95917       73.6222      0.253014       10.3221        100    
reco:pmtrackcalo:Calorimetry                          8.6754e-05     0.0156433     0.569662     0.00102568     0.0687632       100    
reco:pmtrackpid:Chi2ParticleID                        3.5107e-05    0.000384145   0.00411119    0.000168058   0.000665619      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000243612     2.6285        96.1417      0.226313       10.6031        100    
reco:pmtrajfitcalo:Calorimetry                        8.7405e-05     0.0324307      1.1912      0.00125973     0.149701        100    
reco:pmtrajfitpid:Chi2ParticleID                      3.5608e-05    0.000241843   0.00129489    0.000183954   0.000203188      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00123057      3.96129       96.3048      0.252772       13.8824        100    
reco:pmtracktccalo:Calorimetry                        8.5392e-05    0.00931951     0.307674     0.00104245     0.0404496       100    
reco:pmtracktcpid:Chi2ParticleID                      3.4986e-05    0.000339474    0.0044798    0.000167387   0.000561063      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000224005     1.9587        55.8855      0.262351       6.69182        100    
reco:pmtrajfittccalo:Calorimetry                      8.3007e-05     0.0235008     0.949145     0.00116697     0.107943        100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.5618e-05    0.000464242   0.00658776    0.000184615   0.000886633      100    
reco:blurredcluster:BlurredClustering                 0.00191816      1.16253       10.8704      0.512854       1.67264        100    
reco:emtrkmichelid:EmTrackMichelId                     0.319235       12.9417       227.438       2.81348       32.6857        100    
reco:emshower:EMShower                                0.000373578    0.0335839     0.322532     0.00810719     0.0552498       100    
reco:cvnmap:CVNMapper                                  2.625e-05     0.0128904     0.0491795    0.00962817     0.0114252       100    
reco:cvneva:CVNEvaluator                              2.0359e-05     0.677295       1.12614      0.881711      0.408458        100    
reco:energyrecnumu:EnergyReco                         0.000471113    0.0285447     0.900548     0.00534887     0.108383        100    
reco:energyrecnue:EnergyReco                          0.000180483   0.00241289     0.0503116    0.000551566   0.00714233       100    
reco:energyrecnc:EnergyReco                           0.00184769    0.00480632     0.0555573    0.00321619    0.00681638       100    
reco:energyrecnumurange:EnergyReco                    0.000199659    0.0263719     0.923111     0.00295106     0.110786        100    
reco:energyrecnumumcs:EnergyReco                      0.00202451     0.0279767     0.931994     0.00357011     0.110453        100    
reco:opdec:Deconvolution                               0.120921      0.301464      0.549603      0.298213      0.0980697       100    
reco:ophitspe:OpHitFinderDeco                          0.531348       2.13836       4.26846       2.17577      0.874105        100    
reco:opflash:OpFlashFinder                            0.00049572    0.00188083     0.0176676    0.00159715    0.00180442       100    
reco:rns:RandomNumberSaver                            2.4185e-05    6.07037e-05   0.000560282   5.11665e-05   6.43432e-05      100    
reco:anglereconue:NuAngularReco                       0.000215629   0.00183132     0.042317     0.000481157   0.00557557       100    
reco:anglereconumu:NuAngularReco                      9.5361e-05    0.00151005     0.0429952    0.000253566   0.00547958       100    
reco:anglereconuepfps:NuAngularReco                   0.000155225   0.00173036     0.042578     0.000423137    0.0055434       100    
reco:anglereconumupfps:NuAngularReco                  0.000122873   0.00173404     0.042095     0.000326274   0.00577714       100    
[art]:TriggerResults:TriggerResultInserter            1.0119e-05    1.80525e-05   8.5482e-05    1.6787e-05    8.81358e-06      100    
end_path:out1:RootOutput                               3.356e-06    9.52681e-06   3.5497e-05    9.3575e-06    5.02934e-06      100    
end_path:out1:RootOutput(write)                        0.0295144     0.268607       1.39051      0.267483      0.204056        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 38508.8 MB
  Peak resident set size usage (VmHWM): 2444.03 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_967_20231203T154456Z_gen_g4_detsim_hitreco__20240508T030108Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_967_20231203T154456Z_gen_g4_detsim_hitreco__20240508T030108Z_reco2_reco_2024-11-23T_031213Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_967_20231203T154456Z_gen_g4_detsim_hitreco__20240508T030108Z_reco2_reco_data_2024-11-23T_031213Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work
justIN time: 2024-11-23 10:52:02 UTC       justIN version: 01.01.09