justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 301697.93@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID301697.93@justin-prod-sched01.dune.hep.ac.uk
Workflow ID4213
Stage ID1
User nameismerio@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-11-23 04:11:36
SiteUK_Durham
EntryDUNE_UK_SGridDurham_ce3
Last heartbeat2024-11-23 05:49:24
From worker nodeHostnamen251.dur.scotgrid.ac.uk
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-11-23 04:12:40
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_1171_20231203T070741Z_gen_g4_detsim_hitreco__20240508T042449Z_reco2.root
JobscriptExit code0
Real time1h (5788s)
CPU time1h (5238s = 90%)
Outputting started2024-11-23 05:49:09
Output files
Finished2024-11-23 05:49:24
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:47:57 GMT run: 6407968 subRun: 1 event: 117200
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:47:57 GMT run: 6407968 subRun: 1 event: 117200
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:47:57 GMT run: 6407968 subRun: 1 event: 117200
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:47:57 GMT run: 6407968 subRun: 1 event: 117200
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:47:57 GMT run: 6407968 subRun: 1 event: 117200
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:47:57 GMT run: 6407968 subRun: 1 event: 117200
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:47:57 GMT run: 6407968 subRun: 1 event: 117200
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:47:57 GMT run: 6407968 subRun: 1 event: 117200
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 2699, 2812, 2762
minwire 0: 22
minwire 1: 1657
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0486209, 0.0661826, 0.885197, 
Output 1: 0.311842, 0.365474, 0.213191, 0.109493, 
Output 2: 0.263392, 0.736608, 

668
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2383!
23-Nov-2024 05:49:07 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_1171_20231203T070741Z_gen_g4_detsim_hitreco__20240508T042449Z_reco2_reco_data_2024-11-23T_041247Z.root"
23-Nov-2024 05:49:07 GMT  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/70/54/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_1171_20231203T070741Z_gen_g4_detsim_hitreco__20240508T042449Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.654501       52.2058       933.321       9.87506       131.332        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                  0.0178       0.0402562     0.0686148     0.0383471     0.0133125       100    
reco:linecluster:LineCluster                           0.101803       1.25364       43.5272      0.190212       5.68085        100    
reco:trajcluster:TrajCluster                           0.0510917      2.92006       114.531      0.0704326      13.239         100    
reco:pandora:StandardPandora                           0.0697023      5.03492       106.993       1.25061       15.7383        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000131981    0.0112111     0.194954      0.0019465     0.0281909       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000198897    0.0119684       0.167      0.00385384     0.0228592       100    
reco:pandoracalo:Calorimetry                           9.437e-05     0.0535367      1.22606      0.0017181     0.181625        100    
reco:pandorapid:Chi2ParticleID                        3.7511e-05    0.000202189   0.00104943    0.000152851   0.000186853      100    
reco:trkshowersplit:TrackShowerHits                   9.0743e-05     0.0719831      2.17286     0.00158754     0.267134        100    
reco:pmtrack:PMAlgTrackMaker                          0.000983677     5.64487       150.838       0.25521       18.4779        100    
reco:pmtrackcalo:Calorimetry                          5.8902e-05     0.0261556     0.681194     0.00123514     0.0879889       100    
reco:pmtrackpid:Chi2ParticleID                        2.5468e-05    0.000474615   0.00470647    0.000189445   0.000790739      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000166997     5.74697       162.877      0.366568       20.4645        100    
reco:pmtrajfitcalo:Calorimetry                        5.6738e-05     0.0558438      1.21995     0.00203087     0.185121        100    
reco:pmtrajfitpid:Chi2ParticleID                      2.4757e-05    0.000249927   0.00119144    0.000165259   0.000240523      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000790491     7.15916       158.208      0.411645       21.6097        100    
reco:pmtracktccalo:Calorimetry                        5.7589e-05     0.0131708     0.254145     0.00108647     0.0380917       100    
reco:pmtracktcpid:Chi2ParticleID                      2.3625e-05    0.000398543   0.00268202    0.000178423   0.000570075      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000151116     2.8663        44.6584      0.379957       6.96516        100    
reco:pmtrajfittccalo:Calorimetry                      5.4353e-05     0.0328176     0.647735     0.00163483     0.098553        100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.3244e-05    0.000561164   0.00494063    0.000190792   0.000912995      100    
reco:blurredcluster:BlurredClustering                  9.975e-05      1.21744       9.66179       0.50521       1.84795        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000168199     16.6972       177.241       3.39693       33.3205        100    
reco:emshower:EMShower                                0.000226931    0.0324656     0.276932      0.0116426     0.0538343       100    
reco:cvnmap:CVNMapper                                 2.8614e-05     0.011505      0.0359145    0.00890076     0.0100114       100    
reco:cvneva:CVNEvaluator                              1.8105e-05     0.576123       1.16752       0.61886      0.363044        100    
reco:energyrecnumu:EnergyReco                          0.0003954     0.0843688      2.45922     0.00468425     0.306883        100    
reco:energyrecnue:EnergyReco                          0.000136519    0.0031051     0.0432667    0.000557919   0.00727913       100    
reco:energyrecnc:EnergyReco                           0.00153774    0.00546586     0.0522905    0.00302949    0.00794061       100    
reco:energyrecnumurange:EnergyReco                    0.000194219    0.081694       2.46757      0.0030122     0.304748        100    
reco:energyrecnumumcs:EnergyReco                      0.00072692     0.0835688      2.47834      0.0040905     0.305866        100    
reco:opdec:Deconvolution                               0.058678      0.291996      0.512426      0.287559      0.0983993       100    
reco:ophitspe:OpHitFinderDeco                          0.224133       1.8966        4.28056       1.57612      0.955234        100    
reco:opflash:OpFlashFinder                            0.000420648   0.00151976    0.00504254    0.00135825    0.000752682      100    
reco:rns:RandomNumberSaver                             2.121e-05    4.68822e-05   0.000441338   3.96105e-05   4.23706e-05      100    
reco:anglereconue:NuAngularReco                       0.000180573   0.00257738     0.0382699    0.000561891   0.00614468       100    
reco:anglereconumu:NuAngularReco                      7.4241e-05    0.00227199     0.0380257    0.000273404   0.00593141       100    
reco:anglereconuepfps:NuAngularReco                   0.000127953    0.0024425     0.0377912    0.000436989   0.00591764       100    
reco:anglereconumupfps:NuAngularReco                   9.995e-05    0.00236373     0.0366067    0.000332952   0.00600425       100    
[art]:TriggerResults:TriggerResultInserter             8.927e-06    1.42007e-05   6.1457e-05     1.28e-05     5.95104e-06      100    
end_path:out1:RootOutput                               2.825e-06    7.85213e-06   3.2953e-05     7.489e-06    4.25198e-06      100    
end_path:out1:RootOutput(write)                        0.0142099     0.266159       1.10817      0.236566      0.221568        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 38525.8 MB
  Peak resident set size usage (VmHWM): 2463.56 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_1171_20231203T070741Z_gen_g4_detsim_hitreco__20240508T042449Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_1171_20231203T070741Z_gen_g4_detsim_hitreco__20240508T042449Z_reco2_reco_2024-11-23T_041247Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_1171_20231203T070741Z_gen_g4_detsim_hitreco__20240508T042449Z_reco2_reco_data_2024-11-23T_041247Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work
justIN time: 2024-11-27 02:48:47 UTC       justIN version: 01.01.09