justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 301667.68@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID301667.68@justin-prod-sched01.dune.hep.ac.uk
Workflow ID4213
Stage ID1
User nameismerio@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-11-23 03:14:21
SiteUK_Lancaster
EntryUBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat2024-11-23 05:34:47
From worker nodeHostnamecomp21-10
cpuinfoIntel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-11-23 03:17:33
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1498_203_20231027T230707Z_gen_g4_detsim_hitreco__20240503T044758Z_reco2.root
JobscriptExit code0
Real time2h (8219s)
CPU time2h (7464s = 90%)
Outputting started2024-11-23 05:34:32
Output files
Finished2024-11-23 05:34:47
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

ring two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:34:13 GMT run: 1498 subRun: 1 event: 40800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:34:13 GMT run: 1498 subRun: 1 event: 40800
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:34:13 GMT run: 1498 subRun: 1 event: 40800
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:34:13 GMT run: 1498 subRun: 1 event: 40800
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:34:13 GMT run: 1498 subRun: 1 event: 40800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:34:13 GMT run: 1498 subRun: 1 event: 40800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:34:13 GMT run: 1498 subRun: 1 event: 40800
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 471, 612, 438
minwire 0: 731
minwire 1: 2039
minwire 2: 461
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 6.89816e-05, 0.999927, 3.96349e-06, 
Output 1: 0.871121, 0.12825, 0.000600864, 2.77705e-05, 
Output 2: 0.998932, 0.00106753, 

552
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1945!
23-Nov-2024 05:34:31 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1498_203_20231027T230707Z_gen_g4_detsim_hitreco__20240503T044758Z_reco2_reco_data_2024-11-23T_031736Z.root"
23-Nov-2024 05:34:31 GMT  Closed input file "root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/c3/7e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1498_203_20231027T230707Z_gen_g4_detsim_hitreco__20240503T044758Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.703891       37.8958       1128.92       11.8555       110.572        200    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0115932     0.0919077     0.295098      0.0725419     0.0540593       200    
reco:linecluster:LineCluster                           0.0879116     0.550683       11.647       0.311452       1.16964        200    
reco:trajcluster:TrajCluster                           0.0307687      1.52237       80.9514      0.102681       7.34575        200    
reco:pandora:StandardPandora                           0.0652483      4.99603       445.696       1.25102       34.4652        200    
reco:pandoraTrack:LArPandoraTrackCreation             0.000140677   0.00924137     0.376766     0.00172079     0.0337068       200    
reco:pandoraShower:LArPandoraModularShowerCreation    0.00018564     0.0122781      0.14366      0.0027942     0.025096        200    
reco:pandoracalo:Calorimetry                          9.9551e-05     0.042714       2.80673     0.00119082     0.273006        200    
reco:pandorapid:Chi2ParticleID                        3.8146e-05    0.00016277    0.00142476    0.000124801    0.0001501       200    
reco:trkshowersplit:TrackShowerHits                   0.000126311    0.0250426      1.02446     0.000860652    0.107259        200    
reco:pmtrack:PMAlgTrackMaker                          0.00128139      2.64306       91.7742      0.241546       9.41089        200    
reco:pmtrackcalo:Calorimetry                          8.3002e-05     0.0168469      0.87173     0.000929796    0.0855742       200    
reco:pmtrackpid:Chi2ParticleID                        3.0875e-05    0.000291029   0.00391621    0.00014177    0.000484477      200    
reco:pmtrajfit:PMAlgTrajFitter                        0.000197537     2.6608        111.922      0.311343       10.4184        200    
reco:pmtrajfitcalo:Calorimetry                        6.7141e-05     0.042942       2.8291       0.001446      0.269224        200    
reco:pmtrajfitpid:Chi2ParticleID                      3.0042e-05    0.000199004   0.00190335    0.000142638   0.000213377      200    
reco:pmtracktc:PMAlgTrackMaker                        0.00101538      3.50683       130.517      0.222034       12.5328        200    
reco:pmtracktccalo:Calorimetry                        6.9329e-05     0.0106288     0.481455     0.000921798    0.0449503       200    
reco:pmtracktcpid:Chi2ParticleID                      2.8051e-05    0.000269389   0.00324966    0.000149169   0.000387132      200    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000179113     1.78719       55.6094      0.291429       5.69182        200    
reco:pmtrajfittccalo:Calorimetry                      6.7176e-05     0.024839       1.19898     0.00132173     0.121173        200    
reco:pmtrajfittcpid:Chi2ParticleID                    2.7396e-05    0.000372081   0.00501627     0.0001676    0.000626857      200    
reco:blurredcluster:BlurredClustering                 7.0678e-05      1.24383       12.4615      0.489332       1.91472        200    
reco:emtrkmichelid:EmTrackMichelId                    0.000104209     14.9527       260.451       3.80245       32.191         200    
reco:emshower:EMShower                                0.000159459    0.0259901     0.320612     0.00494016     0.0463055       200    
reco:cvnmap:CVNMapper                                 1.9793e-05     0.0135744     0.0542648     0.013856      0.0104265       200    
reco:cvneva:CVNEvaluator                              1.5358e-05     0.595543       1.13052      0.801998      0.373894        200    
reco:energyrecnumu:EnergyReco                         0.000277546    0.0696009      3.48494     0.00548984     0.296662        200    
reco:energyrecnue:EnergyReco                          0.000136149   0.00292131      0.12682     0.000524431    0.0112109       200    
reco:energyrecnc:EnergyReco                            0.0017259    0.00473976     0.129242     0.00231516     0.0113304       200    
reco:energyrecnumurange:EnergyReco                    0.000175309    0.0670798      3.47664     0.00211296     0.296956        200    
reco:energyrecnumumcs:EnergyReco                      0.00169968     0.0687785      3.64545     0.00320292     0.305728        200    
reco:opdec:Deconvolution                               0.0993064     0.397257       2.26842      0.378768      0.212998        200    
reco:ophitspe:OpHitFinderDeco                          0.102309       2.27356       4.55331       2.3569       0.990328        200    
reco:opflash:OpFlashFinder                            0.000272786   0.00137709    0.00420465    0.00131332    0.000623541      200    
reco:rns:RandomNumberSaver                            1.8616e-05    3.91062e-05    0.000687     2.9743e-05    6.32249e-05      200    
reco:anglereconue:NuAngularReco                       0.000156926   0.00248223      0.12135     0.000390689    0.0103935       200    
reco:anglereconumu:NuAngularReco                      7.2938e-05    0.00215156     0.110831     0.00022628    0.00974682       200    
reco:anglereconuepfps:NuAngularReco                   0.000122855    0.0022843     0.112268     0.000319094   0.00980955       200    
reco:anglereconumupfps:NuAngularReco                  0.000101209   0.00222673      0.11086     0.000302713   0.00975683       200    
[art]:TriggerResults:TriggerResultInserter             8.884e-06    1.17685e-05   4.3929e-05    1.06315e-05   3.49629e-06      200    
end_path:out1:RootOutput                               2.588e-06    5.5604e-06    2.0125e-05     4.118e-06    2.99748e-06      200    
end_path:out1:RootOutput(write)                        0.0122474     0.223264       1.27016      0.217134      0.174564        200    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 10418.1 MB
  Peak resident set size usage (VmHWM): 3125.05 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1498_203_20231027T230707Z_gen_g4_detsim_hitreco__20240503T044758Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1498_203_20231027T230707Z_gen_g4_detsim_hitreco__20240503T044758Z_reco2_reco_2024-11-23T_031736Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1498_203_20231027T230707Z_gen_g4_detsim_hitreco__20240503T044758Z_reco2_reco_data_2024-11-23T_031736Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work
justIN time: 2024-11-23 11:15:23 UTC       justIN version: 01.01.09