justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 301435.76@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID301435.76@justin-prod-sched01.dune.hep.ac.uk
Workflow ID4213
Stage ID1
User nameismerio@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-11-22 22:01:05
SiteES_PIC
EntryDUNE_T1_ES_PIC_ce15-multicore
Last heartbeat2024-11-22 22:54:26
From worker nodeHostnametds211.pic.es
cpuinfoIntel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit216000 (60 hours)
Inner Apptainer?True
Job statefinished
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-11-22 22:07:45
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_495_20231203T041432Z_gen_g4_detsim_hitreco__20240508T071847Z_reco2.root
JobscriptExit code0
Real time39m (2348s)
CPU time30m (1830s = 77%)
Outputting started2024-11-22 22:46:54
Output fileshttps://fndcadoor.fnal.gov:2880/dune/scratch/users/ismerio/04213/1/Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_495_20231203T041432Z_gen_g4_detsim_hitreco__20240508T071847Z_reco2.root
Finished2024-11-22 22:54:26
Saved logsjustin-logs:301435.76-justin-prod-sched01.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

le  22-Nov-2024 23:46:23 CET run: 66006611 subRun: 1 event: 49600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 23:46:24 CET run: 66006611 subRun: 1 event: 49600
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 23:46:24 CET run: 66006611 subRun: 1 event: 49600
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 23:46:24 CET run: 66006611 subRun: 1 event: 49600
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 23:46:24 CET run: 66006611 subRun: 1 event: 49600
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 23:46:24 CET run: 66006611 subRun: 1 event: 49600
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 23:46:24 CET run: 66006611 subRun: 1 event: 49600
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  22-Nov-2024 23:46:48 CET run: 66006611 subRun: 1 event: 49600
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 1084, 1242, 806
minwire 0: 483
minwire 1: 2053
minwire 2: 36
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0326959, 0.926888, 0.0404165, 
Output 1: 0.216642, 0.360291, 0.258586, 0.16448, 
Output 2: 0.118049, 0.881951, 

670
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2337!
22-Nov-2024 23:46:52 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_495_20231203T041432Z_gen_g4_detsim_hitreco__20240508T071847Z_reco2_reco_data_2024-11-22T_220748Z.root"
22-Nov-2024 23:46:53 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/98/ca/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_495_20231203T041432Z_gen_g4_detsim_hitreco__20240508T071847Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.37152       18.2742       149.881       8.11094       26.1533        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0465688     0.0809005     0.138723      0.0759892     0.0242654       100    
reco:linecluster:LineCluster                           0.267873      0.508068       2.7329       0.450582      0.266338        100    
reco:trajcluster:TrajCluster                           0.134145      0.401491       5.7232       0.139099       0.84696        100    
reco:pandora:StandardPandora                           0.144233       1.7037        9.94539       1.44918       1.12579        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000136774   0.00342247     0.0438033    0.000773431   0.00647685       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.00018494    0.00748683     0.0888884    0.00209019     0.0136071       100    
reco:pandoracalo:Calorimetry                          0.000109263   0.00761064     0.204991     0.000497632    0.0240003       100    
reco:pandorapid:Chi2ParticleID                        4.5477e-05    0.000131031   0.000676541   0.000107852   9.09493e-05      100    
reco:trkshowersplit:TrackShowerHits                   7.1774e-05     0.0079807     0.174946     0.000369727    0.0245397       100    
reco:pmtrack:PMAlgTrackMaker                          0.00120531      1.18052       13.9789      0.100732       2.64668        100    
reco:pmtrackcalo:Calorimetry                          8.4693e-05    0.00442944     0.0866151    0.000579204    0.0114009       100    
reco:pmtrackpid:Chi2ParticleID                        3.5082e-05    0.000208977   0.00116683    0.000122477   0.00024386       100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000200791     1.10655       12.4122      0.120599       2.32732        100    
reco:pmtrajfitcalo:Calorimetry                        8.0039e-05    0.00817579     0.208777     0.000667492    0.0247301       100    
reco:pmtrajfitpid:Chi2ParticleID                      3.5484e-05    0.000156513   0.00132303    0.000113081   0.000154385      100    
reco:pmtracktc:PMAlgTrackMaker                         0.0010878      1.35861       17.2365      0.0900403      3.20176        100    
reco:pmtracktccalo:Calorimetry                        8.3643e-05     0.0035422     0.0552299    0.00049472    0.00755045       100    
reco:pmtracktcpid:Chi2ParticleID                      3.5745e-05    0.00019345    0.00109767    0.000117283   0.000197833      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000190019    0.895551       10.3524      0.105483       1.75564        100    
reco:pmtrajfittccalo:Calorimetry                      7.8631e-05    0.00697994     0.137529     0.000625124    0.0195625       100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.5644e-05    0.000259285   0.00194864    0.000125553   0.000342496      100    
reco:blurredcluster:BlurredClustering                 8.6888e-05     0.915109       7.38771      0.532401       1.17118        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000125516     7.4381        73.7315       1.80908       12.9135        100    
reco:emshower:EMShower                                0.000190303    0.024143      0.261594     0.00318995     0.0484731       100    
reco:cvnmap:CVNMapper                                 2.4583e-05     0.0102299     0.0447983     0.0108478     0.0100986       100    
reco:cvneva:CVNEvaluator                              2.0149e-05     0.450243      0.844463      0.699158      0.361158        100    
reco:energyrecnumu:EnergyReco                         0.00028935     0.0255826     0.904188     0.00320307     0.099676        100    
reco:energyrecnue:EnergyReco                          0.000167436   0.00145423     0.0148862    0.000554456   0.00233252       100    
reco:energyrecnc:EnergyReco                           0.00216811    0.00338789     0.0186342    0.00247007    0.00251345       100    
reco:energyrecnumurange:EnergyReco                    0.000185519    0.0241514     0.890078     0.00232726     0.0985743       100    
reco:energyrecnumumcs:EnergyReco                      0.000837482    0.0260939     0.902229     0.00290455     0.101696        100    
reco:opdec:Deconvolution                               0.153059      0.456038      0.772306      0.461951      0.122161        100    
reco:ophitspe:OpHitFinderDeco                          0.402541       1.42053       2.46458       1.41278      0.507849        100    
reco:opflash:OpFlashFinder                            0.000126282   0.00119913    0.00366225    0.00115035    0.000606294      100    
reco:rns:RandomNumberSaver                            1.9279e-05    3.18988e-05   0.000266594   2.74185e-05   2.4795e-05       100    
reco:anglereconue:NuAngularReco                       0.000148861   0.00083883     0.0105838    0.00026423    0.00164186       100    
reco:anglereconumu:NuAngularReco                       8.796e-05    0.00070847     0.0101316    0.00015602    0.00159104       100    
reco:anglereconuepfps:NuAngularReco                   0.000128183   0.000835085    0.0102276    0.000279202   0.00160587       100    
reco:anglereconumupfps:NuAngularReco                  0.000118879   0.000775649    0.0100808    0.000230367   0.00159995       100    
[art]:TriggerResults:TriggerResultInserter            1.0775e-05    1.42669e-05   5.4795e-05    1.27815e-05   4.8403e-06       100    
end_path:out1:RootOutput                               2.714e-06    5.28973e-06   2.0128e-05     4.23e-06     2.98234e-06      100    
end_path:out1:RootOutput(write)                       0.00915373     0.187069      0.577875      0.217225      0.129975        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8288.15 MB
  Peak resident set size usage (VmHWM): 2197.41 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_495_20231203T041432Z_gen_g4_detsim_hitreco__20240508T071847Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_495_20231203T041432Z_gen_g4_detsim_hitreco__20240508T071847Z_reco2_reco_2024-11-22T_220748Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_495_20231203T041432Z_gen_g4_detsim_hitreco__20240508T071847Z_reco2_reco_data_2024-11-22T_220748Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work
justIN time: 2024-11-23 12:08:33 UTC       justIN version: 01.01.09