justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 301669.103@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID301669.103@justin-prod-sched01.dune.hep.ac.uk
Workflow ID4213
Stage ID1
User nameismerio@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-11-23 03:17:34
SiteUK_Durham
EntryDUNE_UK_SGridDurham_ce3
Last heartbeat2024-11-23 05:34:11
From worker nodeHostnamen255.dur.scotgrid.ac.uk
cpuinfoAMD EPYC 7763 64-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-11-23 03:19:54
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207422_291_20231121T151116Z_gen_g4_detsim_hitreco__20240507T182225Z_reco2.root
JobscriptExit code0
Real time2h (8038s)
CPU time1h (7191s = 89%)
Outputting started2024-11-23 05:33:53
Output files
Finished2024-11-23 05:34:11
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

ure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:33:20 GMT run: 6207422 subRun: 1 event: 58400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:33:20 GMT run: 6207422 subRun: 1 event: 58400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:33:20 GMT run: 6207422 subRun: 1 event: 58400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:33:20 GMT run: 6207422 subRun: 1 event: 58400
1st wire C:0 T:11 P:2 W:1129 does not exist (max wire number: 480)
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:33:20 GMT run: 6207422 subRun: 1 event: 58400
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:33:20 GMT run: 6207422 subRun: 1 event: 58400
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:33:20 GMT run: 6207422 subRun: 1 event: 58400
TuneFullTree failed.
%MSG
Boundary wire vector sizes: 1273, 1370, 927
minwire 0: 640
minwire 1: 2081
minwire 2: 337
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0144954, 0.000561832, 0.984943, 
Output 1: 0.550535, 0.32897, 0.0847826, 0.0357122, 
Output 2: 0.576238, 0.423762, 

709
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2928!
23-Nov-2024 05:33:51 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207422_291_20231121T151116Z_gen_g4_detsim_hitreco__20240507T182225Z_reco2_reco_data_2024-11-23T_031959Z.root"
23-Nov-2024 05:33:51 GMT  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/7c/3d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207422_291_20231121T151116Z_gen_g4_detsim_hitreco__20240507T182225Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.12367       36.0961       625.288       10.8634       82.6688        200    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0181771     0.0386522     0.0629741     0.037795      0.0124327       200    
reco:linecluster:LineCluster                             0.164       0.458504       18.8358      0.195435       1.47454        200    
reco:trajcluster:TrajCluster                           0.0508777     0.863946       20.7761      0.0632514      3.01309        200    
reco:pandora:StandardPandora                           0.0621677      3.18962       170.497       1.30795       12.8441        200    
reco:pandoraTrack:LArPandoraTrackCreation             0.00015811     0.0083513     0.141192     0.00197651     0.0194963       200    
reco:pandoraShower:LArPandoraModularShowerCreation     0.0002215     0.0170017     0.461752     0.00352884     0.0423429       200    
reco:pandoracalo:Calorimetry                          0.000123394    0.0275938     0.879169     0.00138557     0.0976608       200    
reco:pandorapid:Chi2ParticleID                         4.772e-05     0.0002066    0.00166928    0.000163515   0.000177017      200    
reco:trkshowersplit:TrackShowerHits                   0.000109698    0.0331622     0.937316     0.000957054    0.112403        200    
reco:pmtrack:PMAlgTrackMaker                          0.00141667      3.36466       104.278      0.216196       10.9248        200    
reco:pmtrackcalo:Calorimetry                          8.5853e-05     0.0151939     0.367202     0.00123662     0.0499898       200    
reco:pmtrackpid:Chi2ParticleID                        3.4635e-05    0.000382371   0.00471081    0.000178924   0.000620724      200    
reco:pmtrajfit:PMAlgTrajFitter                        0.000214888     2.9878        83.8463      0.350039       9.36511        200    
reco:pmtrajfitcalo:Calorimetry                        8.0252e-05     0.0281493     0.870714      0.0019598     0.0967682       200    
reco:pmtrajfitpid:Chi2ParticleID                      3.3453e-05    0.000248755   0.00159855    0.000184034   0.000225581      200    
reco:pmtracktc:PMAlgTrackMaker                        0.00116577      4.37377       98.0766      0.253557       13.5732        200    
reco:pmtracktccalo:Calorimetry                        8.2005e-05    0.00985455     0.211757     0.00129262     0.0294534       200    
reco:pmtracktcpid:Chi2ParticleID                      3.4755e-05    0.000357061   0.00305928    0.000187891   0.000488406      200    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000192896     2.23984       52.6454      0.365433       6.14507        200    
reco:pmtrajfittccalo:Calorimetry                      8.0051e-05     0.0225544     0.656466      0.001471      0.078496        200    
reco:pmtrajfittcpid:Chi2ParticleID                    3.3954e-05    0.000499008   0.00555121    0.000201842   0.000819823      200    
reco:blurredcluster:BlurredClustering                  7.94e-05       1.1693        11.8887       0.48349       1.64684        200    
reco:emtrkmichelid:EmTrackMichelId                     0.0312638      13.5249       177.516       3.28008       27.4506        200    
reco:emshower:EMShower                                0.000306101    0.0311554     0.412437     0.00600323     0.0565118       200    
reco:cvnmap:CVNMapper                                 2.5488e-05     0.0121668     0.0460007     0.0115811     0.0103525       200    
reco:cvneva:CVNEvaluator                              2.0479e-05     0.656504       1.14643      0.888948      0.416817        200    
reco:energyrecnumu:EnergyReco                         0.000398796    0.0529477      2.21467     0.00524626     0.215886        200    
reco:energyrecnue:EnergyReco                          0.000183728   0.00266059     0.0450387    0.000641797   0.00585073       200    
reco:energyrecnc:EnergyReco                           0.00253259    0.00527782     0.0497674    0.00329261    0.00619538       200    
reco:energyrecnumurange:EnergyReco                    0.000186984    0.050361       2.20772     0.00290756     0.214815        200    
reco:energyrecnumumcs:EnergyReco                      0.000977523    0.0521994      2.1995      0.00384266     0.214156        200    
reco:opdec:Deconvolution                               0.0836925     0.299223      0.639104      0.295084      0.103914        200    
reco:ophitspe:OpHitFinderDeco                          0.345806       2.26987       4.53628       2.24371       0.9382         200    
reco:opflash:OpFlashFinder                            0.000393206   0.00171249    0.00706671    0.00166516    0.00080034       200    
reco:rns:RandomNumberSaver                            2.3244e-05    4.50625e-05   0.00057965    4.3843e-05    3.94123e-05      200    
reco:anglereconue:NuAngularReco                       0.000181344   0.00209295     0.0379902    0.000502362   0.00500753       200    
reco:anglereconumu:NuAngularReco                      9.5852e-05    0.00181613     0.0369405    0.000276048   0.00482075       200    
reco:anglereconuepfps:NuAngularReco                   0.00014746    0.00197429     0.0377275    0.000434464   0.00484137       200    
reco:anglereconumupfps:NuAngularReco                  0.000133293   0.00188419     0.0378793    0.000350033    0.0048454       200    
[art]:TriggerResults:TriggerResultInserter            1.2433e-05    1.63493e-05   6.4131e-05     1.556e-05     4.292e-06       200    
end_path:out1:RootOutput                               3.827e-06    7.37793e-06    2.658e-05    7.6095e-06    2.84698e-06      200    
end_path:out1:RootOutput(write)                        0.018878      0.277126       1.24146      0.272801      0.209504        200    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 38570.5 MB
  Peak resident set size usage (VmHWM): 2502.64 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207422_291_20231121T151116Z_gen_g4_detsim_hitreco__20240507T182225Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207422_291_20231121T151116Z_gen_g4_detsim_hitreco__20240507T182225Z_reco2_reco_2024-11-23T_031959Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207422_291_20231121T151116Z_gen_g4_detsim_hitreco__20240507T182225Z_reco2_reco_data_2024-11-23T_031959Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work
justIN time: 2024-11-23 10:31:09 UTC       justIN version: 01.01.09