Jobsub ID 108684.8@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108684.8@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 02:06:37
Site	UK_Lancaster
Entry	UBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat	2024-11-23 03:16:51
From worker node	Hostname	comp08-10
	cpuinfo	Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 02:23:01
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50573344_443_20231203T081117Z_gen_g4_detsim_hitreco__20240508T033256Z_reco2.root
Jobscript	Exit code	0
	Real time	53m (3215s)
	CPU time	48m (2881s = 89%)
Outputting started	2024-11-23 03:16:37
Output files
Finished	2024-11-23 03:16:51
List job events Wrapper job log
Jobscript log (last 10,000 characters)

G
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 03:16:15 GMT run: 50573344 subRun: 1 event: 44400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 03:16:15 GMT run: 50573344 subRun: 1 event: 44400
1st wire C:0 T:7 P:2 W:521 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 03:16:15 GMT run: 50573344 subRun: 1 event: 44400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 03:16:15 GMT run: 50573344 subRun: 1 event: 44400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 03:16:15 GMT run: 50573344 subRun: 1 event: 44400
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 03:16:15 GMT run: 50573344 subRun: 1 event: 44400
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 03:16:15 GMT run: 50573344 subRun: 1 event: 44400
TuneFullTree failed.
%MSG
Boundary wire vector sizes: 760, 708, 188
minwire 0: 736
minwire 1: 1895
minwire 2: 555
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0402602, 0.000865382, 0.958874, 
Output 1: 0.0331581, 0.189047, 0.501691, 0.276105, 
Output 2: 0.900576, 0.0994236, 

699
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2683!
23-Nov-2024 03:16:36 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50573344_443_20231203T081117Z_gen_g4_detsim_hitreco__20240508T033256Z_reco2_reco_data_2024-11-23T_022305Z.root"
23-Nov-2024 03:16:36 GMT  Closed input file "root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/03/4c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50573344_443_20231203T081117Z_gen_g4_detsim_hitreco__20240508T033256Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.03827       29.2207       321.388       11.4522       56.1078        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0129847     0.135942       2.13416      0.087622      0.215431        100    
reco:linecluster:LineCluster                           0.127069      0.418152       2.8061       0.316074      0.397556        100    
reco:trajcluster:TrajCluster                           0.0317281      1.07329       36.4921      0.127485       4.37316        100    
reco:pandora:StandardPandora                           0.0848104      2.11747       20.9342       1.41896       3.02995        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.00014725    0.00824244     0.144305     0.00182615     0.0232228       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000193154     0.01366      0.205713     0.00318191     0.0287753       100    
reco:pandoracalo:Calorimetry                          0.000119596    0.0150448     0.316071     0.00120065     0.0512067       100    
reco:pandorapid:Chi2ParticleID                        4.1836e-05    0.000186262   0.00164925    0.000135661   0.000191071      100    
reco:trkshowersplit:TrackShowerHits                   0.000125392    0.0181135     0.426666     0.000668656    0.0653408       100    
reco:pmtrack:PMAlgTrackMaker                          0.00130206      1.82456       31.276       0.198259       5.17694        100    
reco:pmtrackcalo:Calorimetry                           9.118e-05     0.0112158     0.234227     0.00094412     0.0387149       100    
reco:pmtrackpid:Chi2ParticleID                         3.61e-05     0.000261071   0.00223942    0.000143408   0.000390853      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000217645     1.64787       29.1303      0.310946       4.51899        100    
reco:pmtrajfitcalo:Calorimetry                        7.1732e-05     0.0159375     0.340077     0.00141222     0.0536004       100    
reco:pmtrajfitpid:Chi2ParticleID                      3.1337e-05    0.000193026   0.00098846    0.00015053    0.000178115      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00118477      2.32072       52.6719      0.196737       7.46342        100    
reco:pmtracktccalo:Calorimetry                        8.3519e-05    0.00768467     0.142718     0.00114638     0.0224092       100    
reco:pmtracktcpid:Chi2ParticleID                      3.2763e-05    0.00024466    0.00184003    0.000156613   0.000312573      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000193707     1.2631        18.4376      0.299851       3.04859        100    
reco:pmtrajfittccalo:Calorimetry                      6.9704e-05     0.0150545     0.365488     0.00126013     0.0538494       100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.8916e-05    0.000318785   0.00296273    0.000171443   0.000488348      100    
reco:blurredcluster:BlurredClustering                  6.946e-05      1.17792       10.007       0.516428       1.67902        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000111874     13.3441       159.254       3.54181       26.8294        100    
reco:emshower:EMShower                                0.000186269    0.0252736     0.347486     0.00307143     0.0528242       100    
reco:cvnmap:CVNMapper                                 1.9472e-05     0.0172158     0.0538425     0.0189084     0.0122714       100    
reco:cvneva:CVNEvaluator                              1.6123e-05     0.636917      0.949391      0.837069      0.379135        100    
reco:energyrecnumu:EnergyReco                         0.000356112    0.0323715     0.876734     0.00613905     0.0972314       100    
reco:energyrecnue:EnergyReco                          0.000153772    0.0028701     0.0538562    0.000703629   0.00838747       100    
reco:energyrecnc:EnergyReco                           0.00202293    0.00464077     0.056506     0.00248236    0.00845067       100    
reco:energyrecnumurange:EnergyReco                    0.000310876    0.029383      0.863804     0.00248574     0.0960767       100    
reco:energyrecnumumcs:EnergyReco                      0.00156825     0.0305334     0.866417      0.0032181     0.096424        100    
reco:opdec:Deconvolution                               0.107181       0.3928        0.73302      0.368222      0.140853        100    
reco:ophitspe:OpHitFinderDeco                          0.207483       2.37768       4.1625        2.46917      0.970233        100    
reco:opflash:OpFlashFinder                            0.000158494   0.00146799    0.00526616    0.00145043    0.000703893      100    
reco:rns:RandomNumberSaver                            2.2338e-05    4.09393e-05   0.000326617   3.9251e-05    3.06259e-05      100    
reco:anglereconue:NuAngularReco                       0.000168594   0.00235932     0.0511107    0.000419477   0.00778951       100    
reco:anglereconumu:NuAngularReco                      7.4078e-05     0.0020877     0.047395     0.000224523   0.00749567       100    
reco:anglereconuepfps:NuAngularReco                   0.000128503   0.00220827     0.0473405    0.00035347    0.00739631       100    
reco:anglereconumupfps:NuAngularReco                  0.000106336   0.00217364     0.049283     0.000301584   0.00752663       100    
[art]:TriggerResults:TriggerResultInserter             9.529e-06    1.2875e-05    4.5924e-05    1.2346e-05    4.00266e-06      100    
end_path:out1:RootOutput                               2.593e-06    5.79318e-06   1.9304e-05     6.306e-06    2.52222e-06      100    
end_path:out1:RootOutput(write)                       0.00708697     0.229387      0.878121      0.224619      0.155736        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9747.07 MB
  Peak resident set size usage (VmHWM): 2585.65 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50573344_443_20231203T081117Z_gen_g4_detsim_hitreco__20240508T033256Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50573344_443_20231203T081117Z_gen_g4_detsim_hitreco__20240508T033256Z_reco2_reco_2024-11-23T_022305Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50573344_443_20231203T081117Z_gen_g4_detsim_hitreco__20240508T033256Z_reco2_reco_data_2024-11-23T_022305Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work