Jobsub ID 301692.62@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301692.62@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:05:14
Site	UK_Lancaster
Entry	UBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat	2024-11-23 04:51:50
From worker node	Hostname	comp19-03
	cpuinfo	Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:07:32
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_750_20231203T065716Z_gen_g4_detsim_hitreco__20240508T075924Z_reco2.root
Jobscript	Exit code	0
	Real time	44m (2645s)
	CPU time	40m (2427s = 91%)
Outputting started	2024-11-23 04:51:39
Output files
Finished	2024-11-23 04:51:50
List job events Wrapper job log
Jobscript log (last 10,000 characters)

t2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:51:19 GMT run: 50572032 subRun: 1 event: 75100
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:51:19 GMT run: 50572032 subRun: 1 event: 75100
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:51:19 GMT run: 50572032 subRun: 1 event: 75100
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:51:19 GMT run: 50572032 subRun: 1 event: 75100
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:51:19 GMT run: 50572032 subRun: 1 event: 75100
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:51:19 GMT run: 50572032 subRun: 1 event: 75100
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:51:19 GMT run: 50572032 subRun: 1 event: 75100
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:51:19 GMT run: 50572032 subRun: 1 event: 75100
TuneFullTree failed.
%MSG
Boundary wire vector sizes: 656, 603, 463
minwire 0: 2538
minwire 1: 911
minwire 2: 2226
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.000200151, 0.999787, 1.29991e-05, 
Output 1: 0.913469, 0.0855169, 0.000992015, 2.15538e-05, 
Output 2: 0.998069, 0.00193064, 

624
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2392!
23-Nov-2024 04:51:37 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_750_20231203T065716Z_gen_g4_detsim_hitreco__20240508T075924Z_reco2_reco_data_2024-11-23T_040735Z.root"
23-Nov-2024 04:51:37 GMT  Closed input file "root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/56/f7/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_750_20231203T065716Z_gen_g4_detsim_hitreco__20240508T075924Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.65657       24.5795       233.229       11.2164       39.3551        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0110481     0.0976964      0.42889      0.0726899     0.0686532       100    
reco:linecluster:LineCluster                           0.131359      0.359089       3.65193      0.257304      0.391139        100    
reco:trajcluster:TrajCluster                           0.0363265     0.606581       13.0127      0.115194       1.94682        100    
reco:pandora:StandardPandora                            1.05304       1.6578        10.3577       1.21667       1.5535         100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000129396   0.00664946     0.115068     0.00158642     0.0168613       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000162376    0.0116762     0.177965     0.00300915     0.0255507       100    
reco:pandoracalo:Calorimetry                           9.342e-05     0.0103802     0.189096     0.00107687     0.0300516       100    
reco:pandorapid:Chi2ParticleID                        3.5354e-05    0.000143614   0.00112543    0.000108408   0.00014131       100    
reco:trkshowersplit:TrackShowerHits                   0.000101115    0.0134555     0.281161     0.000777158    0.0443065       100    
reco:pmtrack:PMAlgTrackMaker                          0.00109818      1.35608       20.9822      0.236665       3.32489        100    
reco:pmtrackcalo:Calorimetry                          7.3265e-05    0.00847499     0.187635     0.00107572     0.0265332       100    
reco:pmtrackpid:Chi2ParticleID                        2.8602e-05    0.00022974    0.00175621    0.000119275   0.000312626      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000187811     1.31062       23.5138      0.294941       3.33365        100    
reco:pmtrajfitcalo:Calorimetry                        5.8678e-05     0.0113642     0.201276     0.00147919     0.0323911       100    
reco:pmtrajfitpid:Chi2ParticleID                      2.5423e-05    0.000162777   0.000841071   0.000130067   0.000138242      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000979624     1.67727       29.4346      0.288491       4.38303        100    
reco:pmtracktccalo:Calorimetry                        7.0194e-05    0.00662442     0.124795      0.0011896     0.0179441       100    
reco:pmtracktcpid:Chi2ParticleID                      2.7487e-05    0.000223707   0.00158339    0.000128698   0.000273409      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.00017863      1.21993       14.3847      0.302055       2.6181         100    
reco:pmtrajfittccalo:Calorimetry                      6.0644e-05     0.0128917     0.267664     0.00137866     0.0401312       100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.4984e-05    0.000296021   0.00217987    0.000158026   0.000420459      100    
reco:blurredcluster:BlurredClustering                 0.00262778      1.16814       8.06486      0.577286       1.59938        100    
reco:emtrkmichelid:EmTrackMichelId                     0.278621       11.5863       118.865       4.23896       20.4998        100    
reco:emshower:EMShower                                0.000222113    0.0280856     0.286758     0.00488288     0.0524373       100    
reco:cvnmap:CVNMapper                                  2.268e-05     0.0171272     0.0466601     0.0129971      0.0125         100    
reco:cvneva:CVNEvaluator                              1.5094e-05     0.497404      0.753665      0.607707      0.241737        100    
reco:energyrecnumu:EnergyReco                         0.000268829    0.0217197     0.211859     0.00523116     0.039622        100    
reco:energyrecnue:EnergyReco                          0.000128854    0.002159      0.0397569    0.000514672   0.00582561       100    
reco:energyrecnc:EnergyReco                           0.00177573     0.003854      0.0417046     0.0021607    0.00591229       100    
reco:energyrecnumurange:EnergyReco                    0.000197848    0.0196181     0.207632     0.00263385     0.0393019       100    
reco:energyrecnumumcs:EnergyReco                       0.001659      0.0206672     0.207312      0.0039757     0.0387746       100    
reco:opdec:Deconvolution                               0.107775      0.382555      0.810983      0.369604      0.140557        100    
reco:ophitspe:OpHitFinderDeco                          0.518825       2.23499       4.17294       2.32174      0.810092        100    
reco:opflash:OpFlashFinder                            0.000239732   0.00127655    0.00358413    0.00131295    0.000534605      100    
reco:rns:RandomNumberSaver                            1.7288e-05    2.64628e-05   0.00023129    2.35215e-05   2.1339e-05       100    
reco:anglereconue:NuAngularReco                       0.000140261   0.00178254     0.0356721    0.000366556   0.00518935       100    
reco:anglereconumu:NuAngularReco                      6.4085e-05     0.0015995     0.0352872    0.000192073    0.0051406       100    
reco:anglereconuepfps:NuAngularReco                   0.000109381   0.00172526     0.0361391    0.000305306   0.00522122       100    
reco:anglereconumupfps:NuAngularReco                  9.4482e-05    0.00167443     0.0356502    0.000264126    0.0051807       100    
[art]:TriggerResults:TriggerResultInserter             8.44e-06     1.0653e-05    3.7595e-05    9.7065e-06    3.13666e-06      100    
end_path:out1:RootOutput                               2.432e-06    4.52404e-06   1.7043e-05     3.986e-06    2.1997e-06       100    
end_path:out1:RootOutput(write)                        0.0164959      0.21963      0.728193      0.210602       0.12694        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9323.69 MB
  Peak resident set size usage (VmHWM): 2250.81 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_750_20231203T065716Z_gen_g4_detsim_hitreco__20240508T075924Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_750_20231203T065716Z_gen_g4_detsim_hitreco__20240508T075924Z_reco2_reco_2024-11-23T_040735Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_750_20231203T065716Z_gen_g4_detsim_hitreco__20240508T075924Z_reco2_reco_data_2024-11-23T_040735Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work