Jobsub ID 108726.11@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108726.11@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 03:52:17
Site	UK_Brunel
Entry	CMSHTPC_T2_UK_London_Brunel_dc2_22
Last heartbeat	2024-11-23 04:36:39
From worker node	Hostname	wn-b2-15-00.brunel.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 03:57:53
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574782_593_20231203T105133Z_gen_g4_detsim_hitreco__20240509T200108Z_reco2.root
Jobscript	Exit code	0
	Real time	38m (2311s)
	CPU time	34m (2061s = 89%)
Outputting started	2024-11-23 04:36:25
Output files
Finished	2024-11-23 04:36:39
List job events Wrapper job log
Jobscript log (last 10,000 characters)

nt: 59400
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:35:44 GMT run: 50574782 subRun: 1 event: 59400
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:35:44 GMT run: 50574782 subRun: 1 event: 59400
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:35:44 GMT run: 50574782 subRun: 1 event: 59400
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 04:36:20 GMT run: 50574782 subRun: 1 event: 59400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 04:36:20 GMT run: 50574782 subRun: 1 event: 59400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 04:36:20 GMT run: 50574782 subRun: 1 event: 59400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 04:36:20 GMT run: 50574782 subRun: 1 event: 59400
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 1920, 1628, 1461
minwire 0: 1998
minwire 1: 488
minwire 2: 1960
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0208004, 0.82722, 0.151979, 
Output 1: 0.0896357, 0.21658, 0.250357, 0.443428, 
Output 2: 0.217731, 0.782269, 

686
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2420!
23-Nov-2024 04:36:24 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574782_593_20231203T105133Z_gen_g4_detsim_hitreco__20240509T200108Z_reco2_reco_data_2024-11-23T_035756Z.root"
23-Nov-2024 04:36:24 GMT  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/49/fe/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574782_593_20231203T105133Z_gen_g4_detsim_hitreco__20240509T200108Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.707374       19.7276       208.083       8.77329       31.0354        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0222072     0.0550494     0.295449      0.0556301     0.0296468       100    
reco:linecluster:LineCluster                           0.126728       0.31866       4.72698       0.22514      0.520066        100    
reco:trajcluster:TrajCluster                           0.0635951     0.368796       10.2794      0.0761212      1.21927        100    
reco:pandora:StandardPandora                           0.0714259      1.56535       12.9882       1.25263       1.45041        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000112511   0.00417447     0.0400143    0.00133785    0.00713507       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000169086    0.0148846      0.48105     0.00290167     0.0524005       100    
reco:pandoracalo:Calorimetry                          9.3204e-05     0.0113178     0.292275     0.000931029    0.0411217       100    
reco:pandorapid:Chi2ParticleID                        3.8796e-05    0.00013573    0.00121508    0.000108253   0.000130605      100    
reco:trkshowersplit:TrackShowerHits                   6.0319e-05    0.00831436     0.210137     0.000635892    0.0282411       100    
reco:pmtrack:PMAlgTrackMaker                          0.00117821      1.22809       25.1136      0.153468       3.42004        100    
reco:pmtrackcalo:Calorimetry                          7.9545e-05    0.00464443     0.057499     0.000669104    0.0095457       100    
reco:pmtrackpid:Chi2ParticleID                        3.2448e-05    0.00020222    0.00134252    0.000119468   0.000229198      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.00018434      1.41774       30.2232      0.268679       3.70196        100    
reco:pmtrajfitcalo:Calorimetry                        7.5203e-05     0.0118436     0.291007      0.0013547     0.0411825       100    
reco:pmtrajfitpid:Chi2ParticleID                      3.2534e-05    0.000152657   0.000600287   0.000130824   0.000107308      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00102443      1.60481       38.2325      0.285846       4.52863        100    
reco:pmtracktccalo:Calorimetry                        7.8437e-05    0.00466353     0.0571529    0.000824966   0.00948169       100    
reco:pmtracktcpid:Chi2ParticleID                      3.1977e-05    0.000204239   0.00111167    0.000128869   0.000214116      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000173402     1.11048       14.3094      0.278458       2.21157        100    
reco:pmtrajfittccalo:Calorimetry                      7.6776e-05    0.00913029     0.207174     0.00101427     0.0292844       100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.2123e-05    0.000267618   0.00228157    0.000145789   0.000358822      100    
reco:blurredcluster:BlurredClustering                 7.8342e-05     0.868672       5.83805      0.549701       1.05652        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000112392     8.70185       81.849        3.30671       13.7261        100    
reco:emshower:EMShower                                0.000166184    0.0231614      0.16003      0.0040672     0.0372721       100    
reco:cvnmap:CVNMapper                                  2.129e-05    0.00956692     0.0357838     0.0097186    0.00761983       100    
reco:cvneva:CVNEvaluator                               1.694e-05     0.479653      0.984299      0.642319      0.309818        100    
reco:energyrecnumu:EnergyReco                         0.000280436    0.0281461     0.714417     0.00369435     0.083249        100    
reco:energyrecnue:EnergyReco                          0.000153303   0.00149379     0.0135951    0.000617444   0.00227388       100    
reco:energyrecnc:EnergyReco                           0.00201964     0.0033205     0.016672     0.00234057    0.00245881       100    
reco:energyrecnumurange:EnergyReco                    0.000159577    0.0268902     0.709782      0.0022621     0.0827402       100    
reco:energyrecnumumcs:EnergyReco                      0.00197441     0.0282429     0.715226     0.00296477     0.0831443       100    
reco:opdec:Deconvolution                               0.0683361     0.314711      0.577729      0.312056      0.0908241       100    
reco:ophitspe:OpHitFinderDeco                          0.191093       1.29881       2.32968       1.3169       0.464659        100    
reco:opflash:OpFlashFinder                            0.000118685   0.00117762    0.00633964    0.00111114    0.000696788      100    
reco:rns:RandomNumberSaver                            1.5276e-05    2.62542e-05   0.000394776   2.1655e-05    3.7504e-05       100    
reco:anglereconue:NuAngularReco                       9.5239e-05    0.000995785    0.0116785     0.0003041    0.00186302       100    
reco:anglereconumu:NuAngularReco                      7.6932e-05    0.00088612     0.0112511    0.000217092   0.00177444       100    
reco:anglereconuepfps:NuAngularReco                   0.000116648   0.00100289     0.0113847    0.000307649   0.00188915       100    
reco:anglereconumupfps:NuAngularReco                  0.000112955   0.000962665    0.0114218    0.000262647    0.0018087       100    
[art]:TriggerResults:TriggerResultInserter             9.393e-06    1.20217e-05   5.0365e-05     1.124e-05    4.1728e-06       100    
end_path:out1:RootOutput                               2.587e-06    4.11949e-06   1.9319e-05     3.189e-06    2.41884e-06      100    
end_path:out1:RootOutput(write)                        0.0092311     0.197744      0.618361      0.214471      0.119601        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8264.45 MB
  Peak resident set size usage (VmHWM): 2174.37 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574782_593_20231203T105133Z_gen_g4_detsim_hitreco__20240509T200108Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574782_593_20231203T105133Z_gen_g4_detsim_hitreco__20240509T200108Z_reco2_reco_2024-11-23T_035756Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574782_593_20231203T105133Z_gen_g4_detsim_hitreco__20240509T200108Z_reco2_reco_data_2024-11-23T_035756Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work