Jobsub ID 301437.57@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301437.57@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-22 22:04:16
Site	UK_QMUL
Entry	DUNE_UK_London_QMUL_arcce02
Last heartbeat	2024-11-22 23:24:56
From worker node	Hostname	cn110.htc.esc.qmul
	cpuinfo	Intel(R) Xeon(R) Gold 6248R CPU @ 3.00GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	aborted
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-22 22:16:38
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6416859_955_20231203T084605Z_gen_g4_detsim_hitreco__20240509T211626Z_reco2.root
Outputting started	2024-11-22 23:18:41
Output files
Finished	2024-11-22 23:24:56
List job events Wrapper job log
Jobscript log (last 10,000 characters)

k3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 23:18:25 GMT run: 6416859 subRun: 1 event: 95600
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 23:18:25 GMT run: 6416859 subRun: 1 event: 95600
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 23:18:25 GMT run: 6416859 subRun: 1 event: 95600
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 23:18:25 GMT run: 6416859 subRun: 1 event: 95600
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 23:18:25 GMT run: 6416859 subRun: 1 event: 95600
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 23:18:25 GMT run: 6416859 subRun: 1 event: 95600
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  22-Nov-2024 23:18:34 GMT run: 6416859 subRun: 1 event: 95600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  22-Nov-2024 23:18:34 GMT run: 6416859 subRun: 1 event: 95600
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 266, 338, 263
minwire 0: 1566
minwire 1: 385
minwire 2: 1867
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.623278, 0.167222, 0.2095, 
Output 1: 0.306499, 0.486782, 0.182051, 0.0246679, 
Output 2: 0.502826, 0.497174, 

443
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1408!
22-Nov-2024 23:18:38 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6416859_955_20231203T084605Z_gen_g4_detsim_hitreco__20240509T211626Z_reco2_reco_data_2024-11-22T_221644Z.root"
22-Nov-2024 23:18:38 GMT  Closed input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/dc/44/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6416859_955_20231203T084605Z_gen_g4_detsim_hitreco__20240509T211626Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              0.36988       34.8016       463.496       12.471        76.6268        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00235049     0.0742548     0.535739      0.0228202     0.112629        100    
reco:linecluster:LineCluster                           0.0198706     0.683306       19.521       0.0488466      2.36472        100    
reco:trajcluster:TrajCluster                          0.00338485      1.68309       52.5614      0.0317292      6.96194        100    
reco:pandora:StandardPandora                           0.0302582      2.5092        41.1543       1.3733        4.8828         100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000130222   0.00703958     0.150071     0.00205212     0.0178895       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000174132   0.00859816     0.085468     0.00353351     0.0149044       100    
reco:pandoracalo:Calorimetry                          8.8767e-05     0.0292128     0.731745     0.00144938     0.111155        100    
reco:pandorapid:Chi2ParticleID                        3.6889e-05    0.000145884   0.000607785   0.000118812   0.000100974      100    
reco:trkshowersplit:TrackShowerHits                    7.921e-05     0.0275061     0.595763     0.000929232    0.0961139       100    
reco:pmtrack:PMAlgTrackMaker                          0.00110992      2.54224       49.345       0.234951       8.19292        100    
reco:pmtrackcalo:Calorimetry                          7.5717e-05     0.0119172     0.385308     0.00103324     0.045593        100    
reco:pmtrackpid:Chi2ParticleID                        3.1167e-05    0.000290821   0.00327519    0.000136471   0.000484635      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000192026     2.58121       45.2143       0.2865        8.01575        100    
reco:pmtrajfitcalo:Calorimetry                        5.8794e-05     0.030676      0.808394     0.00195296     0.116868        100    
reco:pmtrajfitpid:Chi2ParticleID                      2.6604e-05    0.000179214   0.000792938   0.000141353   0.000147249      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00098385      3.0187        55.4231      0.283365       9.37909        100    
reco:pmtracktccalo:Calorimetry                        7.1069e-05    0.00719065     0.224008     0.00105007     0.0261064       100    
reco:pmtracktcpid:Chi2ParticleID                      2.8685e-05    0.00073004     0.0483695    0.000146741   0.00479971       100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000186831     1.52415       24.0777      0.281403       3.8604         100    
reco:pmtrajfittccalo:Calorimetry                      5.9556e-05     0.0210211     0.569904     0.00112796     0.0768509       100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.5093e-05    0.000359871   0.00338847    0.00015376    0.000591221      100    
reco:blurredcluster:BlurredClustering                 6.3715e-05     0.953892       8.22594      0.482405       1.29319        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000104965     15.7745       204.096       5.28139       32.4624        100    
reco:emshower:EMShower                                0.000158984    0.0264429     0.184718     0.00462591     0.0405319       100    
reco:cvnmap:CVNMapper                                 1.6909e-05     0.0137911     0.0467758     0.0143659     0.0101127       100    
reco:cvneva:CVNEvaluator                              1.5431e-05     0.595258       1.02461      0.796075      0.363638        100    
reco:energyrecnumu:EnergyReco                         0.000265267    0.0326996      1.42644     0.00492103     0.145128        100    
reco:energyrecnue:EnergyReco                          0.000119476   0.00223638     0.0525976    0.000639808    0.0061764       100    
reco:energyrecnc:EnergyReco                           0.00175484    0.00380457     0.0573776    0.00213122    0.00641816       100    
reco:energyrecnumurange:EnergyReco                    0.000210293    0.0284059      1.22231     0.00193761      0.12571        100    
reco:energyrecnumumcs:EnergyReco                      0.00143145     0.0295783      1.22654     0.00289167     0.125982        100    
reco:opdec:Deconvolution                               0.0161987     0.320353       1.84276       0.20961      0.334894        100    
reco:ophitspe:OpHitFinderDeco                          0.116538       2.04539       4.32303       2.05459      0.885142        100    
reco:opflash:OpFlashFinder                            0.000202108   0.00126841    0.00299566    0.00130619    0.000496862      100    
reco:rns:RandomNumberSaver                            1.8988e-05    3.08648e-05   0.000264223   2.6745e-05    2.45225e-05      100    
reco:anglereconue:NuAngularReco                       0.000146313   0.00166185     0.0415563    0.000434726   0.00482783       100    
reco:anglereconumu:NuAngularReco                      6.2466e-05    0.00143539     0.0406603    0.000249791   0.00470434       100    
reco:anglereconuepfps:NuAngularReco                   0.000104671    0.0015503     0.0408874    0.000361189   0.00472312       100    
reco:anglereconumupfps:NuAngularReco                  8.8513e-05    0.00150606     0.0410187    0.000319611   0.00473921       100    
[art]:TriggerResults:TriggerResultInserter             8.288e-06    1.10755e-05   3.9624e-05    1.05995e-05   3.27005e-06      100    
end_path:out1:RootOutput                               2.322e-06    4.1612e-06    1.4608e-05     3.515e-06    1.72184e-06      100    
end_path:out1:RootOutput(write)                        0.010001      0.205014      0.927871      0.203662       0.15164        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 17103.3 MB
  Peak resident set size usage (VmHWM): 2456.44 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6416859_955_20231203T084605Z_gen_g4_detsim_hitreco__20240509T211626Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6416859_955_20231203T084605Z_gen_g4_detsim_hitreco__20240509T211626Z_reco2_reco_2024-11-22T_221644Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6416859_955_20231203T084605Z_gen_g4_detsim_hitreco__20240509T211626Z_reco2_reco_data_2024-11-22T_221644Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work