Jobsub ID 108689.96@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108689.96@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 02:27:41
Site	UK_Lancaster
Entry	UBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat	2024-11-23 04:05:22
From worker node	Hostname	comp05-22
	cpuinfo	Intel(R) Xeon(R) Platinum 8352Y CPU @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 02:37:42
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207207_401_20231120T135040Z_gen_g4_detsim_hitreco__20240503T061643Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (5248s)
	CPU time	1h (4788s = 91%)
Outputting started	2024-11-23 04:05:10
Output files
Finished	2024-11-23 04:05:22
List job events Wrapper job log
Jobscript log (last 10,000 characters)

4:05:01 GMT run: 6207207 subRun: 1 event: 80400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 04:05:01 GMT run: 6207207 subRun: 1 event: 80400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 04:05:01 GMT run: 6207207 subRun: 1 event: 80400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:05:01 GMT run: 6207207 subRun: 1 event: 80400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:05:01 GMT run: 6207207 subRun: 1 event: 80400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:05:01 GMT run: 6207207 subRun: 1 event: 80400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:05:01 GMT run: 6207207 subRun: 1 event: 80400
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 222, 233, 113
minwire 0: 179
minwire 1: 2109
minwire 2: 109
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.189584, 0.0056684, 0.804748, 
Output 1: 0.436468, 0.475279, 0.0851379, 0.00311481, 
Output 2: 0.918338, 0.0816623, 

487
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1413!
23-Nov-2024 04:05:09 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207207_401_20231120T135040Z_gen_g4_detsim_hitreco__20240503T061643Z_reco2_reco_data_2024-11-23T_023744Z.root"
23-Nov-2024 04:05:09 GMT  Closed input file "root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/0a/71/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207207_401_20231120T135040Z_gen_g4_detsim_hitreco__20240503T061643Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.621097       24.6061       362.269       9.41493       44.7496        200    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0112495     0.104955       1.05817      0.0793909     0.0913373       200    
reco:linecluster:LineCluster                           0.115656      0.470398       16.0098      0.287771       1.16557        200    
reco:trajcluster:TrajCluster                           0.0315927     0.764152       36.8946      0.111002       3.33053        200    
reco:pandora:StandardPandora                           0.108512       1.81461       24.9827       1.26905       2.56289        200    
reco:pandoraTrack:LArPandoraTrackCreation             0.00013324    0.00505899     0.0920925    0.00157377     0.010385        200    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000172173    0.0106133     0.270007      0.0029871     0.0247767       200    
reco:pandoracalo:Calorimetry                          9.9895e-05     0.0152907     0.601451     0.000977923    0.0559544       200    
reco:pandorapid:Chi2ParticleID                        3.8065e-05    0.000139338   0.000594865   0.000112914   9.38855e-05      200    
reco:trkshowersplit:TrackShowerHits                   5.5164e-05     0.014956      0.581178     0.000615692    0.0579776       200    
reco:pmtrack:PMAlgTrackMaker                          0.00129251      1.73254       52.7476      0.199943       5.23533        200    
reco:pmtrackcalo:Calorimetry                          7.8479e-05    0.00676521     0.171724     0.000760075    0.0190403       200    
reco:pmtrackpid:Chi2ParticleID                        3.2089e-05    0.000233269   0.00207211    0.000128379   0.000324281      200    
reco:pmtrajfit:PMAlgTrajFitter                        0.00019226      1.59963       38.825       0.286827       4.35163        200    
reco:pmtrajfitcalo:Calorimetry                        6.3698e-05     0.0155661     0.547093     0.00139799     0.0537294       200    
reco:pmtrajfitpid:Chi2ParticleID                      2.8632e-05    0.000161405   0.000647117   0.00012978    0.000121349      200    
reco:pmtracktc:PMAlgTrackMaker                        0.00106876      2.0193        45.6847      0.222547       5.72763        200    
reco:pmtracktccalo:Calorimetry                        7.1683e-05    0.00489631     0.0864298    0.000865248    0.0113956       200    
reco:pmtracktcpid:Chi2ParticleID                       2.967e-05    0.000220075   0.00157969    0.00013299    0.000269725      200    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000169782     1.23068       19.6131       0.26944       2.81006        200    
reco:pmtrajfittccalo:Calorimetry                      6.1518e-05     0.0105134     0.242349     0.00108282     0.0328373       200    
reco:pmtrajfittcpid:Chi2ParticleID                    2.7475e-05    0.000296209   0.00281519    0.000143101   0.000453514      200    
reco:blurredcluster:BlurredClustering                 6.0134e-05      1.03275       7.23907      0.424845       1.37234        200    
reco:emtrkmichelid:EmTrackMichelId                    0.000102756     10.9562       117.111       3.18092       19.7584        200    
reco:emshower:EMShower                                0.000151646    0.0230571     0.341262     0.00427699     0.043663        200    
reco:cvnmap:CVNMapper                                 1.6951e-05     0.0122099     0.0381567     0.0114497    0.00897898       200    
reco:cvneva:CVNEvaluator                              1.5797e-05     0.588729      0.976265      0.779039      0.345495        200    
reco:energyrecnumu:EnergyReco                         0.000268052    0.0289095      0.73312     0.00490122     0.0861069       200    
reco:energyrecnue:EnergyReco                          0.000138219   0.00156237     0.0286733    0.000490525   0.00309163       200    
reco:energyrecnc:EnergyReco                           0.00178455    0.00330056     0.0307326    0.00227576    0.00324348       200    
reco:energyrecnumurange:EnergyReco                    0.000206729    0.0264279     0.722592     0.00210032     0.0852096       200    
reco:energyrecnumumcs:EnergyReco                      0.00106705     0.0277284     0.725005     0.00293629     0.0857145       200    
reco:opdec:Deconvolution                               0.0652296     0.363152      0.713099      0.357205      0.135201        200    
reco:ophitspe:OpHitFinderDeco                          0.114606       1.49344       2.5435        1.5371       0.567184        200    
reco:opflash:OpFlashFinder                            0.000115383   0.00132845    0.00288121    0.00130854    0.00054959       200    
reco:rns:RandomNumberSaver                             1.716e-05    2.93135e-05   0.000208791   2.7372e-05    1.53417e-05      200    
reco:anglereconue:NuAngularReco                       0.000140369   0.00113018     0.0248116    0.000314371   0.00255372       200    
reco:anglereconumu:NuAngularReco                      6.9672e-05    0.000956987    0.0247301    0.000177283   0.00250438       200    
reco:anglereconuepfps:NuAngularReco                   0.000113961   0.00106079     0.0242937    0.000280268    0.0024865       200    
reco:anglereconumupfps:NuAngularReco                  9.5468e-05    0.00102027     0.0246544    0.00023405     0.0025158       200    
[art]:TriggerResults:TriggerResultInserter             8.782e-06    1.12465e-05    3.933e-05    1.05485e-05   2.73955e-06      200    
end_path:out1:RootOutput                               2.475e-06    4.52007e-06     1.7e-05     3.5135e-06    1.96199e-06      200    
end_path:out1:RootOutput(write)                       0.00530399     0.220563      0.742314       0.22453      0.136175        200    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 12864.6 MB
  Peak resident set size usage (VmHWM): 2486.85 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207207_401_20231120T135040Z_gen_g4_detsim_hitreco__20240503T061643Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207207_401_20231120T135040Z_gen_g4_detsim_hitreco__20240503T061643Z_reco2_reco_2024-11-23T_023744Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207207_401_20231120T135040Z_gen_g4_detsim_hitreco__20240503T061643Z_reco2_reco_data_2024-11-23T_023744Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work