Jobsub ID 301643.7@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301643.7@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 02:15:38
Site	UK_Lancaster
Entry	UBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat	2024-11-23 03:01:03
From worker node	Hostname	comp23-21
	cpuinfo	Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 02:23:06
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6402377_841_20231202T112956Z_gen_g4_detsim_hitreco__20240507T211315Z_reco2.root
Jobscript	Exit code	0
	Real time	37m (2263s)
	CPU time	33m (2033s = 89%)
Outputting started	2024-11-23 03:00:49
Output files
Finished	2024-11-23 03:01:03
List job events Wrapper job log
Jobscript log (last 10,000 characters)

3:00:40 GMT run: 6402377 subRun: 1 event: 84200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 03:00:40 GMT run: 6402377 subRun: 1 event: 84200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 03:00:41 GMT run: 6402377 subRun: 1 event: 84200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 03:00:41 GMT run: 6402377 subRun: 1 event: 84200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 03:00:41 GMT run: 6402377 subRun: 1 event: 84200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 03:00:41 GMT run: 6402377 subRun: 1 event: 84200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 03:00:41 GMT run: 6402377 subRun: 1 event: 84200
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 166, 85, 128
minwire 0: 2207
minwire 1: 115
minwire 2: 2532
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.376757, 0.0278561, 0.595387, 
Output 1: 0.017181, 0.310012, 0.595874, 0.0769327, 
Output 2: 0.434832, 0.565168, 

328
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 870!
23-Nov-2024 03:00:48 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6402377_841_20231202T112956Z_gen_g4_detsim_hitreco__20240507T211315Z_reco2_reco_data_2024-11-23T_022309Z.root"
23-Nov-2024 03:00:48 GMT  Closed input file "root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/2c/8d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6402377_841_20231202T112956Z_gen_g4_detsim_hitreco__20240507T211315Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.29362       20.7061       185.007       10.9013       29.8832        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0108603     0.121741      0.623528      0.0823949     0.109241        100    
reco:linecluster:LineCluster                           0.136265       0.32648       1.18917      0.282676      0.183595        100    
reco:trajcluster:TrajCluster                           0.0338923     0.394526       10.6501       0.10525       1.26696        100    
reco:pandora:StandardPandora                            1.1246        1.53587       8.02497       1.29825      0.911421        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000130374   0.00406758     0.0616413    0.00169103    0.00782561       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000174494    0.0151242     0.720932     0.00351796     0.0718035       100    
reco:pandoracalo:Calorimetry                          0.00010112    0.00857191     0.152469     0.00113485     0.0246515       100    
reco:pandorapid:Chi2ParticleID                        3.7694e-05    0.000137128   0.00116978    0.000113471   0.000123778      100    
reco:trkshowersplit:TrackShowerHits                   0.000112569   0.00886768     0.251185     0.000682896    0.0306075       100    
reco:pmtrack:PMAlgTrackMaker                          0.00116731      1.09489       18.8363      0.237946       2.80353        100    
reco:pmtrackcalo:Calorimetry                          7.3545e-05    0.00526596     0.110477     0.000826877    0.0144937       100    
reco:pmtrackpid:Chi2ParticleID                         2.949e-05    0.000198322   0.00121907    0.00012153    0.000235956      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000191907     1.09457       18.634       0.244682       2.56554        100    
reco:pmtrajfitcalo:Calorimetry                        6.2201e-05    0.00860964     0.142379     0.00124954     0.0233108       100    
reco:pmtrajfitpid:Chi2ParticleID                      2.6855e-05    0.000148909   0.000583201   0.000127743   9.2837e-05       100    
reco:pmtracktc:PMAlgTrackMaker                        0.000972465     1.26943       20.4096      0.243321       3.17682        100    
reco:pmtracktccalo:Calorimetry                        6.8492e-05    0.00509676     0.191253     0.000883248    0.019656        100    
reco:pmtracktcpid:Chi2ParticleID                      2.8374e-05    0.000188517   0.00112704    0.000126424   0.000200188      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000181918    0.854642       9.93015      0.288625       1.6097         100    
reco:pmtrajfittccalo:Calorimetry                      6.3435e-05     0.0070255     0.135108     0.00118857     0.0194853       100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.6819e-05    0.000241611   0.00166234    0.000131272   0.000301242      100    
reco:blurredcluster:BlurredClustering                 0.00322733     0.973557       6.12353       0.49617       1.26084        100    
reco:emtrkmichelid:EmTrackMichelId                     0.270257       9.43794       102.737       3.56837       16.1178        100    
reco:emshower:EMShower                                0.000232979    0.0174781     0.151437     0.00141125     0.0318408       100    
reco:cvnmap:CVNMapper                                 2.2569e-05     0.0123604     0.0379973     0.0129167    0.00882479       100    
reco:cvneva:CVNEvaluator                              1.6232e-05     0.568173      0.870848      0.755143      0.346755        100    
reco:energyrecnumu:EnergyReco                         0.000255368    0.0343406     0.655566     0.00468463     0.0879229       100    
reco:energyrecnue:EnergyReco                          0.000134944   0.00127614     0.020173     0.000472558   0.00252065       100    
reco:energyrecnc:EnergyReco                           0.00181528    0.00302097     0.0224695    0.00220156     0.0026032       100    
reco:energyrecnumurange:EnergyReco                    0.00018889     0.0322896      0.65184     0.00206649     0.0874988       100    
reco:energyrecnumumcs:EnergyReco                      0.000980746    0.0334646     0.651251     0.00255536     0.0872455       100    
reco:opdec:Deconvolution                                0.12419      0.391942      0.798353      0.385423      0.145697        100    
reco:ophitspe:OpHitFinderDeco                          0.328792       2.24005       3.90911       2.24166      0.864181        100    
reco:opflash:OpFlashFinder                            0.000263208    0.0012131    0.00445462    0.00112407    0.00057018       100    
reco:rns:RandomNumberSaver                             1.811e-05    2.87612e-05   0.000298687   2.4457e-05    2.78598e-05      100    
reco:anglereconue:NuAngularReco                       0.000139378   0.000934471    0.0167427    0.000345554   0.00201311       100    
reco:anglereconumu:NuAngularReco                      6.6261e-05    0.000771172    0.0162642    0.000214958   0.00194261       100    
reco:anglereconuepfps:NuAngularReco                   0.000107243    0.0008838     0.0164324    0.000312315   0.00195595       100    
reco:anglereconumupfps:NuAngularReco                   9.381e-05    0.000843131    0.0164955    0.000277744   0.00196456       100    
[art]:TriggerResults:TriggerResultInserter             9.311e-06    1.13595e-05   4.1804e-05    1.0836e-05    3.4381e-06       100    
end_path:out1:RootOutput                               2.852e-06    4.61942e-06   1.5484e-05    3.7775e-06    1.99719e-06      100    
end_path:out1:RootOutput(write)                        0.0127935     0.198108      0.674035      0.204592      0.116647        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9433.63 MB
  Peak resident set size usage (VmHWM): 2283.53 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6402377_841_20231202T112956Z_gen_g4_detsim_hitreco__20240507T211315Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6402377_841_20231202T112956Z_gen_g4_detsim_hitreco__20240507T211315Z_reco2_reco_2024-11-23T_022309Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6402377_841_20231202T112956Z_gen_g4_detsim_hitreco__20240507T211315Z_reco2_reco_data_2024-11-23T_022309Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work