Jobsub ID 108688.127@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108688.127@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 02:26:36
Site	UK_Lancaster
Entry	UBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat	2024-11-23 03:20:38
From worker node	Hostname	comp23-28
	cpuinfo	Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 02:36:42
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_1007_20231203T051438Z_gen_g4_detsim_hitreco__20240508T071008Z_reco2.root
Jobscript	Exit code	0
	Real time	43m (2622s)
	CPU time	37m (2229s = 85%)
Outputting started	2024-11-23 03:20:25
Output files
Finished	2024-11-23 03:20:38
List job events Wrapper job log
Jobscript log (last 10,000 characters)

20:12 GMT run: 66006611 subRun: 1 event: 100800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 03:20:12 GMT run: 66006611 subRun: 1 event: 100800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 03:20:12 GMT run: 66006611 subRun: 1 event: 100800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 03:20:12 GMT run: 66006611 subRun: 1 event: 100800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 03:20:19 GMT run: 66006611 subRun: 1 event: 100800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 03:20:19 GMT run: 66006611 subRun: 1 event: 100800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 03:20:19 GMT run: 66006611 subRun: 1 event: 100800
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 309, 339, 232
minwire 0: 1249
minwire 1: 2269
minwire 2: 636
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.000226478, 0.99966, 0.000113529, 
Output 1: 0.0103411, 0.568503, 0.414114, 0.00704146, 
Output 2: 0.98824, 0.0117601, 

625
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2287!
23-Nov-2024 03:20:24 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_1007_20231203T051438Z_gen_g4_detsim_hitreco__20240508T071008Z_reco2_reco_data_2024-11-23T_023646Z.root"
23-Nov-2024 03:20:24 GMT  Closed input file "root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/ab/f9/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_1007_20231203T051438Z_gen_g4_detsim_hitreco__20240508T071008Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.19097       22.8921       271.339       11.2678       38.6687        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0112405     0.118542       1.08736      0.0800215     0.128273        100    
reco:linecluster:LineCluster                           0.134278       0.47792       3.2397       0.345575      0.438115        100    
reco:trajcluster:TrajCluster                           0.0314157     0.773176       15.6941      0.144378       2.29096        100    
reco:pandora:StandardPandora                           0.104226       1.90219       40.0142        1.26         3.98949        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000124579   0.00485835      0.10654     0.00183805     0.012468        100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000152838   0.00817903     0.130273     0.00293373     0.018679        100    
reco:pandoracalo:Calorimetry                          9.0216e-05     0.0104765     0.260954     0.00131338     0.0320885       100    
reco:pandorapid:Chi2ParticleID                        3.3854e-05    0.000129259   0.000595249   0.000110018   9.51334e-05      100    
reco:trkshowersplit:TrackShowerHits                   7.0909e-05     0.0141227     0.286066     0.000934961    0.0462387       100    
reco:pmtrack:PMAlgTrackMaker                          0.00110612      1.41395       26.2137      0.208787       3.61823        100    
reco:pmtrackcalo:Calorimetry                          6.7873e-05     0.0065811     0.177607     0.000821969    0.0211466       100    
reco:pmtrackpid:Chi2ParticleID                        2.7478e-05    0.000222018   0.00172727    0.00013169    0.000291869      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000169527     1.24109       18.3346      0.293179       2.7937         100    
reco:pmtrajfitcalo:Calorimetry                        5.9523e-05     0.011303      0.275204     0.00152394     0.0336224       100    
reco:pmtrajfitpid:Chi2ParticleID                      2.4198e-05    0.000151317   0.000710879   0.000123539   0.000129535      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000920532     1.77381       27.4224      0.231552       4.44276        100    
reco:pmtracktccalo:Calorimetry                        6.5592e-05    0.00502669     0.143248     0.000858548    0.0163922       100    
reco:pmtracktcpid:Chi2ParticleID                      2.6319e-05    0.000197676   0.00132653    0.000128764   0.000229474      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000159548    0.976865       12.6602      0.297015       2.01417        100    
reco:pmtrajfittccalo:Calorimetry                      5.8578e-05     0.010671      0.236397     0.00119508     0.0327825       100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.4211e-05    0.000271451   0.00183885    0.000141064   0.000373048      100    
reco:blurredcluster:BlurredClustering                 5.6533e-05     0.826081       4.67654      0.350022       1.00913        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000115828     10.1801       115.874       4.31455       18.1971        100    
reco:emshower:EMShower                                0.000176436    0.0226914     0.208794      0.0035775     0.041002        100    
reco:cvnmap:CVNMapper                                 1.8487e-05     0.0123189     0.0393518     0.0124352    0.00996254       100    
reco:cvneva:CVNEvaluator                              1.4175e-05     0.499338      0.872098      0.714868      0.343489        100    
reco:energyrecnumu:EnergyReco                         0.000287296    0.0403019      2.48016      0.0042573     0.249287        100    
reco:energyrecnue:EnergyReco                          0.000121851   0.00170421     0.0356058    0.000637944    0.0041884       100    
reco:energyrecnc:EnergyReco                           0.00151955    0.00302208     0.0370904    0.00192125    0.00431234       100    
reco:energyrecnumurange:EnergyReco                    0.000203217    0.0379928      2.44169     0.00182185     0.245645        100    
reco:energyrecnumumcs:EnergyReco                      0.00150552     0.0391122      2.45786     0.00233893     0.247117        100    
reco:opdec:Deconvolution                               0.0652794     0.416186       2.4688       0.369243      0.270733        100    
reco:ophitspe:OpHitFinderDeco                          0.223787       1.87442       3.57628       1.9975       0.797923        100    
reco:opflash:OpFlashFinder                            0.000221972   0.00111801    0.00230736    0.00116649    0.00048334       100    
reco:rns:RandomNumberSaver                            1.6113e-05    3.00497e-05   0.000218956   2.82655e-05   2.06953e-05      100    
reco:anglereconue:NuAngularReco                       0.00013082    0.00120198     0.0302856    0.000354925   0.00356029       100    
reco:anglereconumu:NuAngularReco                      6.1684e-05    0.00102233     0.0296429    0.000209685   0.00348976       100    
reco:anglereconuepfps:NuAngularReco                   0.000103306    0.0011341     0.0298185    0.000300843   0.00350494       100    
reco:anglereconumupfps:NuAngularReco                  8.8907e-05    0.00108763     0.0302686    0.000260716   0.00354437       100    
[art]:TriggerResults:TriggerResultInserter             8.18e-06     1.09608e-05   3.6023e-05    1.0343e-05    3.12089e-06      100    
end_path:out1:RootOutput                               2.147e-06    4.13995e-06    1.234e-05    4.6095e-06    1.77553e-06      100    
end_path:out1:RootOutput(write)                       0.00989122     0.181995      0.694845      0.202145      0.125067        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9475.45 MB
  Peak resident set size usage (VmHWM): 2384.47 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_1007_20231203T051438Z_gen_g4_detsim_hitreco__20240508T071008Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_1007_20231203T051438Z_gen_g4_detsim_hitreco__20240508T071008Z_reco2_reco_2024-11-23T_023646Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006611_1007_20231203T051438Z_gen_g4_detsim_hitreco__20240508T071008Z_reco2_reco_data_2024-11-23T_023646Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work