Jobsub ID 103233.0@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	103233.0@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4143
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-19 05:32:19
Site	UK_RAL-PPD
Entry	CMSHTPC_T2_UK_SGrid_RALPP_hep207
Last heartbeat	2024-11-19 07:08:50
From worker node	Hostname	heplnc151.pp.rl.ac.uk
	cpuinfo	AMD EPYC 7452 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-19 05:34:23
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496296_551_20231201T212748Z_gen_g4_detsim_hitreco__20240508T045246Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (5639s)
	CPU time	1h (4791s = 84%)
Outputting started	2024-11-19 07:08:22
Output files
Finished	2024-11-19 07:08:50
List job events Wrapper job log
Jobscript log (last 10,000 characters)

19-Nov-2024 07:08:01 GMT run: 74496296 subRun: 1 event: 55200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  19-Nov-2024 07:08:01 GMT run: 74496296 subRun: 1 event: 55200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  19-Nov-2024 07:08:13 GMT run: 74496296 subRun: 1 event: 55200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  19-Nov-2024 07:08:13 GMT run: 74496296 subRun: 1 event: 55200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  19-Nov-2024 07:08:13 GMT run: 74496296 subRun: 1 event: 55200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  19-Nov-2024 07:08:13 GMT run: 74496296 subRun: 1 event: 55200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  19-Nov-2024 07:08:13 GMT run: 74496296 subRun: 1 event: 55200
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 358, 272, 271
minwire 0: 1307
minwire 1: 1020
minwire 2: 1415
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.212498, 0.0641777, 0.723324, 
Output 1: 0.0201972, 0.0499449, 0.216389, 0.713469, 
Output 2: 0.266914, 0.733086, 

612
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2275!
19-Nov-2024 07:08:20 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496296_551_20231201T212748Z_gen_g4_detsim_hitreco__20240508T045246Z_reco2_reco_data_2024-11-19T_053454Z.root"
19-Nov-2024 07:08:20 GMT  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/70/ea/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496296_551_20231201T212748Z_gen_g4_detsim_hitreco__20240508T045246Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.00448       51.3199       764.74        18.1902       99.172         100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0263952     0.0567282     0.119561      0.059022      0.0174257       100    
reco:linecluster:LineCluster                            0.15682      0.441543       11.5285      0.264772       1.13145        100    
reco:trajcluster:TrajCluster                           0.0673141      1.01502       36.7538      0.0977549      3.93974        100    
reco:pandora:StandardPandora                           0.203104       3.86256       71.7724       2.37556       7.43823        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.00021184     0.0125077     0.231911     0.00250165     0.0279548       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000279468    0.0185128     0.186383     0.00443531     0.033176        100    
reco:pandoracalo:Calorimetry                          0.000154191    0.0332083      1.1421      0.00169743     0.127564        100    
reco:pandorapid:Chi2ParticleID                        5.3461e-05    0.000319213    0.0045943    0.000221975   0.000480788      100    
reco:trkshowersplit:TrackShowerHits                   0.000139413    0.0310335     0.966138     0.00131905     0.107771        100    
reco:pmtrack:PMAlgTrackMaker                          0.00181746      3.88465       99.8264      0.309543       11.814         100    
reco:pmtrackcalo:Calorimetry                          0.000109326    0.0156761     0.315121     0.00130896     0.0450324       100    
reco:pmtrackpid:Chi2ParticleID                        3.8082e-05    0.00056956    0.00462013    0.000246737   0.000807747      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000298484     4.17249       124.186       0.42949       13.6763        100    
reco:pmtrajfitcalo:Calorimetry                        8.6924e-05     0.0370966      1.04563     0.00241839     0.130457        100    
reco:pmtrajfitpid:Chi2ParticleID                      3.8152e-05    0.000357164   0.00210604    0.000256048   0.00032055       100    
reco:pmtracktc:PMAlgTrackMaker                        0.00168093      4.43858       88.8056      0.288989       12.0414        100    
reco:pmtracktccalo:Calorimetry                        0.000103295    0.0123263     0.214192     0.00143741     0.0311202       100    
reco:pmtracktcpid:Chi2ParticleID                      4.1418e-05    0.000536911   0.00808358    0.00024367    0.000959663      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000271272     2.64355       50.5009      0.414209       6.30381        100    
reco:pmtrajfittccalo:Calorimetry                       8.973e-05     0.0299526     0.955847     0.00197639     0.109703        100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.8683e-05    0.000697694   0.00940359    0.00028901    0.00117296       100    
reco:blurredcluster:BlurredClustering                 0.000240624     1.72184       10.967       0.672768       2.2368         100    
reco:emtrkmichelid:EmTrackMichelId                    0.000198925     23.6969       254.395       5.73568       42.5307        100    
reco:emshower:EMShower                                0.000251645    0.0472293     0.453208      0.0139322     0.0746221       100    
reco:cvnmap:CVNMapper                                 3.5647e-05     0.0204891     0.0753327     0.0237789     0.0155662       100    
reco:cvneva:CVNEvaluator                              2.4646e-05      1.67242       3.45224       2.07237       1.0519         100    
reco:energyrecnumu:EnergyReco                         0.000436214    0.0762141      2.76416     0.00929708     0.301854        100    
reco:energyrecnue:EnergyReco                          0.000176533   0.00455148     0.0741118    0.00116932    0.00957758       100    
reco:energyrecnc:EnergyReco                           0.00280519    0.00853007     0.103998     0.00472191     0.0116698       100    
reco:energyrecnumurange:EnergyReco                    0.00026493     0.0638695      2.57894     0.00394045     0.282323        100    
reco:energyrecnumumcs:EnergyReco                      0.00179137     0.0661167      2.49618      0.0079304      0.27475        100    
reco:opdec:Deconvolution                               0.0782746     0.440294      0.864768      0.450882      0.132492        100    
reco:ophitspe:OpHitFinderDeco                          0.217778       2.39513       4.10022       2.44394       0.89762        100    
reco:opflash:OpFlashFinder                            0.000218723   0.00257034     0.0158622    0.00214444    0.00197543       100    
reco:rns:RandomNumberSaver                             2.668e-05    7.51818e-05   0.00122913    6.1166e-05    0.000119167      100    
reco:anglereconue:NuAngularReco                       0.000212982    0.0029701     0.0763047    0.000780223    0.0082841       100    
reco:anglereconumu:NuAngularReco                      9.1884e-05    0.00251966     0.0717779    0.000319598   0.00794688       100    
reco:anglereconuepfps:NuAngularReco                   0.000166244   0.00284054     0.0675207    0.000571324   0.00761759       100    
reco:anglereconumupfps:NuAngularReco                  0.000135516   0.00271303     0.0751266    0.000403722   0.00826113       100    
[art]:TriggerResults:TriggerResultInserter            1.3475e-05    2.22482e-05   9.9498e-05    1.8981e-05    1.38621e-05      100    
end_path:out1:RootOutput                               4.509e-06    1.06067e-05   3.3343e-05    1.02795e-05   4.84878e-06      100    
end_path:out1:RootOutput(write)                        0.0451822     0.380201       1.84574      0.340808      0.286248        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21841.1 MB
  Peak resident set size usage (VmHWM): 2414.56 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496296_551_20231201T212748Z_gen_g4_detsim_hitreco__20240508T045246Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496296_551_20231201T212748Z_gen_g4_detsim_hitreco__20240508T045246Z_reco2_reco_2024-11-19T_053454Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496296_551_20231201T212748Z_gen_g4_detsim_hitreco__20240508T045246Z_reco2_reco_data_2024-11-19T_053454Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work