Jobsub ID 301836.33@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301836.33@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 06:10:11
Site	UK_RAL-PPD
Entry	CMSHTPC_T2_UK_SGrid_RALPP_hep208
Last heartbeat	2024-11-23 07:29:11
From worker node	Hostname	heplnc149.pp.rl.ac.uk
	cpuinfo	AMD EPYC 7452 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 06:12:47
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_144_20231203T125209Z_gen_g4_detsim_hitreco__20240508T073825Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (4554s)
	CPU time	1h (4372s = 96%)
Outputting started	2024-11-23 07:28:41
Output files
Finished	2024-11-23 07:29:11
List job events Wrapper job log
Jobscript log (last 10,000 characters)

8:25 GMT run: 74511302 subRun: 1 event: 14500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:28:25 GMT run: 74511302 subRun: 1 event: 14500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:28:25 GMT run: 74511302 subRun: 1 event: 14500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:28:25 GMT run: 74511302 subRun: 1 event: 14500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:28:33 GMT run: 74511302 subRun: 1 event: 14500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:28:33 GMT run: 74511302 subRun: 1 event: 14500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:28:33 GMT run: 74511302 subRun: 1 event: 14500
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 240, 199, 201
minwire 0: 1945
minwire 1: 1055
minwire 2: 1823
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.21198, 0.690663, 0.0973568, 
Output 1: 0.359138, 0.458147, 0.161981, 0.0207338, 
Output 2: 0.479968, 0.520032, 

561
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1872!
23-Nov-2024 07:28:40 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_144_20231203T125209Z_gen_g4_detsim_hitreco__20240508T073825Z_reco2_reco_data_2024-11-23T_061253Z.root"
23-Nov-2024 07:28:40 GMT  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/68/d3/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_144_20231203T125209Z_gen_g4_detsim_hitreco__20240508T073825Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.700978        42.31        845.007       12.8902       97.1648        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0231829     0.0470874     0.102107      0.0494347     0.0148861       100    
reco:linecluster:LineCluster                           0.128244      0.388839       12.3108      0.225181       1.20832        100    
reco:trajcluster:TrajCluster                           0.0643063      1.23763       63.7474      0.0781358      6.68304        100    
reco:pandora:StandardPandora                           0.0755211      3.38782       95.067        2.03407       9.38375        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000185009   0.00943079     0.170382     0.00222957     0.0212626       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000277353    0.0235639     0.806126     0.00406241     0.0828752       100    
reco:pandoracalo:Calorimetry                          0.000140255    0.024847      0.963703     0.00142619     0.101998        100    
reco:pandorapid:Chi2ParticleID                        5.6066e-05    0.000266641   0.00287292    0.000197177   0.00032002       100    
reco:trkshowersplit:TrackShowerHits                   7.7185e-05     0.0321373      1.31999     0.000960997    0.143407        100    
reco:pmtrack:PMAlgTrackMaker                          0.00188183      3.40728       110.653      0.231531       12.0835        100    
reco:pmtrackcalo:Calorimetry                          0.000101221    0.0181155     0.714445     0.00122737     0.0771901       100    
reco:pmtrackpid:Chi2ParticleID                        4.1037e-05    0.000477953   0.00626043    0.000220646   0.000782641      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000271492     3.05877       98.257        0.37709       10.5851        100    
reco:pmtrajfitcalo:Calorimetry                        9.4248e-05     0.0253198     0.966265      0.0022088     0.102505        100    
reco:pmtrajfitpid:Chi2ParticleID                      3.9875e-05    0.000310566   0.00146535    0.00022277    0.000245666      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00157746      4.22159       145.202      0.205374       15.6001        100    
reco:pmtracktccalo:Calorimetry                        9.6171e-05     0.0141964     0.367348     0.00123148     0.0497131       100    
reco:pmtracktcpid:Chi2ParticleID                      3.9234e-05    0.000446989   0.00432941    0.000215933   0.000631218      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000239773     2.40862       54.9359       0.37565       6.64523        100    
reco:pmtrajfittccalo:Calorimetry                      9.6321e-05     0.0264388     0.957137     0.00149905     0.107049        100    
reco:pmtrajfittcpid:Chi2ParticleID                    4.0777e-05    0.000599259   0.00696847    0.000265045   0.000957834      100    
reco:blurredcluster:BlurredClustering                 0.000190108     1.34933       12.1483      0.574364       1.94126        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000157698     18.3899       235.438       3.86227       35.3165        100    
reco:emshower:EMShower                                0.000234352    0.0431593      0.39824      0.0105859     0.069682        100    
reco:cvnmap:CVNMapper                                 2.9004e-05     0.0187497     0.0601263     0.0227731     0.0143073       100    
reco:cvneva:CVNEvaluator                              2.5167e-05      1.0887        2.43414       1.38758      0.752804        100    
reco:energyrecnumu:EnergyReco                         0.000453455    0.068552       1.60975     0.00761699     0.222666        100    
reco:energyrecnue:EnergyReco                          0.000190369   0.00299446     0.041807     0.000882615   0.00592274       100    
reco:energyrecnc:EnergyReco                           0.00278507    0.00598742     0.0446794    0.00373478    0.00620345       100    
reco:energyrecnumurange:EnergyReco                    0.000253137    0.0657974      1.64546     0.00378634     0.227653        100    
reco:energyrecnumumcs:EnergyReco                      0.00296756     0.0670935      1.69188     0.00492302     0.225181        100    
reco:opdec:Deconvolution                               0.069955      0.356007      0.906303      0.352974      0.118512        100    
reco:ophitspe:OpHitFinderDeco                          0.193081       2.19039       4.00416       2.18262      0.896763        100    
reco:opflash:OpFlashFinder                            0.000296619   0.00239055    0.00686491    0.00229717    0.00118027       100    
reco:rns:RandomNumberSaver                            2.5138e-05    6.95258e-05   0.00056703    5.95825e-05   5.65967e-05      100    
reco:anglereconue:NuAngularReco                       0.00021733    0.00243066     0.0468178    0.000662139   0.00574928       100    
reco:anglereconumu:NuAngularReco                      9.5601e-05    0.00200191     0.0457894    0.000277013   0.00563843       100    
reco:anglereconuepfps:NuAngularReco                   0.000169019    0.002257      0.0455104    0.000512141   0.00564772       100    
reco:anglereconumupfps:NuAngularReco                  0.000128813    0.0020988     0.0455864    0.00034464    0.00563723       100    
[art]:TriggerResults:TriggerResultInserter            1.4036e-05    2.21338e-05   0.000126368   1.9803e-05    1.23617e-05      100    
end_path:out1:RootOutput                               4.288e-06    1.02747e-05   3.7642e-05     9.583e-06    4.65445e-06      100    
end_path:out1:RootOutput(write)                        0.0139192     0.314818       1.50663       0.31626       0.23463        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21815.8 MB
  Peak resident set size usage (VmHWM): 2393.43 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_144_20231203T125209Z_gen_g4_detsim_hitreco__20240508T073825Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_144_20231203T125209Z_gen_g4_detsim_hitreco__20240508T073825Z_reco2_reco_2024-11-23T_061253Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_144_20231203T125209Z_gen_g4_detsim_hitreco__20240508T073825Z_reco2_reco_data_2024-11-23T_061253Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work