Jobsub ID 108829.22@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108829.22@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:32:18
Site	UK_Durham
Entry	DUNE_UK_SGridDurham_ce4
Last heartbeat	2024-11-23 07:11:46
From worker node	Hostname	n260.dur.scotgrid.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:33:21
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74551443_569_20231207T112618Z_gen_g4_detsim_hitreco__20240510T053432Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (5887s)
	CPU time	1h (5179s = 87%)
Outputting started	2024-11-23 07:11:29
Output files
Finished	2024-11-23 07:11:46
List job events Wrapper job log
Jobscript log (last 10,000 characters)

1 event: 57000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 07:11:22 GMT run: 74551443 subRun: 1 event: 57000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 07:11:22 GMT run: 74551443 subRun: 1 event: 57000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 07:11:22 GMT run: 74551443 subRun: 1 event: 57000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 07:11:22 GMT run: 74551443 subRun: 1 event: 57000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 07:11:22 GMT run: 74551443 subRun: 1 event: 57000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:11:22 GMT run: 74551443 subRun: 1 event: 57000
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 66, 72, 76
minwire 0: 2510
minwire 1: 1234
minwire 2: 2117
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.00244973, 0.000216576, 0.997334, 
Output 1: 0.0153261, 0.983875, 0.000791193, 8.07093e-06, 
Output 2: 0.998612, 0.00138775, 

474
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1526!
23-Nov-2024 07:11:27 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74551443_569_20231207T112618Z_gen_g4_detsim_hitreco__20240510T053432Z_reco2_reco_data_2024-11-23T_053330Z.root"
23-Nov-2024 07:11:27 GMT  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/60/21/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74551443_569_20231207T112618Z_gen_g4_detsim_hitreco__20240510T053432Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.21582       52.4331       2317.97       13.559        234.976        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.036058      0.0730449      0.24199      0.0755959     0.0279199       100    
reco:linecluster:LineCluster                           0.202857       1.10016       40.2871      0.358824       4.87946        100    
reco:trajcluster:TrajCluster                           0.101821       1.75093       82.6663      0.125443       10.0509        100    
reco:pandora:StandardPandora                           0.112534       5.18455       323.162       1.5708        32.0333        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000133592   0.00988886     0.393023     0.00236379     0.0399397       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000175201    0.0207593     0.672706     0.00452004     0.0708881       100    
reco:pandoracalo:Calorimetry                          0.000109918    0.0379393      2.27825     0.00178002     0.230493        100    
reco:pandorapid:Chi2ParticleID                        3.8392e-05    0.000217418   0.00195049    0.000168218   0.000230817      100    
reco:trkshowersplit:TrackShowerHits                   5.9212e-05     0.0531822      2.9889      0.00136652     0.310255        100    
reco:pmtrack:PMAlgTrackMaker                          0.00110987      4.94953       247.481       0.28297       25.6995        100    
reco:pmtrackcalo:Calorimetry                          6.7106e-05     0.0214769      1.40858     0.00159058     0.140775        100    
reco:pmtrackpid:Chi2ParticleID                        2.8394e-05    0.000395151   0.00741904    0.000227814   0.000786258      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.00016944      10.647        858.297      0.476747       85.4351        100    
reco:pmtrajfitcalo:Calorimetry                         6.371e-05     0.039239       2.33565     0.00271762     0.236309        100    
reco:pmtrajfitpid:Chi2ParticleID                       2.614e-05    0.00025019    0.00234256    0.000217495   0.000259037      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000887364     5.55243       291.898      0.329271       30.3581        100    
reco:pmtracktccalo:Calorimetry                        6.4472e-05     0.0139283     0.634459     0.00162394     0.0664156       100    
reco:pmtracktcpid:Chi2ParticleID                       2.62e-05     0.00039837    0.00630166    0.000239487   0.00070558       100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000147518     2.81368       116.239      0.462027       12.117         100    
reco:pmtrajfittccalo:Calorimetry                      6.1145e-05     0.0317633      1.79036     0.00214317     0.183893        100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.5268e-05    0.000536174   0.00944935    0.000252928   0.00107321       100    
reco:blurredcluster:BlurredClustering                 6.9631e-05      1.38825       13.9784      0.633277       2.07091        100    
reco:emtrkmichelid:EmTrackMichelId                    0.00010108      14.6352       316.796       4.54776       35.9155        100    
reco:emshower:EMShower                                0.000150664    0.0373732     0.575104     0.00631752     0.0739217       100    
reco:cvnmap:CVNMapper                                 1.8495e-05     0.0114186     0.0640625    0.00847342     0.0118709       100    
reco:cvneva:CVNEvaluator                              1.8144e-05     0.675961       1.2764        0.86024      0.430671        100    
reco:energyrecnumu:EnergyReco                         0.00048236     0.0599001      1.69812     0.00573182     0.244979        100    
reco:energyrecnue:EnergyReco                          0.00021666     0.0031424     0.101658     0.000953539    0.010541        100    
reco:energyrecnc:EnergyReco                           0.00187831    0.00589168     0.106481     0.00379314     0.0108725       100    
reco:energyrecnumurange:EnergyReco                    0.000243118    0.0573293      1.72356     0.00342634     0.244451        100    
reco:energyrecnumumcs:EnergyReco                      0.00185684     0.0590356      1.68931     0.00463955     0.243168        100    
reco:opdec:Deconvolution                               0.122308      0.426025      0.755258      0.421166      0.137195        100    
reco:ophitspe:OpHitFinderDeco                          0.304233       2.45808       4.97576       2.2415        1.21886        100    
reco:opflash:OpFlashFinder                            0.000140836   0.00172286    0.00559127    0.00154523    0.000968746      100    
reco:rns:RandomNumberSaver                             2.071e-05    4.66191e-05   0.000424661   3.95145e-05   4.11691e-05      100    
reco:anglereconue:NuAngularReco                       0.000184629   0.00266208      0.11095     0.000573312    0.0113367       100    
reco:anglereconumu:NuAngularReco                      9.0551e-05    0.00233222     0.109101     0.000419928    0.0110989       100    
reco:anglereconuepfps:NuAngularReco                   0.000130407   0.00249188     0.108984     0.000543009    0.0110847       100    
reco:anglereconumupfps:NuAngularReco                  0.000107803    0.0024158     0.108473     0.000503865    0.0110437       100    
[art]:TriggerResults:TriggerResultInserter             9.028e-06    1.73866e-05   7.0522e-05    1.62705e-05   7.26479e-06      100    
end_path:out1:RootOutput                               3.416e-06    8.79637e-06   3.1639e-05    7.2435e-06    4.68709e-06      100    
end_path:out1:RootOutput(write)                       0.00795372     0.300057       2.13274      0.292883      0.270724        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 38849.6 MB
  Peak resident set size usage (VmHWM): 2779.62 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74551443_569_20231207T112618Z_gen_g4_detsim_hitreco__20240510T053432Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74551443_569_20231207T112618Z_gen_g4_detsim_hitreco__20240510T053432Z_reco2_reco_2024-11-23T_053330Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74551443_569_20231207T112618Z_gen_g4_detsim_hitreco__20240510T053432Z_reco2_reco_data_2024-11-23T_053330Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work