Jobsub ID 301751.60@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301751.60@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:14:46
Site	UK_Durham
Entry	DUNE_UK_SGridDurham_ce3
Last heartbeat	2024-11-23 06:13:19
From worker node	Hostname	n250.dur.scotgrid.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:16:54
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6455553_229_20231207T113700Z_gen_g4_detsim_hitreco__20240510T052636Z_reco2.root
Jobscript	Exit code	0
	Real time	56m (3369s)
	CPU time	47m (2830s = 84%)
Outputting started	2024-11-23 06:13:03
Output files
Finished	2024-11-23 06:13:19
List job events Wrapper job log
Jobscript log (last 10,000 characters)

failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:12:51 GMT run: 6455553 subRun: 1 event: 23000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:12:51 GMT run: 6455553 subRun: 1 event: 23000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:12:51 GMT run: 6455553 subRun: 1 event: 23000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e pma::Track3D:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:12:51 GMT run: 6455553 subRun: 1 event: 23000
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:12:51 GMT run: 6455553 subRun: 1 event: 23000
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:12:51 GMT run: 6455553 subRun: 1 event: 23000
Hit sorting problem.
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:12:52 GMT run: 6455553 subRun: 1 event: 23000
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 356, 162, 250
minwire 0: 1220
minwire 1: 1344
minwire 2: 1334
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.00556845, 0.000122127, 0.994309, 
Output 1: 0.0301969, 0.868719, 0.100047, 0.00103705, 
Output 2: 0.993296, 0.0067042, 

540
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1716!
23-Nov-2024 06:13:02 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6455553_229_20231207T113700Z_gen_g4_detsim_hitreco__20240510T052636Z_reco2_reco_data_2024-11-23T_051658Z.root"
23-Nov-2024 06:13:02 GMT  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/26/59/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6455553_229_20231207T113700Z_gen_g4_detsim_hitreco__20240510T052636Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.15545       28.4548       258.631       13.2652       41.4855        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0348728     0.0741233     0.690182      0.070856      0.0648606       100    
reco:linecluster:LineCluster                           0.202852      0.434141       1.78447       0.36984       0.22767        100    
reco:trajcluster:TrajCluster                           0.101727       0.49118       11.0094      0.117668       1.50837        100    
reco:pandora:StandardPandora                           0.114163       1.98347       21.6842        1.515        2.25595        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000182174    0.006022      0.0607072     0.0021759     0.0100575       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.00025405     0.0227453     0.537794     0.00560118     0.0713389       100    
reco:pandoracalo:Calorimetry                          0.000141478    0.0138575     0.186993     0.00178941     0.0337919       100    
reco:pandorapid:Chi2ParticleID                        5.5014e-05    0.000226871    0.0028115    0.000167892   0.000288832      100    
reco:trkshowersplit:TrackShowerHits                   0.000115969    0.0162145     0.321345     0.00116615     0.0500414       100    
reco:pmtrack:PMAlgTrackMaker                           0.0013463      2.1953        31.8938       0.28277       5.51858        100    
reco:pmtrackcalo:Calorimetry                          8.8096e-05    0.00879556     0.104077     0.00142633     0.0209715       100    
reco:pmtrackpid:Chi2ParticleID                        3.2913e-05    0.000319391   0.00214803    0.000187765   0.000391417      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000258498     2.14436       41.9144      0.521064       5.61762        100    
reco:pmtrajfitcalo:Calorimetry                        8.7014e-05     0.0144851     0.196954     0.00235404     0.0345462       100    
reco:pmtrajfitpid:Chi2ParticleID                      3.3624e-05    0.000247468   0.00149463    0.00019869    0.000193882      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00119632      2.42023        54.97       0.361916       6.76933        100    
reco:pmtracktccalo:Calorimetry                        8.2897e-05    0.00761929     0.111914     0.00154246     0.0190717       100    
reco:pmtracktcpid:Chi2ParticleID                      3.4625e-05    0.000307917   0.00187806    0.000221733   0.000320458      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000209005     1.52962       16.1204      0.534477       2.70473        100    
reco:pmtrajfittccalo:Calorimetry                      8.6824e-05     0.0103938      0.15852     0.00184295     0.0250253       100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.6369e-05    0.000407596   0.00270173    0.000226117   0.000508599      100    
reco:blurredcluster:BlurredClustering                 8.8126e-05      1.36987       7.07921      0.614511       1.65869        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000135035     11.3851       80.8225       3.96575       17.251         100    
reco:emshower:EMShower                                0.000208494    0.0360864     0.376265      0.0105813     0.0595472       100    
reco:cvnmap:CVNMapper                                 2.5108e-05     0.0139633     0.0504756     0.0121983     0.0113859       100    
reco:cvneva:CVNEvaluator                              2.2092e-05     0.740273       1.08565      0.925447      0.378556        100    
reco:energyrecnumu:EnergyReco                         0.000441143    0.0419089      1.90192     0.00601713      0.19273        100    
reco:energyrecnue:EnergyReco                          0.000212431   0.00199827     0.0171404    0.000719958   0.00310668       100    
reco:energyrecnc:EnergyReco                           0.00236433    0.00489085     0.0219617     0.0037171    0.00352223       100    
reco:energyrecnumurange:EnergyReco                    0.000227068    0.0394665      1.89202     0.00337681     0.191923        100    
reco:energyrecnumumcs:EnergyReco                      0.00091665     0.0410084      1.88429     0.00434101     0.190959        100    
reco:opdec:Deconvolution                               0.171709       0.4422       0.804224      0.430224      0.113389        100    
reco:ophitspe:OpHitFinderDeco                          0.304115       2.6521        5.16618       2.60106      0.999074        100    
reco:opflash:OpFlashFinder                            0.000188967   0.00184952    0.00725911    0.00176534    0.000909357      100    
reco:rns:RandomNumberSaver                            2.3935e-05    5.14287e-05   0.000506536   4.70585e-05   4.82227e-05      100    
reco:anglereconue:NuAngularReco                       0.000203394    0.0014952     0.0148406    0.000613488   0.00248988       100    
reco:anglereconumu:NuAngularReco                       9.517e-05    0.00123345     0.014073     0.000353242   0.00239371       100    
reco:anglereconuepfps:NuAngularReco                   0.000154372   0.00143375     0.0145368    0.00050063    0.00246376       100    
reco:anglereconumupfps:NuAngularReco                  0.000127881   0.00145975     0.0143572    0.000420979   0.00277528       100    
[art]:TriggerResults:TriggerResultInserter            1.0651e-05    1.69553e-05   8.8437e-05    1.57895e-05   7.84856e-06      100    
end_path:out1:RootOutput                               3.647e-06    9.24939e-06   3.6148e-05     9.778e-06    4.11259e-06      100    
end_path:out1:RootOutput(write)                        0.0112859     0.301878      0.816357      0.300169       0.16348        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 38269.9 MB
  Peak resident set size usage (VmHWM): 2213.29 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6455553_229_20231207T113700Z_gen_g4_detsim_hitreco__20240510T052636Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6455553_229_20231207T113700Z_gen_g4_detsim_hitreco__20240510T052636Z_reco2_reco_2024-11-23T_051658Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6455553_229_20231207T113700Z_gen_g4_detsim_hitreco__20240510T052636Z_reco2_reco_data_2024-11-23T_051658Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work