Jobsub ID 301700.16@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301700.16@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:20:24
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce506
Last heartbeat	2024-11-23 05:00:39
From worker node	Hostname	b9p10p4272.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:21:56
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50642276_487_20231208T031937Z_gen_g4_detsim_hitreco__20240510T064013Z_reco2.root
Jobscript	Exit code	0
	Real time	38m (2305s)
	CPU time	40m (2447s = 106%)
Outputting started	2024-11-23 05:00:22
Output files
Finished	2024-11-23 05:00:39
List job events Wrapper job log
Jobscript log (last 10,000 characters)

5:59:25 CET run: 50642276 subRun: 1 event: 48800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:59:25 CET run: 50642276 subRun: 1 event: 48800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:59:25 CET run: 50642276 subRun: 1 event: 48800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:59:25 CET run: 50642276 subRun: 1 event: 48800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:00:16 CET run: 50642276 subRun: 1 event: 48800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:00:16 CET run: 50642276 subRun: 1 event: 48800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:00:16 CET run: 50642276 subRun: 1 event: 48800
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 4368, 3867, 2736
minwire 0: 59
minwire 1: 2294
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0282819, 0.862056, 0.109662, 
Output 1: 0.274694, 0.397021, 0.217294, 0.110991, 
Output 2: 0.0594003, 0.9406, 

715
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2706!
23-Nov-2024 06:00:21 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50642276_487_20231208T031937Z_gen_g4_detsim_hitreco__20240510T064013Z_reco2_reco_data_2024-11-23T_042201Z.root"
23-Nov-2024 06:00:21 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/06/db/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50642276_487_20231208T031937Z_gen_g4_detsim_hitreco__20240510T064013Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.90113       21.1122       417.472       7.41679       49.2801        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.016302      0.0402898     0.105182      0.0386206     0.0155468       100    
reco:linecluster:LineCluster                           0.147302      0.418144       18.1691      0.164744       1.82284        100    
reco:trajcluster:TrajCluster                           0.0477538     0.709506       32.6498      0.0563183      3.48532        100    
reco:pandora:StandardPandora                           0.781515       1.78644       41.3146       1.03771       4.27433        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000114402   0.00442129     0.0859336    0.000933001    0.0105173       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000154252   0.00951089     0.165736     0.00213589     0.0219742       100    
reco:pandoracalo:Calorimetry                          8.9671e-05     0.0181217     0.740494     0.000599163    0.0832911       100    
reco:pandorapid:Chi2ParticleID                        3.2711e-05    0.000131914   0.00142514    9.5852e-05    0.000149594      100    
reco:trkshowersplit:TrackShowerHits                   7.3861e-05     0.0187974     0.666847     0.000528026    0.0778206       100    
reco:pmtrack:PMAlgTrackMaker                          0.00091882      1.9404        63.888       0.119801       7.3075         100    
reco:pmtrackcalo:Calorimetry                          5.5931e-05    0.00713741     0.265894     0.000597867    0.0284718       100    
reco:pmtrackpid:Chi2ParticleID                         2.266e-05     0.0002157    0.00248615    0.000109296   0.000333312      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000137561     1.47024       34.9223       0.19878       4.4341         100    
reco:pmtrajfitcalo:Calorimetry                         5.152e-05     0.0214092      1.10566     0.00089614     0.114957        100    
reco:pmtrajfitpid:Chi2ParticleID                      2.1981e-05    0.000142058   0.000633978   0.000104367   0.00010433       100    
reco:pmtracktc:PMAlgTrackMaker                        0.000717528     2.19718       70.4777      0.124163       8.24142        100    
reco:pmtracktccalo:Calorimetry                        5.4501e-05     0.0042445     0.0949782    0.000667733    0.0123526       100    
reco:pmtracktcpid:Chi2ParticleID                       2.229e-05    0.000187144   0.00153815    0.000118482   0.000222564      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000121802     1.21727       28.0836      0.211972       3.40717        100    
reco:pmtrajfittccalo:Calorimetry                       4.906e-05     0.0117608     0.459159     0.000804894    0.0502573       100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.1411e-05    0.00027721    0.00325023    0.000124842   0.000455014      100    
reco:blurredcluster:BlurredClustering                 0.00357324     0.827623       5.36974      0.366807       1.04946        100    
reco:emtrkmichelid:EmTrackMichelId                      0.14854       7.84787       113.964       1.86062       15.967         100    
reco:emshower:EMShower                                0.000235943    0.0190325     0.205841     0.00557119     0.0319289       100    
reco:cvnmap:CVNMapper                                  1.916e-05    0.00957791     0.063584     0.00676708    0.00982492       100    
reco:cvneva:CVNEvaluator                               1.427e-05     0.557147       1.08394      0.749568      0.380674        100    
reco:energyrecnumu:EnergyReco                         0.000258603    0.0149827     0.356229     0.00328279     0.0433212       100    
reco:energyrecnue:EnergyReco                          0.000107442   0.00139522     0.0264391    0.000397939   0.00312218       100    
reco:energyrecnc:EnergyReco                           0.00132145     0.0033064     0.0291152    0.00266837    0.00336193       100    
reco:energyrecnumurange:EnergyReco                    0.000113741    0.0134917      0.38396     0.00168126     0.0445696       100    
reco:energyrecnumumcs:EnergyReco                      0.00085168     0.0148341     0.360074     0.00286311     0.0428913       100    
reco:opdec:Deconvolution                               0.0840873     0.252155       0.42368      0.257844      0.0753936       100    
reco:ophitspe:OpHitFinderDeco                          0.388004       1.48198       3.65154       1.45043      0.636608        100    
reco:opflash:OpFlashFinder                            0.000350724   0.00116322    0.00430808    0.00112376    0.000569073      100    
reco:rns:RandomNumberSaver                             1.73e-05     3.6302e-05    0.000363295   3.33205e-05   3.50012e-05      100    
reco:anglereconue:NuAngularReco                       0.000150832   0.00101894     0.0205118    0.000309324   0.00232032       100    
reco:anglereconumu:NuAngularReco                      6.6841e-05    0.00082035     0.0202161    0.000170597   0.00228533       100    
reco:anglereconuepfps:NuAngularReco                   0.000112601   0.000965076    0.0207449    0.000275909   0.00234509       100    
reco:anglereconumupfps:NuAngularReco                  9.6651e-05    0.000868035    0.0212385    0.000219868   0.00235351       100    
[art]:TriggerResults:TriggerResultInserter             8.48e-06     1.12767e-05   4.2211e-05     9.68e-06     4.96471e-06      100    
end_path:out1:RootOutput                               2.451e-06    6.28866e-06   2.9541e-05     6.315e-06    3.42157e-06      100    
end_path:out1:RootOutput(write)                        0.023332      0.186426      0.832452      0.182151      0.135317        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21793.7 MB
  Peak resident set size usage (VmHWM): 2357.31 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50642276_487_20231208T031937Z_gen_g4_detsim_hitreco__20240510T064013Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50642276_487_20231208T031937Z_gen_g4_detsim_hitreco__20240510T064013Z_reco2_reco_2024-11-23T_042201Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50642276_487_20231208T031937Z_gen_g4_detsim_hitreco__20240510T064013Z_reco2_reco_data_2024-11-23T_042201Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work