Jobsub ID 301747.8@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301747.8@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:06:17
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce512
Last heartbeat	2024-11-23 05:51:27
From worker node	Hostname	b9p05p1493.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:07:43
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420853_18_20231204T032223Z_gen_g4_detsim_hitreco__20240509T203913Z_reco2.root
Jobscript	Exit code	0
	Real time	43m (2606s)
	CPU time	44m (2674s = 102%)
Outputting started	2024-11-23 05:51:10
Output files
Finished	2024-11-23 05:51:27
List job events Wrapper job log
Jobscript log (last 10,000 characters)

23-Nov-2024 06:51:01 CET run: 6420853 subRun: 1 event: 1900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:51:01 CET run: 6420853 subRun: 1 event: 1900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:51:01 CET run: 6420853 subRun: 1 event: 1900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:51:01 CET run: 6420853 subRun: 1 event: 1900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:51:01 CET run: 6420853 subRun: 1 event: 1900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:51:05 CET run: 6420853 subRun: 1 event: 1900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:51:05 CET run: 6420853 subRun: 1 event: 1900
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 293, 225, 188
minwire 0: 697
minwire 1: 1914
minwire 2: 303
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.707022, 0.214969, 0.0780086, 
Output 1: 0.292745, 0.640014, 0.062893, 0.00434823, 
Output 2: 0.157263, 0.842737, 

572
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1943!
23-Nov-2024 06:51:08 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420853_18_20231204T032223Z_gen_g4_detsim_hitreco__20240509T203913Z_reco2_reco_data_2024-11-23T_050750Z.root"
23-Nov-2024 06:51:08 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/6d/1b/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420853_18_20231204T032223Z_gen_g4_detsim_hitreco__20240509T203913Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.65868       23.9729       356.262       9.66312       47.3607        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0175921     0.0442109     0.442691      0.0380222     0.0481798       100    
reco:linecluster:LineCluster                           0.150733      0.381839       13.3524      0.165225       1.36828        100    
reco:trajcluster:TrajCluster                           0.0478767     0.688182       29.9129      0.0601926      3.22445        100    
reco:pandora:StandardPandora                           0.955634       1.93985       27.5061       1.28426       2.92997        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000130842   0.00512515     0.0514061    0.00157139    0.00977787       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000174763    0.0145845     0.415455      0.002624      0.0480463       100    
reco:pandoracalo:Calorimetry                          9.6292e-05     0.0192494     0.479375      0.0011302     0.0652912       100    
reco:pandorapid:Chi2ParticleID                        3.8941e-05    0.000167815   0.00135154    0.000126517   0.000155418      100    
reco:trkshowersplit:TrackShowerHits                   0.000142912    0.0251363     0.637577     0.000831534    0.0870647       100    
reco:pmtrack:PMAlgTrackMaker                          0.00104849      2.1775        53.7839      0.150741       6.67879        100    
reco:pmtrackcalo:Calorimetry                          6.7991e-05    0.00817205      0.1915      0.000887805    0.0244937       100    
reco:pmtrackpid:Chi2ParticleID                        2.5771e-05    0.00029061    0.00246358    0.000150188   0.000416791      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000166263     1.99683        43.09       0.267722       5.51728        100    
reco:pmtrajfitcalo:Calorimetry                        5.7031e-05     0.0173643     0.437653     0.00127207     0.0589861       100    
reco:pmtrajfitpid:Chi2ParticleID                      2.5471e-05    0.00019111    0.00062771    0.000147922   0.000140051      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000817813     2.43885       58.2708      0.194266       7.45755        100    
reco:pmtracktccalo:Calorimetry                        6.0591e-05    0.00516205     0.0639322    0.000942741    0.0120348       100    
reco:pmtracktcpid:Chi2ParticleID                       2.489e-05    0.000250472   0.00141595    0.000153827   0.00028214       100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000149633     1.31901       19.8065       0.23853       3.07635        100    
reco:pmtrajfittccalo:Calorimetry                      5.5271e-05     0.011706      0.260509     0.00109785     0.0362399       100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.3421e-05    0.000339866   0.00277171    0.000168648   0.000478976      100    
reco:blurredcluster:BlurredClustering                 0.00551889      0.90144       5.15113      0.491586       1.08521        100    
reco:emtrkmichelid:EmTrackMichelId                     0.163428       8.49508       94.4911       2.43119       16.6229        100    
reco:emshower:EMShower                                0.000216173    0.0246466      0.28835     0.00530177     0.0466995       100    
reco:cvnmap:CVNMapper                                  2.05e-05      0.0135125     0.044223      0.0160733    0.00996872       100    
reco:cvneva:CVNEvaluator                              1.5261e-05     0.712555       1.22243      0.909811      0.433527        100    
reco:energyrecnumu:EnergyReco                         0.000420006    0.0250546     0.726442     0.00493739     0.0889707       100    
reco:energyrecnue:EnergyReco                          0.000149372   0.00188689     0.017446     0.000515383   0.00334485       100    
reco:energyrecnc:EnergyReco                           0.00167057    0.00463333     0.0216303     0.0034617    0.00388353       100    
reco:energyrecnumurange:EnergyReco                    0.000169752    0.0220828      0.71438     0.00297274     0.0864004       100    
reco:energyrecnumumcs:EnergyReco                      0.00132214     0.0236935     0.713185     0.00398137     0.0863229       100    
reco:opdec:Deconvolution                               0.107138      0.279254      0.634878      0.275493      0.0874871       100    
reco:ophitspe:OpHitFinderDeco                          0.487548       2.12086       4.73725       2.10313      0.873268        100    
reco:opflash:OpFlashFinder                            0.000433128   0.00155161    0.00418005    0.00148687    0.000664375      100    
reco:rns:RandomNumberSaver                            1.9611e-05    4.43087e-05   0.000323505   4.0941e-05    2.99811e-05      100    
reco:anglereconue:NuAngularReco                       0.000164712   0.00132844     0.0132156    0.000445258   0.00245016       100    
reco:anglereconumu:NuAngularReco                      6.7841e-05     0.0011076     0.012766     0.000227108   0.00246499       100    
reco:anglereconuepfps:NuAngularReco                   0.000106642   0.00126635     0.0129649    0.000372381   0.00246588       100    
reco:anglereconumupfps:NuAngularReco                  0.000107832   0.00117243     0.0130201    0.000282299    0.0024697       100    
[art]:TriggerResults:TriggerResultInserter             8.381e-06    1.40976e-05   5.2491e-05    1.28305e-05   5.3479e-06       100    
end_path:out1:RootOutput                               2.85e-06     7.13511e-06   2.2571e-05     6.94e-06     2.71392e-06      100    
end_path:out1:RootOutput(write)                        0.0245628     0.245643       1.02559       0.25859      0.162794        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21680 MB
  Peak resident set size usage (VmHWM): 2255.73 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420853_18_20231204T032223Z_gen_g4_detsim_hitreco__20240509T203913Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420853_18_20231204T032223Z_gen_g4_detsim_hitreco__20240509T203913Z_reco2_reco_2024-11-23T_050750Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420853_18_20231204T032223Z_gen_g4_detsim_hitreco__20240509T203913Z_reco2_reco_data_2024-11-23T_050750Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work