Jobsub ID 108743.38@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108743.38@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:34:30
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce512
Last heartbeat	2024-11-23 05:20:58
From worker node	Hostname	b9p08p1455.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:39:32
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74506302_1087_20231202T183421Z_gen_g4_detsim_hitreco__20240508T061943Z_reco2.root
Jobscript	Exit code	0
	Real time	41m (2466s)
	CPU time	43m (2580s = 104%)
Outputting started	2024-11-23 05:20:39
Output files
Finished	2024-11-23 05:20:58
List job events Wrapper job log
Jobscript log (last 10,000 characters)

event: 108800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:20:30 CET run: 74506302 subRun: 1 event: 108800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:20:30 CET run: 74506302 subRun: 1 event: 108800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:20:30 CET run: 74506302 subRun: 1 event: 108800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:20:30 CET run: 74506302 subRun: 1 event: 108800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:20:30 CET run: 74506302 subRun: 1 event: 108800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:20:33 CET run: 74506302 subRun: 1 event: 108800
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 155, 158, 139
minwire 0: 647
minwire 1: 1706
minwire 2: 608
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.469935, 0.0746531, 0.455412, 
Output 1: 0.190904, 0.614232, 0.181194, 0.0136704, 
Output 2: 0.0396027, 0.960397, 

462
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1407!
23-Nov-2024 06:20:38 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74506302_1087_20231202T183421Z_gen_g4_detsim_hitreco__20240508T061943Z_reco2_reco_data_2024-11-23T_043938Z.root"
23-Nov-2024 06:20:38 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/b1/a3/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74506302_1087_20231202T183421Z_gen_g4_detsim_hitreco__20240508T061943Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.523812       22.7378       529.404       8.79601       57.2242        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0170154     0.043773      0.168289      0.0426673     0.0208052       100    
reco:linecluster:LineCluster                           0.0942206     0.248699       5.75255      0.174342      0.561801        100    
reco:trajcluster:TrajCluster                           0.0475424     0.513256       32.1784      0.0605812      3.21654        100    
reco:pandora:StandardPandora                           0.0590997      2.02926       63.5985       1.2719        6.23334        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000155411   0.00507989     0.115988     0.00172482     0.0134396       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000213253    0.0234384     0.524539     0.00275054     0.0822303       100    
reco:pandoracalo:Calorimetry                          0.000119492    0.0242835      1.5088      0.000933785    0.151097        100    
reco:pandorapid:Chi2ParticleID                        4.1861e-05    0.00016855    0.00249937    0.000123887   0.000249451      100    
reco:trkshowersplit:TrackShowerHits                   6.6611e-05     0.0160274      0.94298     0.000750393    0.0949429       100    
reco:pmtrack:PMAlgTrackMaker                          0.00094048      1.97048       72.5021       0.1518        7.75801        100    
reco:pmtrackcalo:Calorimetry                           6.224e-05    0.00951951     0.458356     0.00096964     0.047426        100    
reco:pmtrackpid:Chi2ParticleID                        2.4921e-05    0.000241811   0.00271995    0.000139216   0.000375155      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000224903     1.84747       61.035       0.259286       6.94772        100    
reco:pmtrajfitcalo:Calorimetry                        7.9091e-05     0.019746       1.02709     0.00151409     0.104389        100    
reco:pmtrajfitpid:Chi2ParticleID                       3.167e-05    0.000177658   0.000970141   0.000139291   0.000141901      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00115748      2.50882       102.942      0.198803       10.6871        100    
reco:pmtracktccalo:Calorimetry                        7.6911e-05    0.00740959     0.256628     0.00106806     0.0284816       100    
reco:pmtracktcpid:Chi2ParticleID                      3.2211e-05    0.00022742    0.00328607    0.000156542   0.000350991      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000203122     1.41367       41.1449      0.279493       4.62021        100    
reco:pmtrajfittccalo:Calorimetry                      8.2071e-05     0.0141009     0.708039     0.00132452     0.0741631       100    
reco:pmtrajfittcpid:Chi2ParticleID                     3.28e-05     0.000314788   0.00531653    0.000182492   0.000581251      100    
reco:blurredcluster:BlurredClustering                 0.000127351     0.69092       4.67766      0.394543      0.884596        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000123101     8.08364       133.354       2.42387       17.4017        100    
reco:emshower:EMShower                                0.000192292    0.0220072     0.164686     0.00596329     0.0353935       100    
reco:cvnmap:CVNMapper                                 2.6051e-05     0.0101451     0.0338464     0.0127425    0.00839783       100    
reco:cvneva:CVNEvaluator                               1.788e-05     0.602015       1.08645      0.826025      0.440531        100    
reco:energyrecnumu:EnergyReco                         0.000298053    0.0509226      1.01767     0.00415442     0.170748        100    
reco:energyrecnue:EnergyReco                          0.000130382   0.00163438     0.0256488    0.000499431   0.00363558       100    
reco:energyrecnc:EnergyReco                           0.00162278    0.00401565     0.0295285    0.00315553    0.00377418       100    
reco:energyrecnumurange:EnergyReco                    0.000139242    0.048987       1.10062      0.002903      0.172446        100    
reco:energyrecnumumcs:EnergyReco                      0.00108606     0.0510489      1.0817      0.00339239     0.171167        100    
reco:opdec:Deconvolution                               0.0492522     0.264872      0.475552      0.269228      0.0862268       100    
reco:ophitspe:OpHitFinderDeco                          0.141994       1.96777       4.0179        1.89033      0.858892        100    
reco:opflash:OpFlashFinder                            0.000134481   0.00165079    0.00596384    0.00160179    0.000751934      100    
reco:rns:RandomNumberSaver                             2.098e-05    4.54987e-05   0.000328744   4.37505e-05   3.01176e-05      100    
reco:anglereconue:NuAngularReco                       0.000185402   0.00119765     0.0265433    0.000456526   0.00295073       100    
reco:anglereconumu:NuAngularReco                      8.9741e-05    0.000926985    0.0250608    0.000257533   0.00283822       100    
reco:anglereconuepfps:NuAngularReco                   0.000146821   0.00108768     0.0282051    0.000394109   0.00310082       100    
reco:anglereconumupfps:NuAngularReco                  0.000106571   0.000971159    0.0244416    0.000301003   0.00278446       100    
[art]:TriggerResults:TriggerResultInserter             8.78e-06     1.52866e-05   7.6691e-05     1.434e-05    7.04992e-06      100    
end_path:out1:RootOutput                                 3e-06      7.1646e-06     2.892e-05     7.155e-06    3.08303e-06      100    
end_path:out1:RootOutput(write)                        0.011665      0.235675       1.39708      0.257086       0.18754        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21602.7 MB
  Peak resident set size usage (VmHWM): 2189.75 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74506302_1087_20231202T183421Z_gen_g4_detsim_hitreco__20240508T061943Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74506302_1087_20231202T183421Z_gen_g4_detsim_hitreco__20240508T061943Z_reco2_reco_2024-11-23T_043938Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74506302_1087_20231202T183421Z_gen_g4_detsim_hitreco__20240508T061943Z_reco2_reco_data_2024-11-23T_043938Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work