Jobsub ID 108793.3@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108793.3@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:21:12
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce515
Last heartbeat	2024-11-23 07:03:07
From worker node	Hostname	b9p07p2176.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:23:20
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65840069_376_20231122T052513Z_gen_g4_detsim_hitreco__20240503T065824Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (5967s)
	CPU time	1h (5952s = 99%)
Outputting started	2024-11-23 07:02:47
Output files
Finished	2024-11-23 07:03:07
List job events Wrapper job log
Jobscript log (last 10,000 characters)

Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 08:02:33 CET run: 65840069 subRun: 1 event: 75400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 08:02:33 CET run: 65840069 subRun: 1 event: 75400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 08:02:33 CET run: 65840069 subRun: 1 event: 75400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:02:33 CET run: 65840069 subRun: 1 event: 75400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:02:34 CET run: 65840069 subRun: 1 event: 75400
1st wire C:0 T:12 P:2 W:518 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:02:34 CET run: 65840069 subRun: 1 event: 75400
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 435, 152, 304
minwire 0: 1431
minwire 1: 1090
minwire 2: 1511
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0111743, 0.000469596, 0.988356, 
Output 1: 0.889845, 0.0993703, 0.00936962, 0.00141552, 
Output 2: 0.923023, 0.0769773, 

574
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1895!
23-Nov-2024 08:02:46 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65840069_376_20231122T052513Z_gen_g4_detsim_hitreco__20240503T065824Z_reco2_reco_data_2024-11-23T_052325Z.root"
23-Nov-2024 08:02:46 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/5c/3a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65840069_376_20231122T052513Z_gen_g4_detsim_hitreco__20240503T065824Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              0.56029       27.9488       854.05        9.39365       81.9147        200    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0166046     0.0447367     0.308202      0.043419      0.0235599       200    
reco:linecluster:LineCluster                           0.0946859     0.782536       60.3491       0.17901       4.69782        200    
reco:trajcluster:TrajCluster                           0.0478852      1.34842       99.0853      0.0604613      8.33456        200    
reco:pandora:StandardPandora                           0.0582834      2.54173       79.7052       1.28679       7.56256        200    
reco:pandoraTrack:LArPandoraTrackCreation             0.000130742   0.00566993     0.171864     0.00137971     0.016657        200    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000173403    0.0105588     0.243987     0.00266833     0.0252041       200    
reco:pandoracalo:Calorimetry                          0.000111732    0.0263232      1.53536     0.00094339     0.135944        200    
reco:pandorapid:Chi2ParticleID                        3.6941e-05    0.000162287   0.00123697    0.000128082   0.000133084      200    
reco:trkshowersplit:TrackShowerHits                   7.4841e-05     0.0287167      1.09852     0.000617269     0.12831        200    
reco:pmtrack:PMAlgTrackMaker                          0.00108055      2.63941       140.503       0.1499        11.9759        200    
reco:pmtrackcalo:Calorimetry                          6.5491e-05    0.00954302     0.578155     0.000845468    0.0463198       200    
reco:pmtrackpid:Chi2ParticleID                         2.588e-05    0.000275794   0.00527385    0.000148472   0.000509097      200    
reco:pmtrajfit:PMAlgTrajFitter                        0.000183693     2.62555       111.304      0.196101       11.287         200    
reco:pmtrajfitcalo:Calorimetry                        6.2641e-05     0.023923       1.01132     0.00130082     0.110078        200    
reco:pmtrajfitpid:Chi2ParticleID                       2.669e-05    0.000193429   0.00132459    0.000147447   0.000177834      200    
reco:pmtracktc:PMAlgTrackMaker                        0.000896354     2.9601        132.618      0.162201       12.4237        200    
reco:pmtracktccalo:Calorimetry                        6.3681e-05    0.00568356     0.126339     0.000873454    0.0158211       200    
reco:pmtracktcpid:Chi2ParticleID                       2.655e-05    0.00025589     0.001928     0.000144118   0.00033607       200    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000198283     1.52555       46.1876      0.243297        5.028         200    
reco:pmtrajfittccalo:Calorimetry                      6.2941e-05     0.0141127     0.518808     0.00103314     0.0592919       200    
reco:pmtrajfittcpid:Chi2ParticleID                     2.56e-05     0.000330207   0.00354821    0.000161143   0.000523083      200    
reco:blurredcluster:BlurredClustering                 0.000129162    0.933799       6.93194      0.491817       1.27973        200    
reco:emtrkmichelid:EmTrackMichelId                    0.000128592     8.88917       161.272       2.42831       20.6649        200    
reco:emshower:EMShower                                0.000197283    0.028846      0.572186     0.00387229     0.0614099       200    
reco:cvnmap:CVNMapper                                  2.281e-05     0.0109909     0.0434672    0.00747517     0.0102421       200    
reco:cvneva:CVNEvaluator                               1.584e-05     0.746109       6.49004      0.947724      0.694656        200    
reco:energyrecnumu:EnergyReco                         0.000349975    0.0384849      1.31209     0.00434945     0.141775        200    
reco:energyrecnue:EnergyReco                          0.000122132   0.00205452     0.0539888    0.000521074   0.00546823       200    
reco:energyrecnc:EnergyReco                           0.00149283    0.00443776     0.0569344    0.00318268    0.00570956       200    
reco:energyrecnumurange:EnergyReco                    0.000159883    0.0369133      1.27781     0.00292275     0.143699        200    
reco:energyrecnumumcs:EnergyReco                      0.00150646     0.0384914      1.23106     0.00346593     0.147222        200    
reco:opdec:Deconvolution                               0.0505605     0.288255       2.88566      0.288192      0.204672        200    
reco:ophitspe:OpHitFinderDeco                          0.142946       2.08382       7.49097       2.01329      0.992796        200    
reco:opflash:OpFlashFinder                            0.000173973   0.00147689    0.00414404    0.00139093    0.000694893      200    
reco:rns:RandomNumberSaver                            1.9021e-05    3.84673e-05   0.000301485   3.6595e-05    2.08697e-05      200    
reco:anglereconue:NuAngularReco                       0.000168663   0.00148891     0.0439448    0.00037621     0.004651        200    
reco:anglereconumu:NuAngularReco                      7.1171e-05    0.00129324     0.042344     0.000197633    0.0045508       200    
reco:anglereconuepfps:NuAngularReco                   0.000130782   0.00141668     0.0425938    0.000346186   0.00455353       200    
reco:anglereconumupfps:NuAngularReco                  9.8782e-05     0.0013472     0.0430785    0.000275699   0.00455967       200    
[art]:TriggerResults:TriggerResultInserter             8.97e-06     1.46092e-05   0.000113152   1.34705e-05   7.86044e-06      200    
end_path:out1:RootOutput                               2.94e-06     6.12362e-06   3.3641e-05     6.285e-06    2.86389e-06      200    
end_path:out1:RootOutput(write)                        0.0112821     0.244456       1.34619       0.25919      0.191967        200    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21928.9 MB
  Peak resident set size usage (VmHWM): 2502.64 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65840069_376_20231122T052513Z_gen_g4_detsim_hitreco__20240503T065824Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65840069_376_20231122T052513Z_gen_g4_detsim_hitreco__20240503T065824Z_reco2_reco_2024-11-23T_052325Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65840069_376_20231122T052513Z_gen_g4_detsim_hitreco__20240503T065824Z_reco2_reco_data_2024-11-23T_052325Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work