Jobsub ID 301705.11@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301705.11@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:31:13
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce512
Last heartbeat	2024-11-23 05:16:08
From worker node	Hostname	b9p05p1493.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:32:40
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_55_20231201T042259Z_gen_g4_detsim_hitreco__20240507T203616Z_reco2.root
Jobscript	Exit code	0
	Real time	43m (2590s)
	CPU time	44m (2661s = 102%)
Outputting started	2024-11-23 05:15:50
Output files
Finished	2024-11-23 05:16:08
List job events Wrapper job log
Jobscript log (last 10,000 characters)

ajfittc@BeginModule  23-Nov-2024 06:15:29 CET run: 6374820 subRun: 1 event: 5600
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:15:29 CET run: 6374820 subRun: 1 event: 5600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:15:29 CET run: 6374820 subRun: 1 event: 5600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:15:29 CET run: 6374820 subRun: 1 event: 5600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:15:29 CET run: 6374820 subRun: 1 event: 5600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:15:29 CET run: 6374820 subRun: 1 event: 5600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:15:44 CET run: 6374820 subRun: 1 event: 5600
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 883, 775, 559
minwire 0: 257
minwire 1: 1986
minwire 2: 226
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.280771, 0.523152, 0.196077, 
Output 1: 0.469219, 0.508891, 0.0191186, 0.00277152, 
Output 2: 0.298009, 0.701991, 

640
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2280!
23-Nov-2024 06:15:49 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_55_20231201T042259Z_gen_g4_detsim_hitreco__20240507T203616Z_reco2_reco_data_2024-11-23T_043246Z.root"
23-Nov-2024 06:15:49 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/85/c4/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_55_20231201T042259Z_gen_g4_detsim_hitreco__20240507T203616Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.04028       23.8524       220.292       8.21052       43.2025        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0163388     0.0374738     0.0772075     0.0389819     0.010949        100    
reco:linecluster:LineCluster                           0.149185      0.487956       9.91055      0.172772       1.45619        100    
reco:trajcluster:TrajCluster                           0.0475385     0.832673       16.4927      0.0550416      2.85377        100    
reco:pandora:StandardPandora                           0.857153       1.8384        16.7635       1.19931       2.45823        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000112822   0.00436186     0.0385033    0.00114226    0.00699727       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000160013    0.0109656     0.221259     0.00209447     0.0280942       100    
reco:pandoracalo:Calorimetry                          8.6902e-05     0.0176321     0.318732     0.000945575    0.0525605       100    
reco:pandorapid:Chi2ParticleID                         3.301e-05    0.000160816   0.00135573    0.000121457   0.000150727      100    
reco:trkshowersplit:TrackShowerHits                   8.2802e-05     0.0229378     0.419215     0.000572589    0.0741007       100    
reco:pmtrack:PMAlgTrackMaker                          0.000902434     2.33972       41.0656      0.127474       6.41265        100    
reco:pmtrackcalo:Calorimetry                          5.6901e-05     0.0064998     0.119326     0.000644576    0.0178091       100    
reco:pmtrackpid:Chi2ParticleID                        2.3351e-05    0.000250075   0.00256158    0.000126612   0.000372695      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000149722     1.88618       23.9755      0.181758       4.37755        100    
reco:pmtrajfitcalo:Calorimetry                        4.9351e-05     0.0173778     0.298722     0.00091152     0.050988        100    
reco:pmtrajfitpid:Chi2ParticleID                      2.2691e-05    0.000163963   0.000672461   0.000142337   0.000127194      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000756623     2.68596       41.3051      0.124047       7.2466         100    
reco:pmtracktccalo:Calorimetry                        6.0121e-05    0.00384494     0.0427688    0.000655795   0.00795119       100    
reco:pmtracktcpid:Chi2ParticleID                       2.369e-05    0.000215905    0.0012665    0.000131887   0.000244807      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000136042     1.47587       16.5769      0.182988       3.19295        100    
reco:pmtrajfittccalo:Calorimetry                      5.2201e-05     0.0101311     0.166787     0.000797788    0.0286626       100    
reco:pmtrajfittcpid:Chi2ParticleID                     2.173e-05    0.000296222   0.00277735    0.000146383   0.000441683      100    
reco:blurredcluster:BlurredClustering                  0.0019453     0.864315       5.58717      0.484933       1.0023         100    
reco:emtrkmichelid:EmTrackMichelId                     0.138022       8.25468       72.4355       1.7169        14.9403        100    
reco:emshower:EMShower                                0.000248734    0.0220957     0.250516     0.00487166     0.0394971       100    
reco:cvnmap:CVNMapper                                  2.08e-05     0.00959934     0.027648      0.0106405    0.00827905       100    
reco:cvneva:CVNEvaluator                               1.508e-05     0.572603       1.21714       0.74924      0.432936        100    
reco:energyrecnumu:EnergyReco                         0.000269165    0.030202       1.42204     0.00318564     0.145307        100    
reco:energyrecnue:EnergyReco                          0.000111732   0.00139497     0.0128538    0.000486698   0.00225135       100    
reco:energyrecnc:EnergyReco                            0.0013629    0.00333869     0.0168938    0.00271184    0.00260128       100    
reco:energyrecnumurange:EnergyReco                    0.000114922    0.0273842      1.3978      0.00195008      0.14211        100    
reco:energyrecnumumcs:EnergyReco                      0.000704041    0.0287712      1.47377     0.00315565     0.148407        100    
reco:opdec:Deconvolution                               0.0739611     0.273059      0.443473      0.279331      0.0868583       100    
reco:ophitspe:OpHitFinderDeco                          0.308855       1.87744       3.75696       1.88431       0.82936        100    
reco:opflash:OpFlashFinder                            0.000299905   0.00134673    0.00637599    0.00129939    0.000770081      100    
reco:rns:RandomNumberSaver                             1.745e-05    3.72393e-05   0.000506438    3.195e-05    4.81152e-05      100    
reco:anglereconue:NuAngularReco                       0.000154142   0.000981917    0.0100344    0.000341106   0.00161499       100    
reco:anglereconumu:NuAngularReco                      6.9841e-05    0.000749948   0.00989181    0.000175268   0.00153883       100    
reco:anglereconuepfps:NuAngularReco                   0.000113612   0.000906511   0.00988238    0.00034092    0.00156652       100    
reco:anglereconumupfps:NuAngularReco                  9.3932e-05    0.000811558   0.00988666    0.00025648    0.00153519       100    
[art]:TriggerResults:TriggerResultInserter             8.27e-06     1.36265e-05   6.6701e-05    1.2105e-05    6.82337e-06      100    
end_path:out1:RootOutput                               3.02e-06     6.15721e-06    2.798e-05     6.075e-06    3.22967e-06      100    
end_path:out1:RootOutput(write)                        0.0191052     0.201627      0.717481      0.194147      0.144685        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21636.2 MB
  Peak resident set size usage (VmHWM): 2217.28 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_55_20231201T042259Z_gen_g4_detsim_hitreco__20240507T203616Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_55_20231201T042259Z_gen_g4_detsim_hitreco__20240507T203616Z_reco2_reco_2024-11-23T_043246Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6374820_55_20231201T042259Z_gen_g4_detsim_hitreco__20240507T203616Z_reco2_reco_data_2024-11-23T_043246Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work