Jobsub ID 108827.155@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108827.155@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:25:34
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce509
Last heartbeat	2024-11-23 06:32:58
From worker node	Hostname	b9p10p6463.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:27:31
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50642276_245_20231208T014429Z_gen_g4_detsim_hitreco__20240510T063944Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (3910s)
	CPU time	1h (3998s = 102%)
Outputting started	2024-11-23 06:32:42
Output files
Finished	2024-11-23 06:32:58
List job events Wrapper job log
Jobscript log (last 10,000 characters)

SG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 07:32:38 CET run: 50642276 subRun: 1 event: 24600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 07:32:38 CET run: 50642276 subRun: 1 event: 24600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 07:32:38 CET run: 50642276 subRun: 1 event: 24600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 07:32:38 CET run: 50642276 subRun: 1 event: 24600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 07:32:38 CET run: 50642276 subRun: 1 event: 24600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:32:38 CET run: 50642276 subRun: 1 event: 24600
1st wire C:0 T:19 P:2 W:576 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:32:38 CET run: 50642276 subRun: 1 event: 24600
1st wire C:0 T:19 P:2 W:579 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:32:39 CET run: 50642276 subRun: 1 event: 24600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:32:39 CET run: 50642276 subRun: 1 event: 24600
Comparing two wires in the same plane: return failure
%MSG
185
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 326!
23-Nov-2024 07:32:41 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50642276_245_20231208T014429Z_gen_g4_detsim_hitreco__20240510T063944Z_reco2_reco_data_2024-11-23T_052735Z.root"
23-Nov-2024 07:32:41 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/21/30/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50642276_245_20231208T014429Z_gen_g4_detsim_hitreco__20240510T063944Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.636368       37.0889       878.402       9.38469       112.618        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0190516     0.0442894     0.105879      0.0421656     0.0167593       100    
reco:linecluster:LineCluster                           0.0945198      0.73636       28.3292      0.184698       3.22953        100    
reco:trajcluster:TrajCluster                           0.047439       2.00032       93.5179      0.067345       10.6986        100    
reco:pandora:StandardPandora                           0.0581194      3.50255       103.872       1.19504       12.5652        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000143671   0.00808779     0.226009     0.00167521     0.0266285       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000184261    0.0223793     0.658947     0.00306163     0.0896176       100    
reco:pandoracalo:Calorimetry                          0.000107351    0.0505582      2.33112     0.00135752     0.256217        100    
reco:pandorapid:Chi2ParticleID                         3.934e-05    0.000168545   0.00105099    0.000130546   0.000164026      100    
reco:trkshowersplit:TrackShowerHits                    8.077e-05     0.0547032      1.85352     0.000961892    0.258912        100    
reco:pmtrack:PMAlgTrackMaker                          0.00102057      3.97288       103.426      0.184671       15.4514        100    
reco:pmtrackcalo:Calorimetry                           6.466e-05     0.0203588     0.976432     0.000987901    0.103365        100    
reco:pmtrackpid:Chi2ParticleID                         2.451e-05     0.0002881    0.00392886    0.000132146   0.000528773      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000168581     3.79849       117.496      0.367352       14.8968        100    
reco:pmtrajfitcalo:Calorimetry                         5.467e-05     0.0489477      2.41573     0.00194185      0.25807        100    
reco:pmtrajfitpid:Chi2ParticleID                       2.508e-05    0.000180094   0.00116029    0.000138266   0.000172263      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000836286     5.40164       181.248      0.245066       21.9891        100    
reco:pmtracktccalo:Calorimetry                        6.4641e-05     0.0110536     0.353517     0.000947792    0.0425163       100    
reco:pmtracktcpid:Chi2ParticleID                       2.517e-05    0.000257136   0.00256068    0.000136011   0.000405978      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000153441     1.95445       42.0176      0.306179       5.80546        100    
reco:pmtrajfittccalo:Calorimetry                       5.764e-05     0.0242759     0.840778     0.00115423     0.102559        100    
reco:pmtrajfittcpid:Chi2ParticleID                     2.463e-05    0.000363471   0.00431504    0.000165106   0.000652094      100    
reco:blurredcluster:BlurredClustering                 0.000146391    0.957993       8.36024      0.367355       1.46035        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000189521     11.5976       199.043       3.1893        27.466         100    
reco:emshower:EMShower                                0.000238911    0.0328594     0.553737     0.00330668     0.0784909       100    
reco:cvnmap:CVNMapper                                  2.507e-05    0.00885273     0.0344304    0.00710959     0.007896        100    
reco:cvneva:CVNEvaluator                               1.425e-05     0.615942       1.21236      0.745356      0.391467        100    
reco:energyrecnumu:EnergyReco                         0.000263791    0.0232767     0.311845     0.00400207     0.0511051       100    
reco:energyrecnue:EnergyReco                          0.000119761   0.00258406     0.0652593    0.000529249   0.00781458       100    
reco:energyrecnc:EnergyReco                           0.00141944    0.00450809     0.0690713    0.00279315     0.0081655       100    
reco:energyrecnumurange:EnergyReco                    0.000141181    0.0213961     0.302866     0.00281524     0.0502183       100    
reco:energyrecnumumcs:EnergyReco                      0.00146333     0.0227286     0.325875     0.00369853     0.0511282       100    
reco:opdec:Deconvolution                               0.063565       0.27921      0.867709      0.269689      0.101668        100    
reco:ophitspe:OpHitFinderDeco                          0.229574       1.64061       4.3668        1.61472      0.724093        100    
reco:opflash:OpFlashFinder                            0.000224221   0.00120327    0.00391596     0.001134     0.000597726      100    
reco:rns:RandomNumberSaver                             1.762e-05    3.23545e-05   0.000268782    3.078e-05    2.48594e-05      100    
reco:anglereconue:NuAngularReco                       0.000138801   0.00198957     0.0565786    0.000376727   0.00671638       100    
reco:anglereconumu:NuAngularReco                       6.191e-05    0.00179095     0.0554659    0.000214137   0.00664085       100    
reco:anglereconuepfps:NuAngularReco                   0.000107121   0.00189681     0.0540014    0.000333197   0.00649782       100    
reco:anglereconumupfps:NuAngularReco                   8.821e-05     0.0018352     0.0528141    0.000285722   0.00643891       100    
[art]:TriggerResults:TriggerResultInserter             7.87e-06     1.13605e-05    3.712e-05    1.0285e-05    3.81663e-06      100    
end_path:out1:RootOutput                               2.92e-06     6.32302e-06    1.964e-05     6.755e-06    2.57766e-06      100    
end_path:out1:RootOutput(write)                        0.0131135     0.218143       1.27514      0.198407      0.180438        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21971.5 MB
  Peak resident set size usage (VmHWM): 2551.62 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50642276_245_20231208T014429Z_gen_g4_detsim_hitreco__20240510T063944Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50642276_245_20231208T014429Z_gen_g4_detsim_hitreco__20240510T063944Z_reco2_reco_2024-11-23T_052735Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50642276_245_20231208T014429Z_gen_g4_detsim_hitreco__20240510T063944Z_reco2_reco_data_2024-11-23T_052735Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work