Jobsub ID 108744.110@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108744.110@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:38:45
Site	ES_PIC
Entry	DUNE_T1_ES_PIC_ce15-multicore
Last heartbeat	2024-11-23 05:45:00
From worker node	Hostname	tds209.pic.es
	cpuinfo	Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	216000 (60 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:44:07
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74257901_202_20231122T071640Z_gen_g4_detsim_hitreco__20240503T070445Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (3635s)
	CPU time	51m (3099s = 85%)
Outputting started	2024-11-23 05:44:44
Output files
Finished	2024-11-23 05:45:00
List job events Wrapper job log
Jobscript log (last 10,000 characters)

ectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:44:37 CET run: 74257901 subRun: 1 event: 20300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:44:37 CET run: 74257901 subRun: 1 event: 20300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:44:37 CET run: 74257901 subRun: 1 event: 20300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:44:37 CET run: 74257901 subRun: 1 event: 20300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:44:37 CET run: 74257901 subRun: 1 event: 20300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:44:37 CET run: 74257901 subRun: 1 event: 20300
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 51, 33, 35
minwire 0: 1191
minwire 1: 1341
minwire 2: 1181
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max wires due to vertex determination failure: 104, 303
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.993552, 0.000981343, 0.0054664, 
Output 1: 0.0370175, 0.953506, 0.00885383, 0.000622559, 
Output 2: 0.998445, 0.00155467, 

397
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1140!
23-Nov-2024 06:44:42 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74257901_202_20231122T071640Z_gen_g4_detsim_hitreco__20240503T070445Z_reco2_reco_data_2024-11-23T_044411Z.root"
23-Nov-2024 06:44:42 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/d8/da/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74257901_202_20231122T071640Z_gen_g4_detsim_hitreco__20240503T070445Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.47311       31.3126       1163.06       9.83556       117.572        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0456921     0.0834686     0.155973      0.0811308     0.0247808       100    
reco:linecluster:LineCluster                           0.274992      0.547227       4.08054      0.437101      0.496857        100    
reco:trajcluster:TrajCluster                           0.134285       1.10354       57.6275      0.141084       5.89441        100    
reco:pandora:StandardPandora                           0.151628       3.09451       95.0964       2.11078       9.29536        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000145754   0.00902169     0.508141     0.000910346    0.0507869       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000209304    0.013263      0.245597     0.00201097     0.0385091       100    
reco:pandoracalo:Calorimetry                          0.00011186     0.0288565      1.85448     0.000616726    0.186281        100    
reco:pandorapid:Chi2ParticleID                        4.5447e-05    0.000182094   0.00124869    0.000134212   0.000176919      100    
reco:trkshowersplit:TrackShowerHits                   0.000116117    0.0202093     0.942062     0.000516183    0.099874        100    
reco:pmtrack:PMAlgTrackMaker                          0.00134274      2.0132        81.4781      0.0996549      8.54973        100    
reco:pmtrackcalo:Calorimetry                          8.8004e-05     0.0159181      1.12578     0.000723178    0.112268        100    
reco:pmtrackpid:Chi2ParticleID                         3.696e-05    0.000281073   0.00454615    0.000144546   0.000506371      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000203527     5.55446       445.889      0.164179       44.344         100    
reco:pmtrajfitcalo:Calorimetry                        8.3085e-05     0.0284813      1.84497     0.000907003    0.185108        100    
reco:pmtrajfitpid:Chi2ParticleID                      3.6103e-05    0.000193724   0.00135778    0.000140878   0.000173176      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00114827      2.50664       107.142      0.144535       11.2198        100    
reco:pmtracktccalo:Calorimetry                        8.8151e-05     0.0103373     0.680611     0.000783018    0.068013        100    
reco:pmtracktcpid:Chi2ParticleID                      3.6925e-05    0.000259934   0.00389449    0.000155192   0.000437241      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000209698     1.26884       42.1087      0.213429       4.50092        100    
reco:pmtrajfittccalo:Calorimetry                      8.4793e-05     0.0289708      2.17022     0.000792164    0.216169        100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.4861e-05    0.000367191   0.00825699    0.000162129   0.000872559      100    
reco:blurredcluster:BlurredClustering                 0.000123995     1.06844       13.9294      0.445775       1.87777        100    
reco:emtrkmichelid:EmTrackMichelId                     0.0002147      10.3845       285.107       2.31784       31.0352        100    
reco:emshower:EMShower                                0.000268548    0.0237911     0.189413     0.00860737     0.0331975       100    
reco:cvnmap:CVNMapper                                 2.7542e-05     0.0110154     0.0633351     0.011232      0.0103527       100    
reco:cvneva:CVNEvaluator                              2.0194e-05     0.548191       1.31226      0.771171      0.428685        100    
reco:energyrecnumu:EnergyReco                         0.000454593    0.0686357      5.01646     0.00347964     0.500205        100    
reco:energyrecnue:EnergyReco                          0.000182885   0.00300554      0.1522      0.000529612    0.0152511       100    
reco:energyrecnc:EnergyReco                           0.00229171    0.00523333     0.155147     0.00312863     0.0153402       100    
reco:energyrecnumurange:EnergyReco                    0.000277453    0.0616849      4.54496     0.00244376     0.453475        100    
reco:energyrecnumumcs:EnergyReco                      0.000979151    0.0645017      4.77468     0.00338315     0.475713        100    
reco:opdec:Deconvolution                               0.146672      0.511907       1.08174       0.51367      0.129866        100    
reco:ophitspe:OpHitFinderDeco                          0.476003       1.96627        4.321        1.88404      0.687154        100    
reco:opflash:OpFlashFinder                            0.00032691    0.00157049    0.00399376    0.00141754    0.000717377      100    
reco:rns:RandomNumberSaver                            1.9667e-05    3.62872e-05   0.000256796   3.36145e-05   2.40487e-05      100    
reco:anglereconue:NuAngularReco                       0.000159429   0.00228308     0.133375     0.000317326    0.0133784       100    
reco:anglereconumu:NuAngularReco                      9.0017e-05    0.00214827     0.134444     0.000201228    0.0134879       100    
reco:anglereconuepfps:NuAngularReco                   0.000141181   0.00223288     0.128014     0.000336482    0.0128451       100    
reco:anglereconumupfps:NuAngularReco                  0.000129646    0.0021914     0.129976     0.000297367    0.013042        100    
[art]:TriggerResults:TriggerResultInserter            1.1142e-05    1.60396e-05   3.9335e-05    1.4964e-05    4.72155e-06      100    
end_path:out1:RootOutput                               2.854e-06    6.21573e-06   1.5861e-05     6.491e-06    2.52958e-06      100    
end_path:out1:RootOutput(write)                        0.0179454     0.254667       1.47517      0.237252      0.207249        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8684.9 MB
  Peak resident set size usage (VmHWM): 2583.92 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74257901_202_20231122T071640Z_gen_g4_detsim_hitreco__20240503T070445Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74257901_202_20231122T071640Z_gen_g4_detsim_hitreco__20240503T070445Z_reco2_reco_2024-11-23T_044411Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74257901_202_20231122T071640Z_gen_g4_detsim_hitreco__20240503T070445Z_reco2_reco_data_2024-11-23T_044411Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work