Jobsub ID 108736.123@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108736.123@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:19:19
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce509
Last heartbeat	2024-11-23 07:26:37
From worker node	Hostname	b9g37p7399.cern.ch
	cpuinfo	Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:26:19
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6452759_10_20231207T053453Z_gen_g4_detsim_hitreco__20240510T061156Z_reco2.root
Jobscript	Exit code	0
	Real time	2h (10734s)
	CPU time	2h (10465s = 97%)
Outputting started	2024-11-23 07:25:15
Output files
Finished	2024-11-23 07:26:37
List job events Wrapper job log
Jobscript log (last 10,000 characters)

EMShower:emshower@BeginModule  23-Nov-2024 08:24:46 CET run: 6452759 subRun: 1 event: 1100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 08:24:46 CET run: 6452759 subRun: 1 event: 1100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 08:24:46 CET run: 6452759 subRun: 1 event: 1100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 08:24:46 CET run: 6452759 subRun: 1 event: 1100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 08:24:46 CET run: 6452759 subRun: 1 event: 1100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 08:24:46 CET run: 6452759 subRun: 1 event: 1100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 08:24:46 CET run: 6452759 subRun: 1 event: 1100
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 6226, 5276, 2739
minwire 0: 674
minwire 1: 1682
minwire 2: 198
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.00358059, 0.992959, 0.00346052, 
Output 1: 0.0964211, 0.303696, 0.246167, 0.353716, 
Output 2: 0.105875, 0.894125, 

867
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3529!
23-Nov-2024 08:25:04 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6452759_10_20231207T053453Z_gen_g4_detsim_hitreco__20240510T061156Z_reco2_reco_data_2024-11-23T_042624Z.root"
23-Nov-2024 08:25:04 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/a8/2a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6452759_10_20231207T053453Z_gen_g4_detsim_hitreco__20240510T061156Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.598377       104.561       3654.33       14.292        395.01         100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0179433     0.0444351     0.259767      0.0434261     0.0249223       100    
reco:linecluster:LineCluster                           0.0963449      1.87247       38.0464      0.194662       6.76735        100    
reco:trajcluster:TrajCluster                           0.0483569       7.674        243.449      0.074536       35.2303        100    
reco:pandora:StandardPandora                           0.0754009      19.7619       1284.91       2.42986       128.865        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000173797    0.0268998     0.931493     0.00238107     0.102657        100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000232299    0.0408495     0.745669     0.00544287     0.105749        100    
reco:pandoracalo:Calorimetry                          0.000132773    0.168713       8.20946     0.00170577      0.86834        100    
reco:pandorapid:Chi2ParticleID                        5.0417e-05    0.000297437   0.00191624    0.000205568   0.000306327      100    
reco:trkshowersplit:TrackShowerHits                   0.000107519    0.141959       5.1115      0.00147237     0.605782        100    
reco:pmtrack:PMAlgTrackMaker                          0.00159112      10.1352       286.911      0.345761       35.5078        100    
reco:pmtrackcalo:Calorimetry                          0.000103456    0.0672544      3.65521     0.00131529     0.376157        100    
reco:pmtrackpid:Chi2ParticleID                        4.1387e-05    0.000668395    0.0127758    0.000213781   0.00155909       100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000261883     9.44005        390.2       0.482809        41.4          100    
reco:pmtrajfitcalo:Calorimetry                        9.3803e-05     0.167716       8.14622     0.00240619     0.865398        100    
reco:pmtrajfitpid:Chi2ParticleID                       3.97e-05     0.000405068    0.0133144    0.000197329   0.00131912       100    
reco:pmtracktc:PMAlgTrackMaker                        0.00140845      12.4606       482.773      0.347825       53.0425        100    
reco:pmtracktccalo:Calorimetry                        0.000101007    0.0340431      1.59191     0.00172227     0.163396        100    
reco:pmtracktcpid:Chi2ParticleID                      4.1619e-05    0.000586862   0.00972579    0.000230427   0.00120186       100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000260303     6.32854       249.641      0.442478       26.6201        100    
reco:pmtrajfittccalo:Calorimetry                      9.6323e-05     0.100324       4.46922     0.00209351      0.48187        100    
reco:pmtrajfittcpid:Chi2ParticleID                    4.0997e-05    0.000833003    0.0144769    0.000221654   0.00189867       100    
reco:blurredcluster:BlurredClustering                 0.000136237     1.79591       21.1003      0.856245       2.90239        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000214159     29.4816       583.574       4.78378       75.1203        100    
reco:emshower:EMShower                                0.000312089    0.0648589     0.982988     0.00617689     0.144412        100    
reco:cvnmap:CVNMapper                                 3.5151e-05     0.019344      0.0969859     0.0201845     0.0163242       100    
reco:cvneva:CVNEvaluator                              2.1705e-05      1.03904       7.46407       1.01089       0.97143        100    
reco:energyrecnumu:EnergyReco                         0.000460275    0.168478       7.79644     0.00528518     0.811724        100    
reco:energyrecnue:EnergyReco                          0.000199523   0.00872395     0.290404     0.000799087    0.0336193       100    
reco:energyrecnc:EnergyReco                            0.0027982     0.0113571     0.291093     0.00391796     0.0331768       100    
reco:energyrecnumurange:EnergyReco                    0.000209502    0.165422       7.88486      0.003778      0.823329        100    
reco:energyrecnumumcs:EnergyReco                      0.00150552     0.170757       8.02434     0.00477036     0.839693        100    
reco:opdec:Deconvolution                               0.0784069     0.380865      0.710712      0.369529      0.133058        100    
reco:ophitspe:OpHitFinderDeco                          0.143654       2.32151       5.20243       2.19028       1.04134        100    
reco:opflash:OpFlashFinder                            0.000249742   0.00211421    0.00605166    0.00181981    0.00112742       100    
reco:rns:RandomNumberSaver                            2.2223e-05    4.72898e-05   0.000296183   4.3496e-05    2.83308e-05      100    
reco:anglereconue:NuAngularReco                       0.000184993   0.00682663     0.255071     0.000467025    0.028338        100    
reco:anglereconumu:NuAngularReco                      0.000102631   0.00618743      0.25115     0.000321197    0.0268371       100    
reco:anglereconuepfps:NuAngularReco                   0.000184303   0.00658849     0.255011     0.000481901    0.0275074       100    
reco:anglereconumupfps:NuAngularReco                  0.000146756   0.00645656     0.254147     0.000450611    0.027285        100    
[art]:TriggerResults:TriggerResultInserter            1.1599e-05    1.86437e-05   6.3114e-05    1.84395e-05   6.35377e-06      100    
end_path:out1:RootOutput                               3.115e-06    6.56381e-06   1.9785e-05     6.676e-06    2.34219e-06      100    
end_path:out1:RootOutput(write)                        0.0192148     0.434558       3.82219      0.364425      0.462413        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9897.28 MB
  Peak resident set size usage (VmHWM): 2696.07 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6452759_10_20231207T053453Z_gen_g4_detsim_hitreco__20240510T061156Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6452759_10_20231207T053453Z_gen_g4_detsim_hitreco__20240510T061156Z_reco2_reco_2024-11-23T_042624Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6452759_10_20231207T053453Z_gen_g4_detsim_hitreco__20240510T061156Z_reco2_reco_data_2024-11-23T_042624Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work