Jobsub ID 108793.2@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108793.2@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:21:12
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce515
Last heartbeat	2024-11-23 06:18:24
From worker node	Hostname	b9p05p5884.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:23:00
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6464269_328_20231207T030050Z_gen_g4_detsim_hitreco__20240510T035130Z_reco2.root
Jobscript	Exit code	0
	Real time	55m (3305s)
	CPU time	54m (3281s = 99%)
Outputting started	2024-11-23 06:18:05
Output files
Finished	2024-11-23 06:18:24
List job events Wrapper job log
Jobscript log (last 10,000 characters)

23-Nov-2024 07:17:53 CET run: 6464269 subRun: 1 event: 32900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:17:53 CET run: 6464269 subRun: 1 event: 32900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:17:53 CET run: 6464269 subRun: 1 event: 32900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:17:59 CET run: 6464269 subRun: 1 event: 32900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:17:59 CET run: 6464269 subRun: 1 event: 32900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:17:59 CET run: 6464269 subRun: 1 event: 32900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:17:59 CET run: 6464269 subRun: 1 event: 32900
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 269, 253, 235
minwire 0: 1950
minwire 1: 526
minwire 2: 2120
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0168395, 0.191875, 0.791286, 
Output 1: 0.0271315, 0.939913, 0.0322652, 0.000690435, 
Output 2: 0.437239, 0.562761, 

463
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1577!
23-Nov-2024 07:18:03 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6464269_328_20231207T030050Z_gen_g4_detsim_hitreco__20240510T035130Z_reco2_reco_data_2024-11-23T_052304Z.root"
23-Nov-2024 07:18:03 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/ec/40/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6464269_328_20231207T030050Z_gen_g4_detsim_hitreco__20240510T035130Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.554272       30.9757       424.499       8.94279       63.8292        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0171147     0.0613142      1.07639      0.0446286     0.111797        100    
reco:linecluster:LineCluster                           0.0957892     0.506683       9.66591      0.183928       1.33457        100    
reco:trajcluster:TrajCluster                           0.0475971      1.14536       35.2888      0.0609493      4.26443        100    
reco:pandora:StandardPandora                           0.0584948      2.09675       21.1284       1.28093       2.86808        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000127963   0.00622261     0.0810094     0.0013432     0.0135659       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000157873    0.0113323     0.168803      0.0028139     0.0243333       100    
reco:pandoracalo:Calorimetry                          0.000101553    0.0242275     0.484875     0.000922939    0.0714112       100    
reco:pandorapid:Chi2ParticleID                        3.8641e-05    0.000170082   0.00199317    0.000133062   0.000213566      100    
reco:trkshowersplit:TrackShowerHits                   9.6702e-05     0.032613      0.799274     0.000530651    0.108839        100    
reco:pmtrack:PMAlgTrackMaker                          0.000976391     3.00151       63.3884      0.175589       8.89288        100    
reco:pmtrackcalo:Calorimetry                          6.1722e-05     0.011774      0.285215     0.000971065    0.0390505       100    
reco:pmtrackpid:Chi2ParticleID                        2.4401e-05    0.000321825    0.0030428    0.000137633   0.000504521      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000170364     2.7988        45.8063      0.269683       7.53991        100    
reco:pmtrajfitcalo:Calorimetry                        5.6841e-05     0.0251756     0.508021     0.00112755     0.0768422       100    
reco:pmtrajfitpid:Chi2ParticleID                       2.522e-05    0.000183391   0.000914259   0.000143903   0.000161085      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000763897     4.0164        104.674      0.266153       13.1604        100    
reco:pmtracktccalo:Calorimetry                        5.9332e-05     0.0077732     0.177913     0.000849038    0.0222345       100    
reco:pmtracktcpid:Chi2ParticleID                      2.3221e-05    0.000289189   0.00241104    0.000139298   0.000404274      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000146093     1.85566       31.3457      0.200286       4.5896         100    
reco:pmtrajfittccalo:Calorimetry                      5.6311e-05     0.0179831     0.386723     0.000846593    0.0553141       100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.3531e-05    0.00041941    0.00390566    0.000148983   0.000706399      100    
reco:blurredcluster:BlurredClustering                 0.000109653     1.13224       10.8148      0.569509       1.59707        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000104362     10.8406       111.154       2.00914       20.8474        100    
reco:emshower:EMShower                                0.000155203    0.0342665     0.296969     0.00933121     0.0567709       100    
reco:cvnmap:CVNMapper                                  2.001e-05    0.00955539     0.0541315    0.00669428    0.00998755       100    
reco:cvneva:CVNEvaluator                               1.637e-05     0.663984       1.19086      0.921817      0.455554        100    
reco:energyrecnumu:EnergyReco                         0.000278635    0.042172       1.5226      0.00413282     0.171519        100    
reco:energyrecnue:EnergyReco                          0.000116272    0.002257      0.0321899    0.000524811   0.00474768       100    
reco:energyrecnc:EnergyReco                           0.00164393    0.00463304     0.0372254    0.00321412    0.00504084       100    
reco:energyrecnumurange:EnergyReco                    0.000116432    0.0402285      1.58752     0.00290441     0.176604        100    
reco:energyrecnumumcs:EnergyReco                      0.00133155     0.0427247      1.63421     0.00338759     0.179833        100    
reco:opdec:Deconvolution                               0.0504006     0.286179       0.68865      0.275666      0.105874        100    
reco:ophitspe:OpHitFinderDeco                          0.163392       2.00479       4.24204       1.88322       0.92359        100    
reco:opflash:OpFlashFinder                            0.000166313   0.00142923    0.00513819    0.00141206    0.000697973      100    
reco:rns:RandomNumberSaver                             1.696e-05    4.36511e-05   0.000567542   3.8621e-05    5.37845e-05      100    
reco:anglereconue:NuAngularReco                       0.000151283   0.00163797     0.0285924    0.000408539   0.00372256       100    
reco:anglereconumu:NuAngularReco                      6.5621e-05    0.00141695     0.0276048    0.000212944   0.00365555       100    
reco:anglereconuepfps:NuAngularReco                   9.8573e-05    0.00154655     0.0280046    0.000351598   0.00363795       100    
reco:anglereconumupfps:NuAngularReco                  8.3982e-05    0.00144791     0.0294746    0.000275681   0.00366876       100    
[art]:TriggerResults:TriggerResultInserter             8.32e-06     1.40653e-05   6.0592e-05    1.2985e-05    5.98507e-06      100    
end_path:out1:RootOutput                               2.67e-06     6.55258e-06    2.773e-05    6.4505e-06    3.36609e-06      100    
end_path:out1:RootOutput(write)                        0.012606      0.241505        1.063       0.244108      0.182457        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21674.3 MB
  Peak resident set size usage (VmHWM): 2233.71 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6464269_328_20231207T030050Z_gen_g4_detsim_hitreco__20240510T035130Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6464269_328_20231207T030050Z_gen_g4_detsim_hitreco__20240510T035130Z_reco2_reco_2024-11-23T_052304Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6464269_328_20231207T030050Z_gen_g4_detsim_hitreco__20240510T035130Z_reco2_reco_data_2024-11-23T_052304Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work