Jobsub ID 301701.1@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301701.1@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:24:39
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce506
Last heartbeat	2024-11-23 05:08:25
From worker node	Hostname	b9p12p9005.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:24:54
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74506302_403_20231202T180519Z_gen_g4_detsim_hitreco__20240508T063807Z_reco2.root
Jobscript	Exit code	0
	Real time	43m (2593s)
	CPU time	46m (2779s = 107%)
Outputting started	2024-11-23 05:08:07
Output files
Finished	2024-11-23 05:08:25
List job events Wrapper job log
Jobscript log (last 10,000 characters)

odule  23-Nov-2024 06:07:59 CET run: 74506302 subRun: 1 event: 40400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:07:59 CET run: 74506302 subRun: 1 event: 40400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:08:03 CET run: 74506302 subRun: 1 event: 40400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:08:03 CET run: 74506302 subRun: 1 event: 40400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:08:03 CET run: 74506302 subRun: 1 event: 40400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:08:03 CET run: 74506302 subRun: 1 event: 40400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:08:03 CET run: 74506302 subRun: 1 event: 40400
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 253, 260, 253
minwire 0: 2252
minwire 1: 6
minwire 2: 2704
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.460831, 0.252835, 0.286334, 
Output 1: 0.415414, 0.464797, 0.112529, 0.00726005, 
Output 2: 0.819799, 0.180201, 

328
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 908!
23-Nov-2024 06:08:06 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74506302_403_20231202T180519Z_gen_g4_detsim_hitreco__20240508T063807Z_reco2_reco_data_2024-11-23T_042459Z.root"
23-Nov-2024 06:08:06 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/5e/40/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74506302_403_20231202T180519Z_gen_g4_detsim_hitreco__20240508T063807Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.16719       23.9982       597.632       9.09547       67.3813        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0163557     0.045944       0.49797      0.044446      0.0471476       100    
reco:linecluster:LineCluster                           0.0945367     0.520727       31.1726      0.168633       3.09224        100    
reco:trajcluster:TrajCluster                           0.0477302      1.02151       69.2246      0.0638453      6.97078        100    
reco:pandora:StandardPandora                           0.0547549      2.4554        72.0928       1.16028       8.17849        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000127821   0.00552008     0.0947792    0.00166403     0.0148604       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000176052    0.0107742     0.199724     0.00299085     0.0266699       100    
reco:pandoracalo:Calorimetry                          9.9511e-05     0.0210724     0.819978     0.00142766     0.0997242       100    
reco:pandorapid:Chi2ParticleID                        3.6261e-05    0.000147474   0.000803039   0.000109207   0.000114016      100    
reco:trkshowersplit:TrackShowerHits                   7.9471e-05     0.0210332     0.991688     0.000921506    0.106254        100    
reco:pmtrack:PMAlgTrackMaker                          0.00100865      2.46473       105.418      0.188941       11.292         100    
reco:pmtrackcalo:Calorimetry                          5.9571e-05    0.00834227     0.223458     0.000918201    0.0309648       100    
reco:pmtrackpid:Chi2ParticleID                         2.404e-05    0.000220379   0.00210454    0.000137432   0.00031675       100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000154792     2.18463       87.0082      0.288094       9.28383        100    
reco:pmtrajfitcalo:Calorimetry                         5.067e-05     0.0216125      1.0117      0.00168425     0.111232        100    
reco:pmtrajfitpid:Chi2ParticleID                      2.1291e-05    0.000158225   0.000705898   0.000134517   0.000121005      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000784289     2.26688       85.0305      0.229964       9.36531        100    
reco:pmtracktccalo:Calorimetry                         5.615e-05    0.00530493     0.120137     0.00085478     0.0163739       100    
reco:pmtracktcpid:Chi2ParticleID                       2.185e-05    0.000208096   0.00211867    0.000131456   0.000268414      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000149072     1.2103        28.4601      0.259553       3.65379        100    
reco:pmtrajfittccalo:Calorimetry                      5.2771e-05     0.0122218     0.413785     0.00115656     0.0516247       100    
reco:pmtrajfittcpid:Chi2ParticleID                     2.217e-05    0.000281919   0.00293879    0.000169257   0.000441523      100    
reco:blurredcluster:BlurredClustering                 0.000121242    0.898263       5.83238      0.472496       1.16783        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000130131     7.97156       106.277       2.91258       16.8522        100    
reco:emshower:EMShower                                0.000185572    0.0238673     0.268167     0.00480648     0.0453799       100    
reco:cvnmap:CVNMapper                                  2.17e-05      0.0100012     0.031317      0.0128871    0.00701949       100    
reco:cvneva:CVNEvaluator                               1.473e-05     0.613162       1.07534      0.746733      0.365393        100    
reco:energyrecnumu:EnergyReco                         0.000334384    0.0334072     0.655317     0.00360003     0.109657        100    
reco:energyrecnue:EnergyReco                          0.000119721   0.00172229     0.0258992    0.000487066   0.00396762       100    
reco:energyrecnc:EnergyReco                           0.00157028    0.00364652     0.0280922    0.00247856    0.00411654       100    
reco:energyrecnumurange:EnergyReco                    0.000156682    0.0289586     0.621176      0.0022087     0.0996754       100    
reco:energyrecnumumcs:EnergyReco                      0.000533896    0.0318565     0.822528     0.00328465     0.111764        100    
reco:opdec:Deconvolution                               0.0904865     0.260905      0.448103      0.267622      0.074524        100    
reco:ophitspe:OpHitFinderDeco                          0.335115       1.63477       3.63655       1.58081       0.6895         100    
reco:opflash:OpFlashFinder                            0.000374984   0.00128511    0.00422841    0.00115919    0.000600347      100    
reco:rns:RandomNumberSaver                            1.7671e-05    4.40523e-05   0.000569627    3.57e-05     6.18328e-05      100    
reco:anglereconue:NuAngularReco                       0.000129131   0.00132439     0.0217658    0.00038984     0.0034131       100    
reco:anglereconumu:NuAngularReco                       5.907e-05    0.00108911     0.0199303    0.000216187   0.00319051       100    
reco:anglereconuepfps:NuAngularReco                   9.7181e-05    0.00122929     0.0198839    0.000352339   0.00318405       100    
reco:anglereconumupfps:NuAngularReco                  9.0731e-05    0.00114158     0.0200635    0.000281253   0.00319601       100    
[art]:TriggerResults:TriggerResultInserter             8.64e-06     1.20287e-05   3.8831e-05    1.1145e-05    3.66974e-06      100    
end_path:out1:RootOutput                               2.94e-06     6.34992e-06    2.229e-05     5.895e-06    2.67924e-06      100    
end_path:out1:RootOutput(write)                        0.0172186     0.201125      0.772312      0.191049      0.136846        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21766.1 MB
  Peak resident set size usage (VmHWM): 2329.14 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74506302_403_20231202T180519Z_gen_g4_detsim_hitreco__20240508T063807Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74506302_403_20231202T180519Z_gen_g4_detsim_hitreco__20240508T063807Z_reco2_reco_2024-11-23T_042459Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74506302_403_20231202T180519Z_gen_g4_detsim_hitreco__20240508T063807Z_reco2_reco_data_2024-11-23T_042459Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work