Jobsub ID 301679.106@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301679.106@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 03:41:13
Site	ES_PIC
Entry	DUNE_T1_ES_PIC_ce13-multicore
Last heartbeat	2024-11-23 04:52:12
From worker node	Hostname	tds426.pic.es
	cpuinfo	AMD EPYC 7502 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	216000 (60 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 03:43:16
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511854_720_20231203T171034Z_gen_g4_detsim_hitreco__20240509T201009Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (4110s)
	CPU time	57m (3445s = 83%)
Outputting started	2024-11-23 04:51:47
Output files
Finished	2024-11-23 04:52:12
List job events Wrapper job log
Jobscript log (last 10,000 characters)

23-Nov-2024 05:50:50 CET run: 74511854 subRun: 1 event: 72100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:50:50 CET run: 74511854 subRun: 1 event: 72100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:51:38 CET run: 74511854 subRun: 1 event: 72100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:51:38 CET run: 74511854 subRun: 1 event: 72100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:51:38 CET run: 74511854 subRun: 1 event: 72100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:51:38 CET run: 74511854 subRun: 1 event: 72100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:51:38 CET run: 74511854 subRun: 1 event: 72100
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 2194, 1460, 1592
minwire 0: 404
minwire 1: 1636
minwire 2: 374
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0287361, 0.922568, 0.0486959, 
Output 1: 0.310205, 0.505746, 0.155203, 0.028845, 
Output 2: 0.0294032, 0.970597, 

677
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2439!
23-Nov-2024 05:51:45 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511854_720_20231203T171034Z_gen_g4_detsim_hitreco__20240509T201009Z_reco2_reco_data_2024-11-23T_034322Z.root"
23-Nov-2024 05:51:45 CET  Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/86/0d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511854_720_20231203T171034Z_gen_g4_detsim_hitreco__20240509T201009Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              3.07387       36.2087       480.387       13.814        73.6875        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0428551     0.0806245     0.152296      0.0831701     0.0272723       100    
reco:linecluster:LineCluster                           0.386521       0.61026       9.37908      0.408367       1.0816         100    
reco:trajcluster:TrajCluster                           0.125824       1.02438       28.549       0.140261       4.01362        100    
reco:pandora:StandardPandora                            1.49352       3.59791       78.7879       1.79608       8.78232        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000170434   0.00770037     0.106407     0.00181887     0.0182974       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000232905    0.0226181     0.475794     0.00363224     0.0657165       100    
reco:pandoracalo:Calorimetry                          0.000127833    0.0212822     0.522139     0.00152389     0.0735272       100    
reco:pandorapid:Chi2ParticleID                        4.7282e-05    0.000328049    0.0129243    0.000163813    0.0012746       100    
reco:trkshowersplit:TrackShowerHits                   0.000115723    0.0245921     0.563927     0.000999022    0.0854606       100    
reco:pmtrack:PMAlgTrackMaker                           0.0016789      2.9643        57.5503      0.263411       8.60279        100    
reco:pmtrackcalo:Calorimetry                          9.3302e-05     0.0120973     0.256922      0.0012065     0.0404219       100    
reco:pmtrackpid:Chi2ParticleID                        4.3801e-05    0.000381123   0.00414591    0.000178809   0.000665676      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000276417     2.15879       33.7108       0.43539       5.16006        100    
reco:pmtrajfitcalo:Calorimetry                        8.8142e-05     0.0220309     0.523548     0.00204173     0.0753428       100    
reco:pmtrajfitpid:Chi2ParticleID                      3.5661e-05    0.000242315   0.00118728     0.0001949    0.000220413      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00158539      4.24161       116.202      0.354076       14.7401        100    
reco:pmtracktccalo:Calorimetry                        9.6502e-05    0.00863231     0.155554     0.00118239     0.0242817       100    
reco:pmtracktcpid:Chi2ParticleID                       4.038e-05    0.000355323   0.00318166    0.00019056    0.000529441      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000248866     2.3649        51.5206      0.425696       7.03352        100    
reco:pmtrajfittccalo:Calorimetry                      8.6312e-05     0.0194992     0.483394     0.00158096     0.0691307       100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.6191e-05    0.00049302    0.00530121    0.000205745   0.000893383      100    
reco:blurredcluster:BlurredClustering                 0.00646862      1.13588       10.8128      0.368425       1.83896        100    
reco:emtrkmichelid:EmTrackMichelId                     0.181329       13.5596        161.8        3.87754       27.6079        100    
reco:emshower:EMShower                                0.000344638    0.0389508     0.408467     0.00775997     0.0623343       100    
reco:cvnmap:CVNMapper                                  3.26e-05      0.0112762     0.0490194    0.00852051     0.0113302       100    
reco:cvneva:CVNEvaluator                              2.3241e-05      0.88693       1.69296       1.20664      0.573343        100    
reco:energyrecnumu:EnergyReco                         0.000512062    0.0496825      1.82098     0.00572825     0.195919        100    
reco:energyrecnue:EnergyReco                          0.000191054   0.00279318     0.0368947    0.000797918   0.00584031       100    
reco:energyrecnc:EnergyReco                            0.0024018    0.00565945     0.0422525    0.00387144    0.00602387       100    
reco:energyrecnumurange:EnergyReco                    0.000393269    0.0471343      1.81144     0.00379008     0.195268        100    
reco:energyrecnumumcs:EnergyReco                      0.00131437     0.0489722      1.81174     0.00485755     0.194679        100    
reco:opdec:Deconvolution                               0.133995      0.408939      0.713055      0.414467      0.121138        100    
reco:ophitspe:OpHitFinderDeco                           0.4344        2.49486       4.54251       2.41533      0.897919        100    
reco:opflash:OpFlashFinder                            0.000260376   0.00172929    0.00731914    0.00162356    0.000914025      100    
reco:rns:RandomNumberSaver                             2.144e-05    4.39193e-05   0.000511902   3.5966e-05    4.92664e-05      100    
reco:anglereconue:NuAngularReco                       0.000197984   0.00176645     0.0316746    0.000448785   0.00451415       100    
reco:anglereconumu:NuAngularReco                      8.7302e-05    0.00152413     0.0305218    0.000271757   0.00437788       100    
reco:anglereconuepfps:NuAngularReco                   0.000160514   0.00172202     0.0310977    0.00043146    0.00446076       100    
reco:anglereconumupfps:NuAngularReco                  0.000146363   0.00164001     0.030814     0.000380849   0.00448443       100    
[art]:TriggerResults:TriggerResultInserter             1.05e-05     1.68972e-05   8.9972e-05     1.51e-05     8.81944e-06      100    
end_path:out1:RootOutput                               3.17e-06     6.67453e-06   2.7801e-05     5.015e-06    4.20314e-06      100    
end_path:out1:RootOutput(write)                        0.0192101     0.323808       1.26964      0.324274      0.224693        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21430.3 MB
  Peak resident set size usage (VmHWM): 2275.23 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511854_720_20231203T171034Z_gen_g4_detsim_hitreco__20240509T201009Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511854_720_20231203T171034Z_gen_g4_detsim_hitreco__20240509T201009Z_reco2_reco_2024-11-23T_034322Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511854_720_20231203T171034Z_gen_g4_detsim_hitreco__20240509T201009Z_reco2_reco_data_2024-11-23T_034322Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work