Jobsub ID 301702.3@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301702.3@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:26:48
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce506
Last heartbeat	2024-11-23 05:09:48
From worker node	Hostname	b9p05p4086.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:27:19
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_180_20231203T043750Z_gen_g4_detsim_hitreco__20240508T073430Z_reco2.root
Jobscript	Exit code	0
	Real time	42m (2531s)
	CPU time	44m (2648s = 104%)
Outputting started	2024-11-23 05:09:31
Output files
Finished	2024-11-23 05:09:48
List job events Wrapper job log
Jobscript log (last 10,000 characters)

t: 18100
1st wire C:0 T:21 P:2 W:680 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:09:23 CET run: 50572032 subRun: 1 event: 18100
1st wire C:0 T:21 P:2 W:1042 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:09:23 CET run: 50572032 subRun: 1 event: 18100
1st wire C:0 T:21 P:2 W:1041 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:09:23 CET run: 50572032 subRun: 1 event: 18100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:09:26 CET run: 50572032 subRun: 1 event: 18100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:09:26 CET run: 50572032 subRun: 1 event: 18100
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 82, 83, 56
minwire 0: 2600
minwire 1: 849
minwire 2: 2325
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max wires due to vertex determination failure: 52, 251
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max wires due to vertex determination failure: 63, 262
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.986576, 0.000556172, 0.0128677, 
Output 1: 0.0116556, 0.0678855, 0.104565, 0.815894, 
Output 2: 0.977666, 0.0223344, 

557
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1777!
23-Nov-2024 06:09:30 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_180_20231203T043750Z_gen_g4_detsim_hitreco__20240508T073430Z_reco2_reco_data_2024-11-23T_042724Z.root"
23-Nov-2024 06:09:30 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/6b/2c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_180_20231203T043750Z_gen_g4_detsim_hitreco__20240508T073430Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.14826       23.3778       332.441       6.85616       48.3617        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0173171     0.0391512     0.0710958     0.0396889     0.0113596       100    
reco:linecluster:LineCluster                           0.147705       0.35711       7.50208      0.175842      0.885932        100    
reco:trajcluster:TrajCluster                           0.0477769     0.700221       19.1983      0.055671       2.40489        100    
reco:pandora:StandardPandora                           0.786567       1.79355       24.5431       1.16211       2.71682        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000120131   0.00540482     0.0806812    0.000870912    0.0124615       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000153492   0.00803994     0.0810438    0.00165165     0.0149116       100    
reco:pandoracalo:Calorimetry                          8.7242e-05     0.019113      0.755355     0.000577342    0.0816487       100    
reco:pandorapid:Chi2ParticleID                         3.463e-05    0.000161095   0.00197574    0.000103342   0.000219278      100    
reco:trkshowersplit:TrackShowerHits                   8.2261e-05     0.022978      0.401694     0.000412995    0.0718315       100    
reco:pmtrack:PMAlgTrackMaker                          0.000972053     2.43332       60.7861      0.0859216      7.52893        100    
reco:pmtrackcalo:Calorimetry                          6.0191e-05     0.0097317      0.28289     0.000466856    0.0350378       100    
reco:pmtrackpid:Chi2ParticleID                        2.3991e-05    0.000239188   0.00207359    0.000101492   0.000356998      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000155802     1.82446       25.4898      0.152023       4.51633        100    
reco:pmtrajfitcalo:Calorimetry                        5.6441e-05     0.0177729     0.697954     0.00071043     0.0748788       100    
reco:pmtrajfitpid:Chi2ParticleID                      2.3471e-05    0.000158569   0.00100662    0.000113331   0.000151757      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00079793      2.73399       59.9163      0.109248       8.43917        100    
reco:pmtracktccalo:Calorimetry                        5.6711e-05    0.00598837     0.127873     0.000485266    0.0176161       100    
reco:pmtracktcpid:Chi2ParticleID                       2.319e-05    0.00021995    0.00154316    0.000112806   0.000305842      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000129422     1.34081       22.7279      0.189287       3.39577        100    
reco:pmtrajfittccalo:Calorimetry                      5.3001e-05     0.0145987     0.328863     0.000638243    0.0496937       100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.2281e-05    0.000304739   0.00362336    0.000126381   0.00053646       100    
reco:blurredcluster:BlurredClustering                  0.0042279     0.804805       6.41896      0.333503       1.18287        100    
reco:emtrkmichelid:EmTrackMichelId                     0.169405       8.38129       99.7007       1.61672       17.7587        100    
reco:emshower:EMShower                                0.000234323    0.0240191     0.222562      0.0033754     0.0416321       100    
reco:cvnmap:CVNMapper                                  2.173e-05    0.00873403     0.037702     0.00565417    0.00873783       100    
reco:cvneva:CVNEvaluator                              1.5581e-05     0.561169       1.11098      0.726312      0.414822        100    
reco:energyrecnumu:EnergyReco                         0.000351604    0.0352181     0.973323     0.00323099     0.123154        100    
reco:energyrecnue:EnergyReco                          0.000119601   0.00202841     0.0282109    0.000403495   0.00455549       100    
reco:energyrecnc:EnergyReco                           0.00150166    0.00412421     0.0321293    0.00282053    0.00515694       100    
reco:energyrecnumurange:EnergyReco                    0.000163692    0.0300827      0.68029     0.00174717     0.100755        100    
reco:energyrecnumumcs:EnergyReco                      0.00138789     0.0324803      0.72514     0.00301269     0.106071        100    
reco:opdec:Deconvolution                               0.115489      0.266634      0.503649      0.252377      0.0923477       100    
reco:ophitspe:OpHitFinderDeco                          0.410832       1.68852       4.14299       1.48367      0.883786        100    
reco:opflash:OpFlashFinder                            0.000340344   0.00126242    0.00474517    0.00117908    0.000681863      100    
reco:rns:RandomNumberSaver                             1.696e-05    3.57206e-05   0.000431386   3.07255e-05   4.09353e-05      100    
reco:anglereconue:NuAngularReco                       0.000135962   0.00150821     0.0250297    0.000318069   0.00356842       100    
reco:anglereconumu:NuAngularReco                      6.1321e-05    0.00124988     0.0242742    0.000158972   0.00334343       100    
reco:anglereconuepfps:NuAngularReco                   0.000104791   0.00138112     0.024567     0.000290604   0.00334924       100    
reco:anglereconumupfps:NuAngularReco                  8.7861e-05    0.00130311     0.0243973    0.000214067   0.00336861       100    
[art]:TriggerResults:TriggerResultInserter             8.43e-06     1.20655e-05   5.3411e-05    1.09455e-05   4.95027e-06      100    
end_path:out1:RootOutput                               2.89e-06     5.66313e-06    2.22e-05      5.555e-06    2.3947e-06       100    
end_path:out1:RootOutput(write)                        0.0232857     0.202939       0.85304       0.18548      0.169647        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21520 MB
  Peak resident set size usage (VmHWM): 2110.77 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_180_20231203T043750Z_gen_g4_detsim_hitreco__20240508T073430Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_180_20231203T043750Z_gen_g4_detsim_hitreco__20240508T073430Z_reco2_reco_2024-11-23T_042724Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50572032_180_20231203T043750Z_gen_g4_detsim_hitreco__20240508T073430Z_reco2_reco_data_2024-11-23T_042724Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work