Jobsub ID 108671.24@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108671.24@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 01:36:07
Site	US_UChicago
Entry	Engage_US_MWT2_uiuc_gk02_condce_mcore
Last heartbeat	2024-11-23 02:49:34
From worker node	Hostname	iut2-c398.iu.edu
	cpuinfo	AMD EPYC 7443 24-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	86400 (24 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 01:53:06
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50542963_648_20231202T061452Z_gen_g4_detsim_hitreco__20240507T205658Z_reco2.root
Jobscript	Exit code	0
	Real time	56m (3373s)
	CPU time	52m (3170s = 93%)
Outputting started	2024-11-23 02:49:20
Output files
Finished	2024-11-23 02:49:34
List job events Wrapper job log
Jobscript log (last 10,000 characters)

lure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 21:49:13 EST run: 50542963 subRun: 1 event: 64900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 21:49:13 EST run: 50542963 subRun: 1 event: 64900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 21:49:13 EST run: 50542963 subRun: 1 event: 64900
1st wire C:0 T:1 P:2 W:670 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  22-Nov-2024 21:49:15 EST run: 50542963 subRun: 1 event: 64900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  22-Nov-2024 21:49:15 EST run: 50542963 subRun: 1 event: 64900
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 44, 41, 49
minwire 0: 457
minwire 1: 2520
minwire 2: 20
Used alternate method to get min and max wires due to vertex determination failure: 105, 304
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max wires due to vertex determination failure: 1, 200
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max wires due to vertex determination failure: 198, 397
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.988726, 0.00171658, 0.00955786, 
Output 1: 0.877206, 0.106486, 0.0137991, 0.00250918, 
Output 2: 0.98245, 0.0175501, 

309
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 861!
22-Nov-2024 21:49:18 EST  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50542963_648_20231202T061452Z_gen_g4_detsim_hitreco__20240507T205658Z_reco2_reco_data_2024-11-23T_015311Z.root"
22-Nov-2024 21:49:18 EST  Closed input file "root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/fardet-hd/ec/13/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50542963_648_20231202T061452Z_gen_g4_detsim_hitreco__20240507T205658Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.82162       30.652        1102.83       8.05392       113.538        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0266882     0.104339      0.404677      0.100428      0.0621284       100    
reco:linecluster:LineCluster                           0.239227      0.671717       31.3513      0.260777       3.12051        100    
reco:trajcluster:TrajCluster                           0.0773464      1.08706       63.6343      0.085231       6.54601        100    
reco:pandora:StandardPandora                           0.899617       4.99169       315.271       1.21105       31.4535        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.00014349    0.00593022      0.19994     0.00109964     0.0211113       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000212119     0.01271      0.244391     0.00212553     0.0364558       100    
reco:pandoracalo:Calorimetry                          0.000113934    0.0254862      1.54136     0.000671913    0.155395        100    
reco:pandorapid:Chi2ParticleID                        3.8572e-05    0.000179196   0.00122582    0.000134052   0.000172945      100    
reco:trkshowersplit:TrackShowerHits                   0.000112552    0.0243077     0.936215     0.000568464    0.108972        100    
reco:pmtrack:PMAlgTrackMaker                          0.000977719     3.08119       168.039      0.0923956      17.0595        100    
reco:pmtrackcalo:Calorimetry                          7.1464e-05     0.0131065     0.716251     0.000704019    0.0728872       100    
reco:pmtrackpid:Chi2ParticleID                        2.7111e-05    0.000292269   0.00351852    0.000160472   0.000458591      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000220334     2.04538       63.9831       0.15944       7.41969        100    
reco:pmtrajfitcalo:Calorimetry                        7.1494e-05     0.0238412      1.36191     0.00107773     0.137962        100    
reco:pmtrajfitpid:Chi2ParticleID                      3.0357e-05    0.000196461   0.00116866    0.000149737   0.00016447       100    
reco:pmtracktc:PMAlgTrackMaker                        0.000775879     3.2798        161.216      0.152202       16.7701        100    
reco:pmtracktccalo:Calorimetry                        6.1346e-05     0.0074845     0.332963     0.000816075    0.0341968       100    
reco:pmtracktcpid:Chi2ParticleID                       2.646e-05    0.00027274    0.00261012    0.000163362   0.000370568      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000217468     1.57003       56.0086      0.229923       5.86367        100    
reco:pmtrajfittccalo:Calorimetry                      7.7656e-05     0.0163795     0.803951     0.000969127    0.0829699       100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.0567e-05    0.000376901   0.00614465    0.00017028    0.000717203      100    
reco:blurredcluster:BlurredClustering                 0.00530223     0.821462       8.62497      0.374402       1.34126        100    
reco:emtrkmichelid:EmTrackMichelId                      0.18999       9.63127       219.161       1.88042       25.3853        100    
reco:emshower:EMShower                                0.000252545    0.0228431     0.236501     0.00732578     0.0361274       100    
reco:cvnmap:CVNMapper                                 2.2843e-05    0.00856428     0.0357055    0.00657141    0.00835557       100    
reco:cvneva:CVNEvaluator                              1.6061e-05      0.57403       1.34071      0.761614      0.422361        100    
reco:energyrecnumu:EnergyReco                         0.000280678    0.0416725      1.81222     0.00364622     0.190118        100    
reco:energyrecnue:EnergyReco                          0.000107743   0.00187661     0.0552531    0.000453107   0.00581215       100    
reco:energyrecnc:EnergyReco                           0.00145093     0.0040812     0.0520977    0.00293361    0.00538897       100    
reco:energyrecnumurange:EnergyReco                    0.000139172    0.0409921      2.00655     0.00275427     0.207305        100    
reco:energyrecnumumcs:EnergyReco                      0.00160174     0.037711       1.53606     0.00331274      0.16276        100    
reco:opdec:Deconvolution                               0.124563       0.35783      0.957511      0.316438      0.169954        100    
reco:ophitspe:OpHitFinderDeco                          0.500555       1.91012       4.15564       1.85321       0.84487        100    
reco:opflash:OpFlashFinder                            0.000342825   0.00136076    0.00359245    0.00127136    0.000639461      100    
reco:rns:RandomNumberSaver                            2.0288e-05    4.90076e-05   0.000316906   4.5726e-05    3.13671e-05      100    
reco:anglereconue:NuAngularReco                       0.000173757    0.0016739     0.0653638    0.00036447    0.00672285       100    
reco:anglereconumu:NuAngularReco                      7.8728e-05    0.00140438     0.0593633    0.000185554   0.00614051       100    
reco:anglereconuepfps:NuAngularReco                   0.000120357   0.00159967     0.0623456    0.000341026   0.00642397       100    
reco:anglereconumupfps:NuAngularReco                  9.6601e-05    0.00151426     0.0638917    0.000252094   0.00657535       100    
[art]:TriggerResults:TriggerResultInserter             8.657e-06    1.54411e-05   5.4603e-05    1.41915e-05   6.04358e-06      100    
end_path:out1:RootOutput                               2.986e-06    6.73085e-06   2.5158e-05    6.6825e-06    3.30584e-06      100    
end_path:out1:RootOutput(write)                        0.0269356     0.228007       1.59099      0.218207      0.206396        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 17545.3 MB
  Peak resident set size usage (VmHWM): 2593.73 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50542963_648_20231202T061452Z_gen_g4_detsim_hitreco__20240507T205658Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50542963_648_20231202T061452Z_gen_g4_detsim_hitreco__20240507T205658Z_reco2_reco_2024-11-23T_015311Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50542963_648_20231202T061452Z_gen_g4_detsim_hitreco__20240507T205658Z_reco2_reco_data_2024-11-23T_015311Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work