Jobsub ID 301669.111@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301669.111@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 03:17:34
Site	UK_Durham
Entry	DUNE_UK_SGridDurham_ce3
Last heartbeat	2024-11-23 04:39:56
From worker node	Hostname	n260.dur.scotgrid.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 03:19:54
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_412_20231203T114247Z_gen_g4_detsim_hitreco__20240509T195605Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (4780s)
	CPU time	1h (4302s = 89%)
Outputting started	2024-11-23 04:39:35
Output files
Finished	2024-11-23 04:39:52
List job events Wrapper job log
Jobscript log (last 10,000 characters)

25 GMT run: 6411871 subRun: 1 event: 41300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:39:25 GMT run: 6411871 subRun: 1 event: 41300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:39:25 GMT run: 6411871 subRun: 1 event: 41300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:39:25 GMT run: 6411871 subRun: 1 event: 41300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:39:25 GMT run: 6411871 subRun: 1 event: 41300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:39:25 GMT run: 6411871 subRun: 1 event: 41300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:39:25 GMT run: 6411871 subRun: 1 event: 41300
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 133, 205, 161
minwire 0: 203
minwire 1: 2157
minwire 2: 94
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.156858, 0.00608982, 0.837052, 
Output 1: 0.398395, 0.591479, 0.00942695, 0.000698707, 
Output 2: 0.45615, 0.54385, 

396
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1104!
23-Nov-2024 04:39:34 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_412_20231203T114247Z_gen_g4_detsim_hitreco__20240509T195605Z_reco2_reco_data_2024-11-23T_031959Z.root"
23-Nov-2024 04:39:34 GMT  Closed input file "root://mover.pp.rl.ac.uk:1094/pnfs/pp.rl.ac.uk/data/dune/fardet-hd/bb/d0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_412_20231203T114247Z_gen_g4_detsim_hitreco__20240509T195605Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.876104       43.0293       622.458       8.77916       105.248        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0179997     0.0390836     0.0993557     0.0372426     0.014043        100    
reco:linecluster:LineCluster                           0.101485       1.2429        43.2085      0.185886       5.59726        100    
reco:trajcluster:TrajCluster                           0.0512319      2.23104       58.3893      0.0584307      8.61148        100    
reco:pandora:StandardPandora                           0.0615374      3.02448       51.6618       1.22927       7.0077         100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000162716   0.00701339     0.0866394    0.00103779     0.0156225       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000239742    0.0116706     0.166137     0.00217516     0.023539        100    
reco:pandoracalo:Calorimetry                          0.000118673    0.0371165      0.80791     0.000726239    0.127847        100    
reco:pandorapid:Chi2ParticleID                        4.5266e-05    0.00017688    0.00061022    0.00012488    0.000133529      100    
reco:trkshowersplit:TrackShowerHits                    8.501e-05     0.0585382      1.47815     0.000598268    0.209397        100    
reco:pmtrack:PMAlgTrackMaker                          0.00143053      5.62783       139.648      0.127607       20.1426        100    
reco:pmtrackcalo:Calorimetry                          8.7765e-05     0.0156011     0.288043     0.000615005    0.0489545       100    
reco:pmtrackpid:Chi2ParticleID                        3.5687e-05    0.000388315   0.00327627    0.000138821   0.000664105      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000229662     4.1628        76.4915      0.191005       12.4719        100    
reco:pmtrajfitcalo:Calorimetry                         8.508e-05     0.0380075     0.805584     0.00122626     0.127664        100    
reco:pmtrajfitpid:Chi2ParticleID                      3.6548e-05    0.000231233   0.00107644    0.00016727    0.000206108      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00117576      5.95305       108.291      0.107532       18.2092        100    
reco:pmtracktccalo:Calorimetry                        8.7014e-05    0.00841349     0.124881     0.000697865    0.0223887       100    
reco:pmtracktcpid:Chi2ParticleID                      3.5377e-05    0.000335016   0.00234507    0.000153324   0.000482248      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000202141     2.38263       34.2299      0.209843       6.11606        100    
reco:pmtrajfittccalo:Calorimetry                      8.0171e-05     0.024978      0.433002     0.000870485     0.07489        100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.5557e-05    0.000495645   0.00398408    0.000155959   0.000826651      100    
reco:blurredcluster:BlurredClustering                 9.7624e-05      1.04111       7.98984       0.35909       1.62864        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000147118     13.5677       127.691       2.24252       27.9285        100    
reco:emshower:EMShower                                0.000218572    0.0348751     0.400155     0.00645388     0.0685015       100    
reco:cvnmap:CVNMapper                                  2.69e-05      0.0101574     0.0406646    0.00724716     0.010189        100    
reco:cvneva:CVNEvaluator                               2.146e-05     0.620265       1.06221      0.858408      0.408868        100    
reco:energyrecnumu:EnergyReco                         0.000419541    0.0368954     0.703859      0.0043348     0.106123        100    
reco:energyrecnue:EnergyReco                          0.000184017   0.00223499     0.0249085    0.000409597   0.00463258       100    
reco:energyrecnc:EnergyReco                           0.00210205    0.00488815     0.0289592    0.00305281    0.00497039       100    
reco:energyrecnumurange:EnergyReco                    0.000214254    0.0347992     0.705525     0.00285334     0.105684        100    
reco:energyrecnumumcs:EnergyReco                      0.00118553     0.0366686     0.707221     0.00352904     0.105504        100    
reco:opdec:Deconvolution                               0.0842781     0.294556      0.594637       0.2894       0.109746        100    
reco:ophitspe:OpHitFinderDeco                          0.152874       2.2031        4.60835       2.17868       1.02958        100    
reco:opflash:OpFlashFinder                            0.000221768   0.00160374    0.00480373    0.00153541    0.000837071      100    
reco:rns:RandomNumberSaver                            2.2613e-05    4.15249e-05   0.000374526   3.56675e-05   3.5427e-05       100    
reco:anglereconue:NuAngularReco                       0.000203022   0.00171504     0.0199387    0.000373979   0.00366519       100    
reco:anglereconumu:NuAngularReco                      0.000109266   0.00148016     0.018411     0.000220676   0.00352065       100    
reco:anglereconuepfps:NuAngularReco                   0.000163779   0.00163721     0.0184487    0.000348731    0.0034702       100    
reco:anglereconumupfps:NuAngularReco                  0.000139844    0.0015621     0.0195335    0.000278314   0.00359916       100    
[art]:TriggerResults:TriggerResultInserter            1.1542e-05    1.48982e-05   6.0393e-05    1.3916e-05    5.16274e-06      100    
end_path:out1:RootOutput                               2.915e-06    6.50003e-06   2.1311e-05    6.0665e-06    2.98833e-06      100    
end_path:out1:RootOutput(write)                        0.0110014      0.26507      0.998792      0.251529      0.209002        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 38408.3 MB
  Peak resident set size usage (VmHWM): 2350.51 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_412_20231203T114247Z_gen_g4_detsim_hitreco__20240509T195605Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_412_20231203T114247Z_gen_g4_detsim_hitreco__20240509T195605Z_reco2_reco_2024-11-23T_031959Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_412_20231203T114247Z_gen_g4_detsim_hitreco__20240509T195605Z_reco2_reco_data_2024-11-23T_031959Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work