Jobsub ID 301687.82@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301687.82@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 03:55:32
Site	UK_Durham
Entry	DUNE_UK_SGridDurham_ce3
Last heartbeat	2024-11-23 05:19:01
From worker node	Hostname	n259.dur.scotgrid.ac.uk
	cpuinfo	AMD EPYC 7763 64-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 03:57:57
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74521450_86_20231205T015411Z_gen_g4_detsim_hitreco__20240509T222209Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (4847s)
	CPU time	1h (4131s = 85%)
Outputting started	2024-11-23 05:18:44
Output files
Finished	2024-11-23 05:19:01
List job events Wrapper job log
Jobscript log (last 10,000 characters)

mshower@BeginModule  23-Nov-2024 05:18:35 GMT run: 74521450 subRun: 1 event: 8700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:18:35 GMT run: 74521450 subRun: 1 event: 8700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:18:35 GMT run: 74521450 subRun: 1 event: 8700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:18:35 GMT run: 74521450 subRun: 1 event: 8700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:18:35 GMT run: 74521450 subRun: 1 event: 8700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:18:35 GMT run: 74521450 subRun: 1 event: 8700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:18:35 GMT run: 74521450 subRun: 1 event: 8700
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 1614, 1575, 1505
minwire 0: 1947
minwire 1: 669
minwire 2: 1940
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0295261, 0.953059, 0.0174152, 
Output 1: 0.365785, 0.58783, 0.0418765, 0.00450867, 
Output 2: 0.120351, 0.879649, 

693
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2606!
23-Nov-2024 05:18:42 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74521450_86_20231205T015411Z_gen_g4_detsim_hitreco__20240509T222209Z_reco2_reco_data_2024-11-23T_035804Z.root"
23-Nov-2024 05:18:42 GMT  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/87/06/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74521450_86_20231205T015411Z_gen_g4_detsim_hitreco__20240509T222209Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.05497       41.8787       1194.79       13.2264       123.394        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0346241     0.0710999     0.123759      0.0750304     0.0207965       100    
reco:linecluster:LineCluster                           0.201466       1.11449       54.467       0.358758       5.42301        100    
reco:trajcluster:TrajCluster                           0.101165       2.01243       134.297      0.116491       13.418         100    
reco:pandora:StandardPandora                           0.109705       3.80267       186.22        1.48084       18.4044        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000139174    0.0102812      0.43262     0.00173219     0.0434838       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.00018992     0.0180972     0.206967     0.00381502     0.0373241       100    
reco:pandoracalo:Calorimetry                          0.000102895    0.0366004      1.85166     0.00145288     0.187371        100    
reco:pandorapid:Chi2ParticleID                        3.8773e-05    0.000214293    0.0017757    0.000148306   0.000239303      100    
reco:trkshowersplit:TrackShowerHits                   7.6715e-05     0.0505898      2.45292      0.0010022     0.252961        100    
reco:pmtrack:PMAlgTrackMaker                          0.00118709      3.80119       136.872      0.254208       14.4328        100    
reco:pmtrackcalo:Calorimetry                          7.2417e-05     0.0159238     0.740433     0.00113177     0.0751257       100    
reco:pmtrackpid:Chi2ParticleID                        2.7312e-05    0.000395181    0.0056063    0.000180016   0.000663508      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000182947     3.62433       151.026      0.466288       15.4998        100    
reco:pmtrajfitcalo:Calorimetry                        6.6727e-05     0.0375965      1.89992     0.00188049     0.191967        100    
reco:pmtrajfitpid:Chi2ParticleID                      2.5719e-05    0.000241775   0.00198482    0.000191518   0.000234074      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000898093     4.96007       190.596      0.254822       19.8703        100    
reco:pmtracktccalo:Calorimetry                        6.5935e-05     0.010406      0.365388     0.00136592     0.038229        100    
reco:pmtracktcpid:Chi2ParticleID                      2.5859e-05    0.000367141   0.00433925    0.000192035   0.000513232      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000151307     2.58518       68.7301      0.426273       7.40827        100    
reco:pmtrajfittccalo:Calorimetry                      5.8732e-05     0.0246304      1.11375     0.00144192     0.113444        100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.9075e-05    0.00052862    0.00759751    0.000213986   0.000919843      100    
reco:blurredcluster:BlurredClustering                 9.2305e-05      1.29807       10.6799      0.789514       1.60912        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000153481     14.2414       242.723       3.59114       28.9302        100    
reco:emshower:EMShower                                0.000225477    0.0343379     0.855813     0.00570256     0.0933729       100    
reco:cvnmap:CVNMapper                                 2.3274e-05     0.010988      0.0604636    0.00816131     0.0111179       100    
reco:cvneva:CVNEvaluator                              1.6071e-05     0.652561       1.0458       0.875208      0.409317        100    
reco:energyrecnumu:EnergyReco                         0.000404207    0.034739      0.779109     0.00537163     0.0918416       100    
reco:energyrecnue:EnergyReco                          0.000202344   0.00324543     0.124821     0.000752982    0.0124925       100    
reco:energyrecnc:EnergyReco                           0.00174954    0.00604011     0.131778     0.00366479     0.0129368       100    
reco:energyrecnumurange:EnergyReco                    0.000247369    0.0325827     0.772352      0.0035544     0.0911664       100    
reco:energyrecnumumcs:EnergyReco                      0.00131793     0.0344305     0.778374     0.00439083     0.0914739       100    
reco:opdec:Deconvolution                               0.119393      0.431698      0.670608      0.427848      0.125376        100    
reco:ophitspe:OpHitFinderDeco                          0.302336       2.60888       4.74964       2.63618       1.03982        100    
reco:opflash:OpFlashFinder                            0.000169933   0.00182109     0.0159054    0.00156939    0.00163103       100    
reco:rns:RandomNumberSaver                            1.6993e-05    4.30348e-05   0.000369551   3.9094e-05    3.50043e-05      100    
reco:anglereconue:NuAngularReco                       0.000146197   0.00246046     0.116389     0.00046658     0.0115997       100    
reco:anglereconumu:NuAngularReco                      7.7838e-05    0.00224881     0.115532     0.000315332    0.011528        100    
reco:anglereconuepfps:NuAngularReco                   0.000132831   0.00240724     0.116282     0.000439568    0.0115912       100    
reco:anglereconumupfps:NuAngularReco                  9.7956e-05    0.00235009     0.115892     0.000365137    0.0115595       100    
[art]:TriggerResults:TriggerResultInserter             9.177e-06    1.6264e-05    5.9843e-05    1.51535e-05    5.855e-06       100    
end_path:out1:RootOutput                               2.926e-06    6.69265e-06   2.7783e-05     6.873e-06    3.64381e-06      100    
end_path:out1:RootOutput(write)                        0.0141345     0.298683       1.74834      0.304863      0.223952        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 38428.2 MB
  Peak resident set size usage (VmHWM): 2361.17 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74521450_86_20231205T015411Z_gen_g4_detsim_hitreco__20240509T222209Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74521450_86_20231205T015411Z_gen_g4_detsim_hitreco__20240509T222209Z_reco2_reco_2024-11-23T_035804Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74521450_86_20231205T015411Z_gen_g4_detsim_hitreco__20240509T222209Z_reco2_reco_data_2024-11-23T_035804Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work