Jobsub ID 108787.84@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108787.84@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:19:04
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_dissel_multicore
Last heartbeat	2024-11-23 06:50:47
From worker node	Hostname	wn-sate-002.farm.nikhef.nl
	cpuinfo	AMD EPYC 7551P 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:20:30
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74531892_648_20231207T040538Z_gen_g4_detsim_hitreco__20240510T050048Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (5402s)
	CPU time	1h (5227s = 96%)
Outputting started	2024-11-23 06:50:32
Output files
Finished	2024-11-23 06:50:47
List job events Wrapper job log
Jobscript log (last 10,000 characters)

Dist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:50:16 CET run: 74531892 subRun: 1 event: 64900
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:50:16 CET run: 74531892 subRun: 1 event: 64900
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:50:27 CET run: 74531892 subRun: 1 event: 64900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:50:27 CET run: 74531892 subRun: 1 event: 64900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:50:27 CET run: 74531892 subRun: 1 event: 64900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:50:27 CET run: 74531892 subRun: 1 event: 64900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:50:27 CET run: 74531892 subRun: 1 event: 64900
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 513, 649, 465
minwire 0: 1101
minwire 1: 2218
minwire 2: 638
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0253575, 0.948969, 0.0256737, 
Output 1: 0.161029, 0.524147, 0.266202, 0.0486222, 
Output 2: 0.0292176, 0.970782, 

710
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2748!
23-Nov-2024 07:50:31 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74531892_648_20231207T040538Z_gen_g4_detsim_hitreco__20240510T050048Z_reco2_reco_data_2024-11-23T_052033Z.root"
23-Nov-2024 07:50:31 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/af/c9/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74531892_648_20231207T040538Z_gen_g4_detsim_hitreco__20240510T050048Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.685143       51.9328       1223.58       10.0292       156.204        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0175145      0.04644       0.28308      0.040086      0.0369386       100    
reco:linecluster:LineCluster                           0.136916      0.919139       26.6986      0.169727       3.78342        100    
reco:trajcluster:TrajCluster                           0.0432635      3.79121       118.851      0.0574405      16.0713        100    
reco:pandora:StandardPandora                           0.0548522      7.87674       383.244       1.63864       39.0477        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000150873    0.0118858      0.30848     0.00199919     0.0376483       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000206648    0.0133462     0.126554     0.00327314     0.0261497       100    
reco:pandoracalo:Calorimetry                          0.000121728    0.0519724      1.81004     0.00125254     0.219223        100    
reco:pandorapid:Chi2ParticleID                        4.7499e-05    0.000187265   0.00152264    0.000136186   0.000198058      100    
reco:trkshowersplit:TrackShowerHits                   0.000101531    0.0588027      1.45012     0.000901083    0.214676        100    
reco:pmtrack:PMAlgTrackMaker                          0.00153207      6.10455       137.312      0.175241       21.6273        100    
reco:pmtrackcalo:Calorimetry                          9.3646e-05     0.0289879     0.968262     0.00098518     0.115293        100    
reco:pmtrackpid:Chi2ParticleID                        3.7581e-05    0.000432298    0.0051808    0.000141005   0.000846539      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000215985     3.85996       110.355      0.251881       13.6527        100    
reco:pmtrajfitcalo:Calorimetry                        8.4228e-05     0.0521186      1.75901     0.00154408     0.215869        100    
reco:pmtrajfitpid:Chi2ParticleID                       3.742e-05    0.000215286   0.00113954    0.000157616   0.000197825      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00136978      6.38116       149.342      0.235916       22.099         100    
reco:pmtracktccalo:Calorimetry                        8.7655e-05     0.0157199     0.387223     0.000963284    0.0526741       100    
reco:pmtracktcpid:Chi2ParticleID                      3.6138e-05    0.000370409   0.00354855    0.000147406   0.000621902      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000199905     3.07548       54.2651      0.329036       8.95802        100    
reco:pmtrajfittccalo:Calorimetry                      8.2485e-05     0.0408492      1.01543     0.00128244      0.14951        100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.6388e-05    0.000539986    0.0056887    0.000166313   0.00106175       100    
reco:blurredcluster:BlurredClustering                 7.2557e-05      1.26608       10.3441      0.619478       1.80066        100    
reco:emtrkmichelid:EmTrackMichelId                     0.023868       15.1064       220.932       2.93188       35.219         100    
reco:emshower:EMShower                                0.000262022    0.0422304     0.439487     0.00788732     0.0779792       100    
reco:cvnmap:CVNMapper                                 2.3835e-05    0.00942977     0.0441571     0.0084978    0.00820889       100    
reco:cvneva:CVNEvaluator                              1.9617e-05     0.751275       1.40729      0.982664       0.45032        100    
reco:energyrecnumu:EnergyReco                         0.000339668    0.0822242      2.14061     0.00506133     0.279792        100    
reco:energyrecnue:EnergyReco                          0.000185218   0.00379733     0.0930979    0.000554386    0.0115646       100    
reco:energyrecnc:EnergyReco                           0.00283957    0.00653881     0.0982856    0.00334379     0.0119574       100    
reco:energyrecnumurange:EnergyReco                    0.000198864    0.0796145      2.12471     0.00301286     0.278248        100    
reco:energyrecnumumcs:EnergyReco                      0.000978518    0.0811173      2.11688      0.0041094     0.276549        100    
reco:opdec:Deconvolution                               0.0612739     0.316296      0.517062      0.304355      0.0972823       100    
reco:ophitspe:OpHitFinderDeco                          0.226135       1.54713       2.85195       1.46585      0.601071        100    
reco:opflash:OpFlashFinder                            0.000236764   0.00143581    0.00336849    0.00128966    0.000664688      100    
reco:rns:RandomNumberSaver                            1.8996e-05    2.97469e-05   0.000307437   2.49665e-05   2.86508e-05      100    
reco:anglereconue:NuAngularReco                       0.00015436    0.00295609     0.0854186    0.000379202    0.0101734       100    
reco:anglereconumu:NuAngularReco                      8.9829e-05     0.0027974     0.0837208    0.000255755    0.0100329       100    
reco:anglereconuepfps:NuAngularReco                   0.000147728   0.00296605     0.0842406    0.000377059    0.0101168       100    
reco:anglereconumupfps:NuAngularReco                  0.000140915   0.00290702     0.0844062    0.000337294    0.0100932       100    
[art]:TriggerResults:TriggerResultInserter             1.071e-05    1.32996e-05    6.942e-05    1.19375e-05   5.98654e-06      100    
end_path:out1:RootOutput                               2.805e-06    5.50657e-06   2.0388e-05     4.273e-06    2.68924e-06      100    
end_path:out1:RootOutput(write)                        0.0161827     0.295485       1.70075      0.250951      0.277436        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8516.63 MB
  Peak resident set size usage (VmHWM): 2543.02 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74531892_648_20231207T040538Z_gen_g4_detsim_hitreco__20240510T050048Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74531892_648_20231207T040538Z_gen_g4_detsim_hitreco__20240510T050048Z_reco2_reco_2024-11-23T_052033Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74531892_648_20231207T040538Z_gen_g4_detsim_hitreco__20240510T050048Z_reco2_reco_data_2024-11-23T_052033Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work