Jobsub ID 108787.60@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108787.60@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:19:04
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_klomp_multicore
Last heartbeat	2024-11-23 06:17:04
From worker node	Hostname	wn-sate-006.farm.nikhef.nl
	cpuinfo	AMD EPYC 7551P 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:19:42
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_1024_20231203T154557Z_gen_g4_detsim_hitreco__20240508T073322Z_reco2.root
Jobscript	Exit code	0
	Real time	57m (3427s)
	CPU time	54m (3254s = 94%)
Outputting started	2024-11-23 06:16:49
Output files
Finished	2024-11-23 06:17:04
List job events Wrapper job log
Jobscript log (last 10,000 characters)

500
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:16:25 CET run: 74511302 subRun: 1 event: 102500
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:16:25 CET run: 74511302 subRun: 1 event: 102500
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:16:25 CET run: 74511302 subRun: 1 event: 102500
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:16:25 CET run: 74511302 subRun: 1 event: 102500
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:16:25 CET run: 74511302 subRun: 1 event: 102500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:16:43 CET run: 74511302 subRun: 1 event: 102500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:16:43 CET run: 74511302 subRun: 1 event: 102500
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 754, 563, 637
minwire 0: 1700
minwire 1: 994
minwire 2: 1578
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.00130174, 0.995661, 0.00303736, 
Output 1: 0.109841, 0.827267, 0.0524744, 0.0104172, 
Output 2: 0.726646, 0.273354, 

672
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2595!
23-Nov-2024 07:16:48 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_1024_20231203T154557Z_gen_g4_detsim_hitreco__20240508T073322Z_reco2_reco_data_2024-11-23T_051945Z.root"
23-Nov-2024 07:16:48 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/e3/40/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_1024_20231203T154557Z_gen_g4_detsim_hitreco__20240508T073322Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.860866       32.1653       417.491       10.4744       70.6803        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0154678     0.0451003     0.371256      0.0421027     0.0358343       100    
reco:linecluster:LineCluster                           0.136255      0.451632       11.815       0.157101       1.55795        100    
reco:trajcluster:TrajCluster                           0.043405       1.36993       39.6877      0.0562369      5.91928        100    
reco:pandora:StandardPandora                           0.0552572      3.83304       123.811       1.67816       12.7993        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.00016496    0.00762078     0.137442     0.00178417     0.0190712       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000224822    0.0103565     0.100116     0.00280915     0.0171422       100    
reco:pandoracalo:Calorimetry                          0.000120847    0.0224506     0.434273     0.00113407     0.0747326       100    
reco:pandorapid:Chi2ParticleID                         4.811e-05    0.000172353   0.000733146   0.000131427   0.000129219      100    
reco:trkshowersplit:TrackShowerHits                   9.2775e-05     0.0313451      0.76341     0.000877887     0.11866        100    
reco:pmtrack:PMAlgTrackMaker                          0.00157402      2.78948       56.6586      0.166075       8.23058        100    
reco:pmtrackcalo:Calorimetry                          8.9879e-05     0.0106451     0.263465     0.00106013     0.0358958       100    
reco:pmtrackpid:Chi2ParticleID                        3.8191e-05    0.000339076   0.00318406    0.000163397   0.000537154      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000262413     2.63242       40.8023      0.332057       7.26583        100    
reco:pmtrajfitcalo:Calorimetry                         9.09e-05      0.0231576     0.438298     0.00168864     0.0753631       100    
reco:pmtrajfitpid:Chi2ParticleID                      3.9203e-05    0.000207072   0.000985029   0.000165225   0.000166408      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00148231      3.25485       56.3185      0.255729       9.53963        100    
reco:pmtracktccalo:Calorimetry                        0.000101691   0.00775565     0.144614      0.0010117     0.021808        100    
reco:pmtracktcpid:Chi2ParticleID                      4.1908e-05    0.000316838   0.00245454    0.00017062    0.000419662      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000244329     1.88529       26.9999      0.288661       4.41631        100    
reco:pmtrajfittccalo:Calorimetry                      8.9177e-05     0.0208277     0.469227     0.00133861     0.072702        100    
reco:pmtrajfittcpid:Chi2ParticleID                     3.739e-05    0.000448659   0.00407117    0.000176191   0.00075419       100    
reco:blurredcluster:BlurredClustering                 0.000770196     1.12584       7.43404      0.666915       1.49286        100    
reco:emtrkmichelid:EmTrackMichelId                     0.0326472      11.6317       118.355       2.98213       22.9092        100    
reco:emshower:EMShower                                0.000307898    0.0265915     0.263976      0.0094345     0.0414508       100    
reco:cvnmap:CVNMapper                                 2.5087e-05     0.0102899     0.0332543    0.00910795    0.00877385       100    
reco:cvneva:CVNEvaluator                              1.9377e-05     0.726059       1.89764       1.00164      0.485498        100    
reco:energyrecnumu:EnergyReco                         0.00034109     0.0494159     0.820592     0.00525942     0.147473        100    
reco:energyrecnue:EnergyReco                          0.000185087   0.00233732     0.0443148    0.000568402   0.00583347       100    
reco:energyrecnc:EnergyReco                           0.00283813     0.0052676     0.0486987    0.00345988    0.00612918       100    
reco:energyrecnumurange:EnergyReco                    0.000230402    0.0470802     0.813921     0.00312464      0.14619        100    
reco:energyrecnumumcs:EnergyReco                      0.00143003     0.0488992     0.816194     0.00407042     0.145871        100    
reco:opdec:Deconvolution                               0.0817377     0.302197      0.563259      0.302285      0.0954186       100    
reco:ophitspe:OpHitFinderDeco                          0.279062       1.51721       2.72653       1.51982      0.586273        100    
reco:opflash:OpFlashFinder                            0.000358162   0.00146369    0.00324788    0.00139141    0.000624585      100    
reco:rns:RandomNumberSaver                            1.9427e-05    3.32043e-05   0.000253807   2.8388e-05    2.40625e-05      100    
reco:anglereconue:NuAngularReco                       0.000172994   0.00172352     0.0384393    0.000355747   0.00483167       100    
reco:anglereconumu:NuAngularReco                      9.3134e-05    0.00158348     0.0377518    0.000205996   0.00477392       100    
reco:anglereconuepfps:NuAngularReco                   0.000151545   0.00172769     0.0377325    0.000344115   0.00478337       100    
reco:anglereconumupfps:NuAngularReco                  0.000146715   0.00168324     0.0381505    0.000316404   0.00481641       100    
[art]:TriggerResults:TriggerResultInserter            1.1462e-05    1.39405e-05   4.8791e-05    1.28095e-05   4.38205e-06      100    
end_path:out1:RootOutput                               3.286e-06    6.27796e-06   2.0128e-05    5.4905e-06    2.79778e-06      100    
end_path:out1:RootOutput(write)                        0.0200147     0.264494       1.04423      0.254974      0.200552        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8242.27 MB
  Peak resident set size usage (VmHWM): 2270.79 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_1024_20231203T154557Z_gen_g4_detsim_hitreco__20240508T073322Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_1024_20231203T154557Z_gen_g4_detsim_hitreco__20240508T073322Z_reco2_reco_2024-11-23T_051945Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74511302_1024_20231203T154557Z_gen_g4_detsim_hitreco__20240508T073322Z_reco2_reco_data_2024-11-23T_051945Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work