Jobsub ID 108787.56@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108787.56@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:19:04
Site	NL_NIKHEF
Entry	VIRGO_NL_NIKHEF_brug_multicore
Last heartbeat	2024-11-23 06:45:19
From worker node	Hostname	wn-choc-009.farm.nikhef.nl
	cpuinfo	Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	129600 (36 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:20:32
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74518431_469_20231204T214753Z_gen_g4_detsim_hitreco__20240510T030557Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (5075s)
	CPU time	1h (4890s = 96%)
Outputting started	2024-11-23 06:45:07
Output files
Finished	2024-11-23 06:45:19
List job events Wrapper job log
Jobscript log (last 10,000 characters)

nt: 47000
TuneFullTree failed.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:44:46 CET run: 74518431 subRun: 1 event: 47000
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:44:46 CET run: 74518431 subRun: 1 event: 47000
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:44:46 CET run: 74518431 subRun: 1 event: 47000
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:45:02 CET run: 74518431 subRun: 1 event: 47000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:45:02 CET run: 74518431 subRun: 1 event: 47000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:45:02 CET run: 74518431 subRun: 1 event: 47000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:45:02 CET run: 74518431 subRun: 1 event: 47000
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 638, 550, 331
minwire 0: 1172
minwire 1: 607
minwire 2: 1277
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.14959, 0.839941, 0.0104692, 
Output 1: 0.86653, 0.122487, 0.00912799, 0.00185553, 
Output 2: 0.700525, 0.299475, 

636
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2301!
23-Nov-2024 07:45:06 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74518431_469_20231204T214753Z_gen_g4_detsim_hitreco__20240510T030557Z_reco2_reco_data_2024-11-23T_052034Z.root"
23-Nov-2024 07:45:06 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/91/dd/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74518431_469_20231204T214753Z_gen_g4_detsim_hitreco__20240510T030557Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              3.05072       48.5807       1373.15       11.7443       151.77         100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0150107     0.0470383     0.389373      0.0436898     0.0389232       100    
reco:linecluster:LineCluster                           0.137537       1.14377       65.2311      0.174048       6.5961         100    
reco:trajcluster:TrajCluster                           0.0435674      2.90405       151.077      0.0630408      15.8168        100    
reco:pandora:StandardPandora                            1.26429        7.673        460.671       1.43683       46.1121        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000134174   0.00946307      0.16171     0.00217444     0.0229978       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000196803    0.0115126      0.12744     0.00354959     0.0208369       100    
reco:pandoracalo:Calorimetry                          0.000120566    0.060608       2.49255     0.00197699     0.276274        100    
reco:pandorapid:Chi2ParticleID                         4.57e-05     0.000170936   0.00131506    0.000130322   0.000153599      100    
reco:trkshowersplit:TrackShowerHits                   0.000134728    0.0506414      1.91104     0.00110844     0.223409        100    
reco:pmtrack:PMAlgTrackMaker                          0.00137314      4.06003       137.754       0.27382       15.2575        100    
reco:pmtrackcalo:Calorimetry                          8.9399e-05     0.019708      0.470117     0.00113388     0.0646749       100    
reco:pmtrackpid:Chi2ParticleID                        3.5372e-05    0.000372809   0.00340493     0.0001548    0.000618074      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000241395     4.9239        158.987      0.418722       18.8298        100    
reco:pmtrajfitcalo:Calorimetry                        7.9128e-05     0.061804       2.53739     0.00242042      0.28007        100    
reco:pmtrajfitpid:Chi2ParticleID                      3.5484e-05    0.000199296   0.000934415   0.000145528   0.000173241      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00124059      4.72193       107.314      0.326009       14.3925        100    
reco:pmtracktccalo:Calorimetry                        8.4804e-05     0.0100661     0.155874     0.00122753     0.0262665       100    
reco:pmtracktcpid:Chi2ParticleID                      3.5755e-05    0.00031726    0.00214566    0.000151861   0.000432014      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000216323     2.32978       45.2099      0.338225       6.20196        100    
reco:pmtrajfittccalo:Calorimetry                      8.1298e-05     0.0340168     0.852006     0.00154385     0.117563        100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.4898e-05    0.000472617   0.00457934    0.000163119   0.000839974      100    
reco:blurredcluster:BlurredClustering                  0.0136827      1.3519        9.77448      0.648043       1.7417         100    
reco:emtrkmichelid:EmTrackMichelId                     0.304351       16.1368       215.866       4.48945       33.6583        100    
reco:emshower:EMShower                                0.00025156     0.0376144     0.294976      0.0148645     0.0570199       100    
reco:cvnmap:CVNMapper                                 2.9475e-05     0.0137541     0.0371134     0.0126146     0.0103556       100    
reco:cvneva:CVNEvaluator                              2.0659e-05     0.623644      0.922965      0.815164       0.36152        100    
reco:energyrecnumu:EnergyReco                         0.000307292    0.0587173      2.93785      0.004758      0.297295        100    
reco:energyrecnue:EnergyReco                          0.000176777   0.00298495     0.0506648    0.000706906   0.00702746       100    
reco:energyrecnc:EnergyReco                           0.00231711     0.0052546     0.0506287    0.00300344    0.00700446       100    
reco:energyrecnumurange:EnergyReco                    0.000175525    0.0567338      2.94415     0.00265457     0.297835        100    
reco:energyrecnumumcs:EnergyReco                      0.00097188     0.0588293      2.96436     0.00383554     0.299638        100    
reco:opdec:Deconvolution                               0.155558      0.307764      0.519708      0.310325      0.0880651       100    
reco:ophitspe:OpHitFinderDeco                          0.607154       1.57293       3.0487        1.65698      0.573486        100    
reco:opflash:OpFlashFinder                            0.000395499   0.00142683    0.00352874    0.00143732    0.000638749      100    
reco:rns:RandomNumberSaver                            1.6247e-05    2.78798e-05   0.00024596    2.41275e-05   2.30233e-05      100    
reco:anglereconue:NuAngularReco                       0.000111403   0.00222421     0.0399227    0.00044092    0.00562029       100    
reco:anglereconumu:NuAngularReco                      8.3214e-05    0.00208569     0.0397507    0.000319546   0.00555556       100    
reco:anglereconuepfps:NuAngularReco                   0.000125563   0.00220474     0.0400133    0.000425094   0.00556845       100    
reco:anglereconumupfps:NuAngularReco                  0.000133964   0.00217066     0.0400175    0.000411806   0.00557683       100    
[art]:TriggerResults:TriggerResultInserter            1.1416e-05    1.38968e-05   4.2539e-05    1.3226e-05    3.44289e-06      100    
end_path:out1:RootOutput                               2.475e-06    4.1624e-06    1.2707e-05       3e-06      1.91339e-06      100    
end_path:out1:RootOutput(write)                        0.0388128      0.27919       1.34646      0.266171      0.210978        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 7391.97 MB
  Peak resident set size usage (VmHWM): 2481.72 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74518431_469_20231204T214753Z_gen_g4_detsim_hitreco__20240510T030557Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74518431_469_20231204T214753Z_gen_g4_detsim_hitreco__20240510T030557Z_reco2_reco_2024-11-23T_052034Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74518431_469_20231204T214753Z_gen_g4_detsim_hitreco__20240510T030557Z_reco2_reco_data_2024-11-23T_052034Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work