Jobsub ID 108782.6@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108782.6@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:10:32
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce512
Last heartbeat	2024-11-23 05:44:44
From worker node	Hostname	b9p09p9768.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:11:13
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6370891_364_20231201T105842Z_gen_g4_detsim_hitreco__20240507T202949Z_reco2.root
Jobscript	Exit code	0
	Real time	33m (1991s)
	CPU time	33m (2019s = 101%)
Outputting started	2024-11-23 05:44:24
Output files
Finished	2024-11-23 05:44:44
List job events Wrapper job log
Jobscript log (last 10,000 characters)

:Track3D:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:43:57 CET run: 6370891 subRun: 1 event: 36500
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 06:43:57 CET run: 6370891 subRun: 1 event: 36500
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:43:57 CET run: 6370891 subRun: 1 event: 36500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:43:57 CET run: 6370891 subRun: 1 event: 36500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:43:58 CET run: 6370891 subRun: 1 event: 36500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:43:58 CET run: 6370891 subRun: 1 event: 36500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:43:58 CET run: 6370891 subRun: 1 event: 36500
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 1407, 962, 1289
minwire 0: 166
minwire 1: 1528
minwire 2: 200
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.00102687, 0.000344607, 0.998628, 
Output 1: 0.435829, 0.4036, 0.121633, 0.0389384, 
Output 2: 0.878272, 0.121728, 

601
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2182!
23-Nov-2024 06:44:23 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6370891_364_20231201T105842Z_gen_g4_detsim_hitreco__20240507T202949Z_reco2_reco_data_2024-11-23T_051118Z.root"
23-Nov-2024 06:44:23 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/be/cd/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6370891_364_20231201T105842Z_gen_g4_detsim_hitreco__20240507T202949Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.636592       17.4469       187.097       8.80507       25.8981        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0173419     0.0399813     0.126427      0.0407716     0.0142061       100    
reco:linecluster:LineCluster                           0.109275      0.232335       3.39884      0.181782      0.326488        100    
reco:trajcluster:TrajCluster                           0.047759      0.324131       8.95563      0.0578382      1.06591        100    
reco:pandora:StandardPandora                           0.0575594      1.57644       24.0545        1.229        2.34646        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000130571   0.00383263     0.0481853    0.00117643    0.00676466       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000187062    0.0103827      0.29135     0.00314106     0.0305235       100    
reco:pandoracalo:Calorimetry                          0.00010342    0.00980899     0.280861     0.000772491    0.0331268       100    
reco:pandorapid:Chi2ParticleID                         3.398e-05    0.000148466   0.000628035   0.000119961   0.00010719       100    
reco:trkshowersplit:TrackShowerHits                   7.3581e-05     0.0137904     0.594501     0.000652065    0.0618743       100    
reco:pmtrack:PMAlgTrackMaker                          0.000968498     1.22701       23.3522      0.166423       3.0432         100    
reco:pmtrackcalo:Calorimetry                          6.1581e-05    0.00412888     0.0612666    0.000733071    0.0095566       100    
reco:pmtrackpid:Chi2ParticleID                         2.396e-05    0.000210499   0.00157294    0.000130016   0.000267716      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000151391     1.19525       17.7604      0.289639       2.6259         100    
reco:pmtrajfitcalo:Calorimetry                         5.497e-05    0.00824436      0.17309     0.00119477     0.0231046       100    
reco:pmtrajfitpid:Chi2ParticleID                       2.236e-05    0.00016181    0.000554464   0.000136511   0.000117897      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000781656     1.44232       27.5972      0.183697       3.70696        100    
reco:pmtracktccalo:Calorimetry                         5.81e-05      0.0034679     0.0524602    0.000794256   0.00734653       100    
reco:pmtracktcpid:Chi2ParticleID                       2.22e-05     0.000214271   0.00154169    0.000139106   0.000241354      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000132341    0.991582       13.8564      0.246492       2.00078        100    
reco:pmtrajfittccalo:Calorimetry                       5.644e-05    0.00715281     0.212926     0.000891537    0.0240583       100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.4721e-05    0.000276899   0.00329621    0.000136016   0.000473806      100    
reco:blurredcluster:BlurredClustering                 0.000110041    0.800551       5.08636      0.425426       1.02219        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000132341     6.56113       65.3998       2.07497       10.6512        100    
reco:emshower:EMShower                                0.000198661    0.0192964     0.117534     0.00443333     0.0290336       100    
reco:cvnmap:CVNMapper                                  2.143e-05    0.00959694     0.0263818     0.0122246    0.00753059       100    
reco:cvneva:CVNEvaluator                               1.538e-05     0.606905       1.08292      0.787546      0.437566        100    
reco:energyrecnumu:EnergyReco                         0.000321223    0.0203922     0.431137     0.00373872     0.0555464       100    
reco:energyrecnue:EnergyReco                          0.000122231   0.00138133     0.0142496    0.000479178    0.0020162       100    
reco:energyrecnc:EnergyReco                           0.00149414    0.00360429     0.0194221    0.00311981    0.00243554       100    
reco:energyrecnumurange:EnergyReco                    0.000157151    0.0178956     0.326519     0.00282352     0.0488637       100    
reco:energyrecnumumcs:EnergyReco                      0.00140707     0.0194977     0.321959     0.00331577     0.0497309       100    
reco:opdec:Deconvolution                               0.0551538      0.27423       0.46056      0.283403      0.0890534       100    
reco:ophitspe:OpHitFinderDeco                          0.211095       1.8173        3.80956       1.77243      0.830514        100    
reco:opflash:OpFlashFinder                            0.000318503   0.00132353    0.00365219    0.00123942    0.000618498      100    
reco:rns:RandomNumberSaver                             1.713e-05    3.73612e-05   0.000326232   3.29155e-05   3.00093e-05      100    
reco:anglereconue:NuAngularReco                       0.000148011   0.000890552    0.0124587    0.000404473   0.00149755       100    
reco:anglereconumu:NuAngularReco                       7.659e-05    0.000690998    0.0111059    0.000207432   0.00140062       100    
reco:anglereconuepfps:NuAngularReco                   0.00011312    0.000843136    0.011293     0.000374103   0.00140989       100    
reco:anglereconumupfps:NuAngularReco                   8.74e-05     0.000754895    0.0111016    0.000276852   0.00140299       100    
[art]:TriggerResults:TriggerResultInserter             8.16e-06     1.31102e-05    4.882e-05     1.15e-05     5.45231e-06      100    
end_path:out1:RootOutput                               2.87e-06     6.31856e-06    2.327e-05     6.015e-06    2.11666e-06      100    
end_path:out1:RootOutput(write)                        0.0157769     0.197936      0.785707      0.194062       0.13595        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21486.4 MB
  Peak resident set size usage (VmHWM): 2061.79 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6370891_364_20231201T105842Z_gen_g4_detsim_hitreco__20240507T202949Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6370891_364_20231201T105842Z_gen_g4_detsim_hitreco__20240507T202949Z_reco2_reco_2024-11-23T_051118Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6370891_364_20231201T105842Z_gen_g4_detsim_hitreco__20240507T202949Z_reco2_reco_data_2024-11-23T_051118Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work