Jobsub ID 108744.106@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108744.106@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:38:45
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce505
Last heartbeat	2024-11-23 06:11:20
From worker node	Hostname	b9p05p5799.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:40:25
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65840069_84_20231120T140406Z_gen_g4_detsim_hitreco__20240507T171024Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (5436s)
	CPU time	1h (5740s = 105%)
Outputting started	2024-11-23 06:11:02
Output files
Finished	2024-11-23 06:11:20
List job events Wrapper job log
Jobscript log (last 10,000 characters)

143166 GeV, p2=0.00933281
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtrack@BeginModule  23-Nov-2024 07:10:57 CET run: 65840069 subRun: 1 event: 17000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtrack@BeginModule  23-Nov-2024 07:10:57 CET run: 65840069 subRun: 1 event: 17000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtrack@BeginModule  23-Nov-2024 07:10:57 CET run: 65840069 subRun: 1 event: 17000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtrack@BeginModule  23-Nov-2024 07:10:57 CET run: 65840069 subRun: 1 event: 17000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 07:10:57 CET run: 65840069 subRun: 1 event: 17000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 07:10:57 CET run: 65840069 subRun: 1 event: 17000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 07:10:57 CET run: 65840069 subRun: 1 event: 17000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 07:10:57 CET run: 65840069 subRun: 1 event: 17000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:10:57 CET run: 65840069 subRun: 1 event: 17000
Comparing two wires in the same plane: return failure
%MSG
305
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 832!
23-Nov-2024 07:11:01 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65840069_84_20231120T140406Z_gen_g4_detsim_hitreco__20240507T171024Z_reco2_reco_data_2024-11-23T_044029Z.root"
23-Nov-2024 07:11:01 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/77/be/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65840069_84_20231120T140406Z_gen_g4_detsim_hitreco__20240507T171024Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.06624       25.5453       803.232       7.12517       73.2419        200    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0161262     0.0653066      1.82549      0.0415844     0.184593        200    
reco:linecluster:LineCluster                           0.0937724     0.794723       78.8491      0.163424       5.91518        200    
reco:trajcluster:TrajCluster                           0.0468919      1.35435       125.798      0.0554453      9.73382        200    
reco:pandora:StandardPandora                           0.0580057      2.22158       62.5184       1.05361       6.82919        200    
reco:pandoraTrack:LArPandoraTrackCreation             0.000100691   0.00507355     0.0839872     0.0010364     0.0111168       200    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000134641    0.0162679     0.448321     0.00206837     0.0512346       200    
reco:pandoracalo:Calorimetry                           8.117e-05     0.0198895      1.03989     0.000710166    0.0947909       200    
reco:pandorapid:Chi2ParticleID                        3.0701e-05    0.000133933   0.00107209    9.6016e-05    0.000127719      200    
reco:trkshowersplit:TrackShowerHits                   7.9071e-05     0.0266642      1.50866     0.000544765     0.13512        200    
reco:pmtrack:PMAlgTrackMaker                          0.000844867     2.40944       107.241      0.128741       9.85817        200    
reco:pmtrackcalo:Calorimetry                           5.387e-05     0.0101096     0.588807     0.000674325    0.0475324       200    
reco:pmtrackpid:Chi2ParticleID                         2.201e-05    0.000238722   0.00559828    0.000105925   0.000480317      200    
reco:pmtrajfit:PMAlgTrajFitter                        0.000132091     1.9418        60.1565      0.203435       6.42311        200    
reco:pmtrajfitcalo:Calorimetry                        4.8381e-05     0.0205678      1.23094     0.000994849    0.101732        200    
reco:pmtrajfitpid:Chi2ParticleID                       2.064e-05    0.000156222   0.000843246   0.000110046   0.000141596      200    
reco:pmtracktc:PMAlgTrackMaker                        0.000705497     2.95575       145.762      0.129572       12.7966        200    
reco:pmtracktccalo:Calorimetry                         5.128e-05    0.00617004     0.133913      0.0006376     0.0174928       200    
reco:pmtracktcpid:Chi2ParticleID                       2.156e-05    0.000221453   0.00331814    0.000114591   0.000347088      200    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000128371     1.43231        44.38       0.234136       4.26964        200    
reco:pmtrajfittccalo:Calorimetry                       4.716e-05     0.0137133     0.758865     0.000854327    0.0629798       200    
reco:pmtrajfittcpid:Chi2ParticleID                    2.0241e-05    0.000296633   0.00527361    0.000125706   0.000530055      200    
reco:blurredcluster:BlurredClustering                 9.5151e-05     0.863492       8.33353      0.409065       1.16547        200    
reco:emtrkmichelid:EmTrackMichelId                    0.000141152     8.73858       163.861       1.94403       18.8965        200    
reco:emshower:EMShower                                0.000171401    0.0211409     0.286321     0.00396436     0.0387928       200    
reco:cvnmap:CVNMapper                                  1.936e-05    0.00797553     0.0327441    0.00554286    0.00727115       200    
reco:cvneva:CVNEvaluator                               1.386e-05     0.546071       1.18379      0.675935      0.381555        200    
reco:energyrecnumu:EnergyReco                         0.000283482    0.0295488     0.769956     0.00308737     0.0855288       200    
reco:energyrecnue:EnergyReco                          0.000105591   0.00169541     0.0396097    0.000390648   0.00411175       200    
reco:energyrecnc:EnergyReco                           0.00135443    0.00360262     0.0448119    0.00248201    0.00451602       200    
reco:energyrecnumurange:EnergyReco                    0.000109861    0.0286335     0.826286     0.00179639      0.09424        200    
reco:energyrecnumumcs:EnergyReco                      0.00127503     0.0284212     0.781892      0.0031512     0.0843167       200    
reco:opdec:Deconvolution                               0.0643076     0.251749      0.437867      0.248646      0.0805478       200    
reco:ophitspe:OpHitFinderDeco                          0.282009       1.53063       3.63655       1.4454        0.76374        200    
reco:opflash:OpFlashFinder                            0.000201552   0.00115649     0.0041046    0.00107544    0.000595477      200    
reco:rns:RandomNumberSaver                             1.54e-05     2.84174e-05   0.000268193   2.77405e-05   1.88646e-05      200    
reco:anglereconue:NuAngularReco                       0.00012674    0.00115614     0.0208074    0.000315717    0.0025451       200    
reco:anglereconumu:NuAngularReco                       5.645e-05    0.00102906     0.0202695    0.000180797   0.00265418       200    
reco:anglereconuepfps:NuAngularReco                   9.3271e-05    0.00115126     0.0229542    0.000284387    0.0027665       200    
reco:anglereconumupfps:NuAngularReco                  8.0611e-05    0.00106569     0.019019     0.000238262   0.00256812       200    
[art]:TriggerResults:TriggerResultInserter             7.57e-06     1.08878e-05    3.559e-05    1.0235e-05    3.04856e-06      200    
end_path:out1:RootOutput                                2.4e-06     4.78453e-06    1.94e-05      4.88e-06     2.14219e-06      200    
end_path:out1:RootOutput(write)                        0.014357      0.191757      0.913898      0.174714      0.143081        200    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21819.9 MB
  Peak resident set size usage (VmHWM): 2425.31 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65840069_84_20231120T140406Z_gen_g4_detsim_hitreco__20240507T171024Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65840069_84_20231120T140406Z_gen_g4_detsim_hitreco__20240507T171024Z_reco2_reco_2024-11-23T_044029Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_65840069_84_20231120T140406Z_gen_g4_detsim_hitreco__20240507T171024Z_reco2_reco_data_2024-11-23T_044029Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work