Jobsub ID 108551.107@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108551.107@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-22 23:16:35
Site	ES_PIC
Entry	DUNE_T1_ES_PIC_ce15-multicore
Last heartbeat	2024-11-23 00:16:30
From worker node	Hostname	tds211.pic.es
	cpuinfo	Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	216000 (60 hours)
	Inner Apptainer?	True
Job state	stalled
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-22 23:33:38
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006610_871_20231202T205401Z_gen_g4_detsim_hitreco__20240508T070338Z_reco2.root
Outputting started	2024-11-23 00:16:30
Output files
Finished	2024-11-23 00:34:32
List job events Wrapper job log
Jobscript log (last 10,000 characters)

16:20 CET run: 66006610 subRun: 1 event: 87200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtrack@BeginModule  23-Nov-2024 01:16:20 CET run: 66006610 subRun: 1 event: 87200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtrack@BeginModule  23-Nov-2024 01:16:20 CET run: 66006610 subRun: 1 event: 87200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 01:16:20 CET run: 66006610 subRun: 1 event: 87200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 01:16:20 CET run: 66006610 subRun: 1 event: 87200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 01:16:20 CET run: 66006610 subRun: 1 event: 87200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 01:16:21 CET run: 66006610 subRun: 1 event: 87200
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 172, 42, 114
minwire 0: 352
minwire 1: 2707
minwire 2: 0
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.011927, 0.000871753, 0.987201, 
Output 1: 0.00448407, 0.980836, 0.0145217, 0.000158418, 
Output 2: 0.990772, 0.00922805, 

352
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1056!
23-Nov-2024 01:16:29 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006610_871_20231202T205401Z_gen_g4_detsim_hitreco__20240508T070338Z_reco2_reco_data_2024-11-22T_233341Z.root"
23-Nov-2024 01:16:29 CET  Closed input file "root://xrootd.pic.es:1094/pnfs/pic.es/data/dune/RSE/fardet-hd/a9/d8/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006610_871_20231202T205401Z_gen_g4_detsim_hitreco__20240508T070338Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.0585198      24.7039       625.166       7.4754        70.7972        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.000967724    0.0180635     0.0535758     0.0197642     0.0103438       100    
reco:linecluster:LineCluster                          0.00166696     0.134367       5.4847       0.0185299      0.64246        100    
reco:trajcluster:TrajCluster                          0.000876693    0.651461       33.744      0.00904312      3.59765        100    
reco:pandora:StandardPandora                          0.00995226      3.56754       188.134       1.35407       18.6509        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000127607   0.00545833      0.12881     0.00159359     0.0156804       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000186247   0.00719807     0.0702471    0.00268378     0.0115117       100    
reco:pandoracalo:Calorimetry                          0.000100846    0.0315061      1.79117     0.000857454    0.187357        100    
reco:pandorapid:Chi2ParticleID                        4.0935e-05    0.000158077   0.00154869    0.000126333   0.000161605      100    
reco:trkshowersplit:TrackShowerHits                   6.6629e-05     0.0144579     0.655336     0.000560739    0.0714095       100    
reco:pmtrack:PMAlgTrackMaker                          0.00116928      1.89952       49.035       0.145271       6.59348        100    
reco:pmtrackcalo:Calorimetry                          8.0278e-05    0.00917262      0.42513     0.000692296    0.0448413       100    
reco:pmtrackpid:Chi2ParticleID                        3.3606e-05    0.000248021    0.0032199    0.000127387   0.00042101       100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000191436     2.10247       78.0389      0.265643       8.54356        100    
reco:pmtrajfitcalo:Calorimetry                        7.8633e-05     0.0319068      1.74647     0.00120064     0.183879        100    
reco:pmtrajfitpid:Chi2ParticleID                      3.3822e-05    0.000166694   0.000550209   0.000147284   0.000104063      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00105686      2.19277       65.6567      0.137525       8.09106        100    
reco:pmtracktccalo:Calorimetry                        7.7809e-05    0.00491528     0.140075     0.000762716    0.0163918       100    
reco:pmtracktcpid:Chi2ParticleID                      3.3791e-05    0.000223674   0.00154517    0.000128462   0.00027514       100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000182271     1.24585       24.4112      0.235214       3.4749         100    
reco:pmtrajfittccalo:Calorimetry                      7.6063e-05     0.0147073     0.550698     0.000896592    0.0643941       100    
reco:pmtrajfittcpid:Chi2ParticleID                     3.343e-05    0.000318815   0.00323172    0.000139202   0.000537224      100    
reco:blurredcluster:BlurredClustering                 6.8026e-05     0.864957       7.05395       0.49396       1.2694         100    
reco:emtrkmichelid:EmTrackMichelId                    0.000143265     9.7201        165.548       2.60692       22.5693        100    
reco:emshower:EMShower                                0.000171004    0.0216095     0.259079      0.0027352     0.0394253       100    
reco:cvnmap:CVNMapper                                 2.5158e-05     0.0105245      0.03576      0.0108192    0.00808574       100    
reco:cvneva:CVNEvaluator                              2.0076e-05     0.574874       1.28513      0.737089      0.368043        100    
reco:energyrecnumu:EnergyReco                         0.000327178    0.0257264     0.915346     0.00446589     0.104266        100    
reco:energyrecnue:EnergyReco                          0.000155346   0.00173144     0.0278144    0.000558864   0.00408084       100    
reco:energyrecnc:EnergyReco                           0.00204377    0.00372173     0.0306048    0.00261201    0.00421863       100    
reco:energyrecnumurange:EnergyReco                    0.000200743    0.0231943     0.889587     0.00217496     0.102239        100    
reco:energyrecnumumcs:EnergyReco                      0.00132382     0.0248614     0.898731     0.00316383     0.103162        100    
reco:opdec:Deconvolution                              0.00422726     0.170652      0.550962      0.164311      0.0883845       100    
reco:ophitspe:OpHitFinderDeco                         0.00272422      1.10733       2.30697       1.12151      0.522721        100    
reco:opflash:OpFlashFinder                            0.000119464   0.00134725    0.00340999     0.0013515    0.000638503      100    
reco:rns:RandomNumberSaver                            1.6374e-05    3.03673e-05   0.000314377   2.5023e-05    2.98027e-05      100    
reco:anglereconue:NuAngularReco                       0.000100345    0.0011494     0.0214522    0.00030738    0.00306508       100    
reco:anglereconumu:NuAngularReco                      8.1812e-05     0.0010178     0.0204585    0.000206736   0.00299267       100    
reco:anglereconuepfps:NuAngularReco                   0.000116476   0.00114678     0.0211485    0.000306716   0.00304047       100    
reco:anglereconumupfps:NuAngularReco                  0.000114698   0.00108655     0.0207836    0.000283643   0.00297693       100    
[art]:TriggerResults:TriggerResultInserter            1.0314e-05    1.35658e-05   5.9649e-05    1.23375e-05   5.31801e-06      100    
end_path:out1:RootOutput                               2.691e-06    4.64545e-06   1.7698e-05     3.203e-06    2.58618e-06      100    
end_path:out1:RootOutput(write)                       0.00855275     0.214557      0.979811      0.226808      0.151659        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8307.31 MB
  Peak resident set size usage (VmHWM): 2217.55 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006610_871_20231202T205401Z_gen_g4_detsim_hitreco__20240508T070338Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006610_871_20231202T205401Z_gen_g4_detsim_hitreco__20240508T070338Z_reco2_reco_2024-11-22T_233341Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66006610_871_20231202T205401Z_gen_g4_detsim_hitreco__20240508T070338Z_reco2_reco_data_2024-11-22T_233341Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work