Jobsub ID 108782.21@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108782.21@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:10:32
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce512
Last heartbeat	2024-11-23 06:02:21
From worker node	Hostname	b9p08p5272.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:11:43
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50609765_717_20231207T075236Z_gen_g4_detsim_hitreco__20240510T023734Z_reco2.root
Jobscript	Exit code	0
	Real time	50m (3022s)
	CPU time	53m (3198s = 105%)
Outputting started	2024-11-23 06:02:06
Output files
Finished	2024-11-23 06:02:21
List job events Wrapper job log
Jobscript log (last 10,000 characters)

o wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:01:53 CET run: 50609765 subRun: 1 event: 71800
1st wire C:0 T:11 P:2 W:656 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:01:54 CET run: 50609765 subRun: 1 event: 71800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:01:54 CET run: 50609765 subRun: 1 event: 71800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:01:54 CET run: 50609765 subRun: 1 event: 71800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:01:54 CET run: 50609765 subRun: 1 event: 71800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:01:54 CET run: 50609765 subRun: 1 event: 71800
1st wire C:0 T:11 P:2 W:678 does not exist (max wire number: 480)
%MSG
Boundary wire vector sizes: 205, 499, 349
minwire 0: 1082
minwire 1: 1531
minwire 2: 794
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0398421, 0.000559485, 0.959598, 
Output 1: 0.0328766, 0.156873, 0.517609, 0.292641, 
Output 2: 0.920961, 0.0790389, 

630
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2356!
23-Nov-2024 07:02:05 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50609765_717_20231207T075236Z_gen_g4_detsim_hitreco__20240510T023734Z_reco2_reco_data_2024-11-23T_051148Z.root"
23-Nov-2024 07:02:05 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/fd/5e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50609765_717_20231207T075236Z_gen_g4_detsim_hitreco__20240510T023734Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.546571       28.3308       779.766       7.40526       96.2336        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0167278     0.0409888     0.0884818     0.0421507     0.0137646       100    
reco:linecluster:LineCluster                           0.0947796     0.902771       41.0234      0.165599       5.00686        100    
reco:trajcluster:TrajCluster                           0.0476213      1.84379       89.692       0.0543611      11.3742        100    
reco:pandora:StandardPandora                           0.057845       2.72841       97.5237       1.19445       10.116         100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000122141   0.00635218     0.190629     0.00106187     0.0214206       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000167982    0.0106148      0.19666     0.00218355     0.0253687       100    
reco:pandoracalo:Calorimetry                          0.000103791    0.0206343     0.665753     0.000705636    0.0904079       100    
reco:pandorapid:Chi2ParticleID                         3.666e-05    0.000149825   0.00108009    0.000104941   0.000146288      100    
reco:trkshowersplit:TrackShowerHits                   7.7881e-05     0.0348082      1.63477     0.000473794     0.18304        100    
reco:pmtrack:PMAlgTrackMaker                          0.00102288      3.38817       133.314      0.108765       16.9144        100    
reco:pmtrackcalo:Calorimetry                           6.378e-05     0.0164575     0.907238     0.000614605    0.0942429       100    
reco:pmtrackpid:Chi2ParticleID                         2.462e-05    0.000287872   0.00755258    0.00010647    0.000814044      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000173712     1.68612       56.8031      0.208997       6.29889        100    
reco:pmtrajfitcalo:Calorimetry                         5.831e-05     0.0201021     0.706931     0.00105767     0.0931286       100    
reco:pmtrajfitpid:Chi2ParticleID                      2.4031e-05    0.000161579   0.00118551    0.000117571   0.00016272       100    
reco:pmtracktc:PMAlgTrackMaker                        0.000779377     3.71293       137.952       0.15771       17.986         100    
reco:pmtracktccalo:Calorimetry                         5.777e-05    0.00716861     0.286727     0.000726716    0.0308908       100    
reco:pmtracktcpid:Chi2ParticleID                      2.1771e-05    0.000231119   0.00476309    0.000120246   0.000510447      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000146882     1.51752       50.4085      0.190712       5.72739        100    
reco:pmtrajfittccalo:Calorimetry                       5.226e-05     0.016345      0.690098     0.000801967    0.0782792       100    
reco:pmtrajfittcpid:Chi2ParticleID                     2.24e-05     0.000290514   0.00394813    0.000132401   0.000533706      100    
reco:blurredcluster:BlurredClustering                 0.000121671    0.693315        8.757       0.230275       1.33162        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000118261     8.7684        178.545       1.68426       23.2935        100    
reco:emshower:EMShower                                0.000180762    0.027686      0.852564     0.00346238     0.0905592       100    
reco:cvnmap:CVNMapper                                  2.038e-05    0.00887751     0.0511307    0.00831417    0.00869622       100    
reco:cvneva:CVNEvaluator                               1.588e-05     0.580433       1.22234      0.737522       0.42467        100    
reco:energyrecnumu:EnergyReco                         0.000286262    0.0415594      1.98824     0.00342328     0.211844        100    
reco:energyrecnue:EnergyReco                          0.000113541   0.00263896     0.0947434    0.000450849   0.00997123       100    
reco:energyrecnc:EnergyReco                           0.00144795     0.0047532     0.101581     0.00288936     0.0106173       100    
reco:energyrecnumurange:EnergyReco                    0.000127361    0.0366557      1.62071     0.00218875     0.178774        100    
reco:energyrecnumumcs:EnergyReco                      0.00144272     0.0376936      1.64843     0.00311629      0.18049        100    
reco:opdec:Deconvolution                               0.0506516     0.250981      0.436261      0.250074      0.0824718       100    
reco:ophitspe:OpHitFinderDeco                          0.166776       1.70733       3.98858       1.60272      0.823831        100    
reco:opflash:OpFlashFinder                            0.000150671   0.00129684    0.00504045     0.001147     0.00074073       100    
reco:rns:RandomNumberSaver                             1.68e-05     3.69373e-05   0.000283292    3.333e-05    2.72029e-05      100    
reco:anglereconue:NuAngularReco                       0.000147411   0.00184318     0.0587758    0.000376863   0.00648798       100    
reco:anglereconumu:NuAngularReco                      7.8811e-05    0.00162034     0.0580538    0.000199687   0.00640836       100    
reco:anglereconuepfps:NuAngularReco                   0.000112761   0.00179011     0.0593999    0.000346917   0.00652483       100    
reco:anglereconumupfps:NuAngularReco                  9.7201e-05    0.00165077     0.0556606    0.000262187   0.00618815       100    
[art]:TriggerResults:TriggerResultInserter             7.87e-06     1.2703e-05     3.702e-05     1.181e-05    4.23672e-06      100    
end_path:out1:RootOutput                                2.7e-06     5.82275e-06   2.2691e-05     5.585e-06    2.83527e-06      100    
end_path:out1:RootOutput(write)                        0.0131939     0.206213       1.09613      0.187152      0.173036        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 22188.6 MB
  Peak resident set size usage (VmHWM): 2760.25 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50609765_717_20231207T075236Z_gen_g4_detsim_hitreco__20240510T023734Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50609765_717_20231207T075236Z_gen_g4_detsim_hitreco__20240510T023734Z_reco2_reco_2024-11-23T_051148Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50609765_717_20231207T075236Z_gen_g4_detsim_hitreco__20240510T023734Z_reco2_reco_data_2024-11-23T_051148Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work