Jobsub ID 108744.147@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108744.147@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:38:45
Site	ES_PIC
Entry	DUNE_T1_ES_PIC_ce14-multicore
Last heartbeat	2024-11-23 06:47:27
From worker node	Hostname	td828.pic.es
	cpuinfo	AMD EPYC 7452 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	216000 (60 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:43:10
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574782_1107_20231203T120148Z_gen_g4_detsim_hitreco__20240509T192127Z_reco2.root
Jobscript	Exit code	0
	Real time	2h (7430s)
	CPU time	1h (6740s = 90%)
Outputting started	2024-11-23 06:47:01
Output files
Finished	2024-11-23 06:47:27
List job events Wrapper job log
Jobscript log (last 10,000 characters)

0
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 07:46:49 CET run: 50574782 subRun: 1 event: 110800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 07:46:49 CET run: 50574782 subRun: 1 event: 110800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:46:49 CET run: 50574782 subRun: 1 event: 110800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:46:49 CET run: 50574782 subRun: 1 event: 110800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:46:49 CET run: 50574782 subRun: 1 event: 110800
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:46:49 CET run: 50574782 subRun: 1 event: 110800
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 75, 91, 84
minwire 0: 1523
minwire 1: 760
minwire 2: 1759
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.998826, 0.000107427, 0.00106642, 
Output 1: 0.0817555, 0.903296, 0.0144157, 0.000532681, 
Output 2: 0.996867, 0.00313327, 

428
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1353!
23-Nov-2024 07:46:59 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574782_1107_20231203T120148Z_gen_g4_detsim_hitreco__20240509T192127Z_reco2_reco_data_2024-11-23T_044317Z.root"
23-Nov-2024 07:46:59 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/6b/aa/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574782_1107_20231203T120148Z_gen_g4_detsim_hitreco__20240509T192127Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.84846       68.999        2176.07       15.0705       259.838        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0474156     0.091881      0.272021      0.0981777     0.0316805       100    
reco:linecluster:LineCluster                           0.269737       1.07746       23.1809      0.453856       3.01913        100    
reco:trajcluster:TrajCluster                           0.135389       3.09347       95.2574      0.149013       14.2944        100    
reco:pandora:StandardPandora                           0.170461       13.3327       715.816       2.24584       76.888         100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000212703    0.0215774      1.16601     0.00197389     0.120847        100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000309555    0.0168881     0.448247     0.00377882     0.0504692       100    
reco:pandoracalo:Calorimetry                          0.000150722    0.0565582      2.07645     0.00125152     0.278315        100    
reco:pandorapid:Chi2ParticleID                        5.9871e-05    0.000325844   0.00408154    0.000200653   0.000501291      100    
reco:trkshowersplit:TrackShowerHits                   0.000107462    0.0596996      2.31207     0.00102006     0.298729        100    
reco:pmtrack:PMAlgTrackMaker                          0.00177331      7.20404       272.328      0.254965       33.3526        100    
reco:pmtrackcalo:Calorimetry                          0.000121212    0.055003       3.44807     0.00128398     0.359854        100    
reco:pmtrackpid:Chi2ParticleID                        4.7541e-05    0.000552712    0.0102357    0.000208243   0.00130086       100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000278944     4.47091       140.345      0.361002       17.2349        100    
reco:pmtrajfitcalo:Calorimetry                        0.000102962    0.0551783      1.88045      0.0017809     0.267506        100    
reco:pmtrajfitpid:Chi2ParticleID                      4.4351e-05    0.000325779   0.00348428    0.000240034   0.000402854      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00170932      7.87838       247.04       0.277422       34.0857        100    
reco:pmtracktccalo:Calorimetry                        9.7711e-05     0.0184672     0.696771     0.00117819     0.0896773       100    
reco:pmtracktcpid:Chi2ParticleID                       4.279e-05    0.000453472   0.00568176    0.000226024   0.000815473      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000355796     4.07444       174.349      0.332725       19.284         100    
reco:pmtrajfittccalo:Calorimetry                      9.6122e-05     0.0505466      2.40331     0.00144257     0.266277        100    
reco:pmtrajfittcpid:Chi2ParticleID                     3.94e-05     0.00062291     0.0103457    0.000241963   0.00135567       100    
reco:blurredcluster:BlurredClustering                 0.000130082     1.54168       19.2448      0.558037       2.62472        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000148872     20.7416       469.945       3.98338       63.2068        100    
reco:emshower:EMShower                                0.000255874    0.053348       1.25474      0.0103436      0.14061        100    
reco:cvnmap:CVNMapper                                 2.5651e-05     0.0147417     0.0759084     0.0114294     0.0140127       100    
reco:cvneva:CVNEvaluator                               2.07e-05       1.12583       3.18928       1.26246      0.791894        100    
reco:energyrecnumu:EnergyReco                         0.000436457    0.0718021      3.91745     0.00606155     0.400313        100    
reco:energyrecnue:EnergyReco                          0.000158083    0.0052731      0.24975     0.000675966    0.0261385       100    
reco:energyrecnc:EnergyReco                           0.00237388    0.00875404     0.268444     0.00419459     0.0277647       100    
reco:energyrecnumurange:EnergyReco                    0.000207493    0.0607313      3.1545      0.00393635     0.327402        100    
reco:energyrecnumumcs:EnergyReco                      0.00236986     0.125387       9.37787     0.00513541     0.935548        100    
reco:opdec:Deconvolution                               0.195019      0.581643      0.971778      0.580592      0.151614        100    
reco:ophitspe:OpHitFinderDeco                          0.560508       2.69437       5.33835       2.59692      0.998008        100    
reco:opflash:OpFlashFinder                            0.000397516   0.00215905    0.00913425    0.00193775    0.00131127       100    
reco:rns:RandomNumberSaver                             2.082e-05    5.07945e-05   0.000433567   4.20505e-05   4.61393e-05      100    
reco:anglereconue:NuAngularReco                       0.000194483   0.00559042      0.24688     0.000497478    0.0299414       100    
reco:anglereconumu:NuAngularReco                      9.5611e-05    0.00524328     0.252326     0.000275854    0.0296798       100    
reco:anglereconuepfps:NuAngularReco                   0.000142212   0.00548197     0.249218     0.000472117    0.0299393       100    
reco:anglereconumupfps:NuAngularReco                  0.000120811   0.00525399     0.235232     0.000396831    0.0285757       100    
[art]:TriggerResults:TriggerResultInserter            1.1391e-05    1.9601e-05    8.8392e-05    1.7036e-05    1.04965e-05      100    
end_path:out1:RootOutput                               3.54e-06     8.20026e-06   4.6321e-05     6.835e-06    6.24408e-06      100    
end_path:out1:RootOutput(write)                        0.0287175     0.387283       2.83349      0.347996      0.392968        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21808.3 MB
  Peak resident set size usage (VmHWM): 2652.72 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574782_1107_20231203T120148Z_gen_g4_detsim_hitreco__20240509T192127Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574782_1107_20231203T120148Z_gen_g4_detsim_hitreco__20240509T192127Z_reco2_reco_2024-11-23T_044317Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50574782_1107_20231203T120148Z_gen_g4_detsim_hitreco__20240509T192127Z_reco2_reco_data_2024-11-23T_044317Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work