Jobsub ID 108746.15@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108746.15@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:39:54
Site	ES_PIC
Entry	DUNE_T1_ES_PIC_ce14-multicore
Last heartbeat	2024-11-23 05:57:35
From worker node	Hostname	tds405.pic.es
	cpuinfo	AMD EPYC 7502 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	216000 (60 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:44:25
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50542214_562_20231201T222628Z_gen_g4_detsim_hitreco__20240507T203920Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (4369s)
	CPU time	1h (3652s = 83%)
Outputting started	2024-11-23 05:57:14
Output files
Finished	2024-11-23 05:57:35
List job events Wrapper job log
Jobscript log (last 10,000 characters)

06:56:22 CET run: 50542214 subRun: 1 event: 56300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:56:22 CET run: 50542214 subRun: 1 event: 56300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:56:22 CET run: 50542214 subRun: 1 event: 56300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:57:04 CET run: 50542214 subRun: 1 event: 56300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:57:04 CET run: 50542214 subRun: 1 event: 56300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:57:04 CET run: 50542214 subRun: 1 event: 56300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:57:04 CET run: 50542214 subRun: 1 event: 56300
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 1665, 1592, 951
minwire 0: 733
minwire 1: 1978
minwire 2: 253
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.00168269, 0.996767, 0.00155052, 
Output 1: 0.717074, 0.246263, 0.0329271, 0.00373588, 
Output 2: 0.430861, 0.569139, 

779
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2948!
23-Nov-2024 06:57:12 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50542214_562_20231201T222628Z_gen_g4_detsim_hitreco__20240507T203920Z_reco2_reco_data_2024-11-23T_044430Z.root"
23-Nov-2024 06:57:12 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/2f/5a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50542214_562_20231201T222628Z_gen_g4_detsim_hitreco__20240507T203920Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              3.87161       37.9465       343.031       13.438        61.5678        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0460501     0.0883134     0.248342      0.0913142     0.0325445       100    
reco:linecluster:LineCluster                           0.415882      0.591895       3.35895      0.460027      0.431605        100    
reco:trajcluster:TrajCluster                           0.135264      0.916817       20.4454      0.151253       2.71418        100    
reco:pandora:StandardPandora                            1.62478       3.23506       69.865        1.82032       7.29256        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000177574   0.00846679     0.0960849    0.00151363     0.0166285       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000233495    0.024758       0.63851     0.00330063     0.0709346       100    
reco:pandoracalo:Calorimetry                          0.000127322    0.0180685     0.311207     0.00101379     0.0460664       100    
reco:pandorapid:Chi2ParticleID                        4.6751e-05    0.000248176   0.00250911    0.000162703   0.000285048      100    
reco:trkshowersplit:TrackShowerHits                   0.000152043    0.0290703     0.562345     0.000914099    0.0914199       100    
reco:pmtrack:PMAlgTrackMaker                          0.00184322      2.84063       33.0433      0.280936       6.33189        100    
reco:pmtrackcalo:Calorimetry                          0.000102802    0.0111081     0.185078     0.000973251    0.0289759       100    
reco:pmtrackpid:Chi2ParticleID                        4.0591e-05    0.000388379   0.00257379    0.000179324   0.00051519       100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000262726     3.39867       50.9895      0.430425       8.12587        100    
reco:pmtrajfitcalo:Calorimetry                        8.8461e-05     0.0189654     0.296247     0.00155373     0.0462451       100    
reco:pmtrajfitpid:Chi2ParticleID                       3.767e-05     0.0002713    0.00131401    0.000188189   0.00023742       100    
reco:pmtracktc:PMAlgTrackMaker                        0.00161268      3.58099       54.8555      0.330269       8.62234        100    
reco:pmtracktccalo:Calorimetry                        9.9712e-05    0.00951951     0.147619     0.00106934     0.0232137       100    
reco:pmtracktcpid:Chi2ParticleID                      4.0231e-05    0.000358914   0.00202691    0.000185134   0.000425534      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000260356     2.0159        20.7338       0.42448       3.70526        100    
reco:pmtrajfittccalo:Calorimetry                      8.3892e-05     0.0159766     0.238728     0.00131356     0.0402688       100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.4611e-05    0.000490417    0.0035113    0.000191929   0.000697792      100    
reco:blurredcluster:BlurredClustering                  0.0141463      1.4569        8.47855      0.565212       1.92228        100    
reco:emtrkmichelid:EmTrackMichelId                     0.314209       14.9156       124.21        3.46035       25.2857        100    
reco:emshower:EMShower                                0.000313086    0.046173      0.293773      0.0184631     0.071412        100    
reco:cvnmap:CVNMapper                                  2.598e-05     0.0132849     0.0502093    0.00892889     0.0123655       100    
reco:cvneva:CVNEvaluator                               2.239e-05     0.973346       1.54547       1.24246      0.554392        100    
reco:energyrecnumu:EnergyReco                         0.000337547    0.0416317     0.513949     0.00631287     0.0955998       100    
reco:energyrecnue:EnergyReco                          0.000194424   0.00271732     0.0242476    0.000760935   0.00469217       100    
reco:energyrecnc:EnergyReco                            0.002424     0.00654921      0.02954     0.00395255    0.00553835       100    
reco:energyrecnumurange:EnergyReco                    0.000334307    0.0364452     0.504678     0.00359699     0.0936999       100    
reco:energyrecnumumcs:EnergyReco                       0.0020498     0.0381087     0.502449      0.0052837     0.0919858       100    
reco:opdec:Deconvolution                               0.212102      0.558987      0.795509      0.565203      0.126431        100    
reco:ophitspe:OpHitFinderDeco                          0.625288       2.68641       4.32658       2.64211      0.924918        100    
reco:opflash:OpFlashFinder                            0.000352308   0.00180617    0.00646499    0.00166349    0.00086716       100    
reco:rns:RandomNumberSaver                            2.2861e-05    4.15687e-05   0.000408359   3.4571e-05    3.91336e-05      100    
reco:anglereconue:NuAngularReco                       0.000207394   0.00214864     0.0205714    0.00044273    0.00405607       100    
reco:anglereconumu:NuAngularReco                      9.2082e-05    0.00167811     0.0202284    0.000284886   0.00357223       100    
reco:anglereconuepfps:NuAngularReco                   0.000157493   0.00190361     0.0204788    0.000407754   0.00366814       100    
reco:anglereconumupfps:NuAngularReco                  0.000138393   0.00181032     0.0209334    0.000357088   0.00364879       100    
[art]:TriggerResults:TriggerResultInserter             1.124e-05    1.61107e-05   0.000111992   1.4045e-05    1.03745e-05      100    
end_path:out1:RootOutput                               3.08e-06     6.24803e-06    3.57e-05     5.1705e-06    3.82278e-06      100    
end_path:out1:RootOutput(write)                        0.0310225     0.352272       1.07553      0.326327       0.22531        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21323.8 MB
  Peak resident set size usage (VmHWM): 2180.94 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50542214_562_20231201T222628Z_gen_g4_detsim_hitreco__20240507T203920Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50542214_562_20231201T222628Z_gen_g4_detsim_hitreco__20240507T203920Z_reco2_reco_2024-11-23T_044430Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50542214_562_20231201T222628Z_gen_g4_detsim_hitreco__20240507T203920Z_reco2_reco_data_2024-11-23T_044430Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work