Jobsub ID 108740.115@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108740.115@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:25:45
Site	ES_PIC
Entry	DUNE_T1_ES_PIC_ce14-multicore
Last heartbeat	2024-11-23 05:34:55
From worker node	Hostname	tds239.pic.es
	cpuinfo	Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	216000 (60 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:33:48
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50620606_659_20231207T113018Z_gen_g4_detsim_hitreco__20240510T040411Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (3653s)
	CPU time	51m (3088s = 84%)
Outputting started	2024-11-23 05:34:42
Output files
Finished	2024-11-23 05:34:55
List job events Wrapper job log
Jobscript log (last 10,000 characters)

CET run: 50620606 subRun: 1 event: 66000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:34:34 CET run: 50620606 subRun: 1 event: 66000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:34:34 CET run: 50620606 subRun: 1 event: 66000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:34:34 CET run: 50620606 subRun: 1 event: 66000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:34:37 CET run: 50620606 subRun: 1 event: 66000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:34:37 CET run: 50620606 subRun: 1 event: 66000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:34:37 CET run: 50620606 subRun: 1 event: 66000
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 133, 106, 86
minwire 0: 1016
minwire 1: 1844
minwire 2: 512
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.00348554, 0.995701, 0.000813338, 
Output 1: 0.272317, 0.545648, 0.172789, 0.0092464, 
Output 2: 0.992723, 0.00727687, 

492
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1706!
23-Nov-2024 06:34:41 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50620606_659_20231207T113018Z_gen_g4_detsim_hitreco__20240510T040411Z_reco2_reco_data_2024-11-23T_043351Z.root"
23-Nov-2024 06:34:41 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/75/ee/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50620606_659_20231207T113018Z_gen_g4_detsim_hitreco__20240510T040411Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.51234       31.0885       486.746       9.30331       76.2469        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0455453     0.0872033     0.174173      0.100775      0.0259474       100    
reco:linecluster:LineCluster                           0.285303      0.847685       24.6972      0.434841       2.58187        100    
reco:trajcluster:TrajCluster                           0.133749       1.93184       56.857       0.142225       8.70263        100    
reco:pandora:StandardPandora                           0.150588       2.96622       64.8107       1.48601       7.36037        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000136314   0.00600429     0.0723034    0.00124081     0.0145767       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000185201    0.0116135     0.416472     0.00252873     0.0422301       100    
reco:pandoracalo:Calorimetry                          0.000114568    0.0300471     0.995134     0.000835129    0.129775        100    
reco:pandorapid:Chi2ParticleID                        4.4654e-05    0.000156593    0.0013893    0.000125024   0.000153357      100    
reco:trkshowersplit:TrackShowerHits                   8.8822e-05     0.0229481     0.627033     0.00053809     0.091213        100    
reco:pmtrack:PMAlgTrackMaker                          0.00126276      2.39822       55.8567      0.189294        8.596         100    
reco:pmtrackcalo:Calorimetry                          8.6124e-05     0.0110477     0.214493     0.000811897    0.0377391       100    
reco:pmtrackpid:Chi2ParticleID                        3.5742e-05    0.000269834   0.00248075    0.000130424   0.00045983       100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000212645     2.70448       77.0244      0.208753       9.83997        100    
reco:pmtrajfitcalo:Calorimetry                        8.2856e-05     0.0305884     0.997581     0.00111492     0.129943        100    
reco:pmtrajfitpid:Chi2ParticleID                      3.6043e-05    0.000170954   0.000733468   0.000137023   0.000125975      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00110789      3.15334       73.1267       0.17714       11.8933        100    
reco:pmtracktccalo:Calorimetry                        8.6251e-05    0.00706567     0.119353     0.000818131    0.0217002       100    
reco:pmtracktcpid:Chi2ParticleID                      3.6419e-05    0.000244059   0.00203703     0.0001357    0.00035502       100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000185922     1.22069       18.3279      0.228229       3.1464         100    
reco:pmtrajfittccalo:Calorimetry                      8.1452e-05     0.0175518     0.411234     0.00101797     0.0630453       100    
reco:pmtrajfittcpid:Chi2ParticleID                     3.576e-05    0.000342841   0.00344604    0.000150919   0.000610109      100    
reco:blurredcluster:BlurredClustering                 8.3848e-05     0.914245       8.84518      0.443523       1.31662        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000118535     11.773        141.984       2.80616       26.2419        100    
reco:emshower:EMShower                                0.000172853    0.0224116     0.201496     0.00438794     0.0375901       100    
reco:cvnmap:CVNMapper                                 2.3442e-05     0.010321      0.0421157     0.0113149    0.00793241       100    
reco:cvneva:CVNEvaluator                              1.9115e-05      0.5473       0.979283      0.745013      0.342387        100    
reco:energyrecnumu:EnergyReco                         0.000341251    0.0336068     0.851376     0.00394257     0.105524        100    
reco:energyrecnue:EnergyReco                          0.000169652   0.00200243     0.0239565    0.000454763   0.00450996       100    
reco:energyrecnc:EnergyReco                           0.00228945    0.00403695     0.0260936    0.00256689    0.00465807       100    
reco:energyrecnumurange:EnergyReco                    0.000209177    0.0320371     0.849344     0.00245334     0.105199        100    
reco:energyrecnumumcs:EnergyReco                      0.00086613     0.0335849     0.848325     0.00371355     0.105092        100    
reco:opdec:Deconvolution                               0.148111      0.477881      0.730057      0.484365       0.1113         100    
reco:ophitspe:OpHitFinderDeco                          0.463794       1.55555       2.64303       1.56267      0.500865        100    
reco:opflash:OpFlashFinder                            0.000237374   0.00127725    0.00408378    0.00121505    0.000609021      100    
reco:rns:RandomNumberSaver                            1.9071e-05    2.98929e-05   0.000287151   2.5823e-05    2.67972e-05      100    
reco:anglereconue:NuAngularReco                       0.000146186   0.00140883     0.0204613    0.000291325   0.00363798       100    
reco:anglereconumu:NuAngularReco                      8.8676e-05    0.00128385     0.0199517    0.000194297    0.0036069       100    
reco:anglereconuepfps:NuAngularReco                   0.000132978    0.0014079     0.0203184    0.000297155   0.00363116       100    
reco:anglereconumupfps:NuAngularReco                  0.000120416   0.00135476     0.0199826    0.000283026    0.0036004       100    
[art]:TriggerResults:TriggerResultInserter            1.1031e-05    1.30898e-05   5.0196e-05    1.2247e-05    4.16641e-06      100    
end_path:out1:RootOutput                               2.59e-06     4.09705e-06   1.6268e-05     3.181e-06    1.87102e-06      100    
end_path:out1:RootOutput(write)                        0.0145965     0.226276      0.857713      0.227389      0.159033        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 11645.6 MB
  Peak resident set size usage (VmHWM): 2331.96 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50620606_659_20231207T113018Z_gen_g4_detsim_hitreco__20240510T040411Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50620606_659_20231207T113018Z_gen_g4_detsim_hitreco__20240510T040411Z_reco2_reco_2024-11-23T_043351Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50620606_659_20231207T113018Z_gen_g4_detsim_hitreco__20240510T040411Z_reco2_reco_data_2024-11-23T_043351Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work