Jobsub ID 102903.0@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	102903.0@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4143
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-19 03:34:31
Site	CZ_FZU
Entry	Nova_CZ_FZU_osgce2
Last heartbeat	2024-11-19 05:30:56
From worker node	Hostname	mikan48.farm.particle.cz
	cpuinfo	Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	172800 (48 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-19 03:36:34
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496296_551_20231201T212748Z_gen_g4_detsim_hitreco__20240508T045246Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (6812s)
	CPU time	1h (6571s = 96%)
Outputting started	2024-11-19 05:30:08
Output files
Finished	2024-11-19 05:30:56
List job events Wrapper job log
Jobscript log (last 10,000 characters)

ule  19-Nov-2024 06:29:26 CET run: 74496296 subRun: 1 event: 55200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  19-Nov-2024 06:29:26 CET run: 74496296 subRun: 1 event: 55200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  19-Nov-2024 06:29:53 CET run: 74496296 subRun: 1 event: 55200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  19-Nov-2024 06:29:53 CET run: 74496296 subRun: 1 event: 55200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  19-Nov-2024 06:29:53 CET run: 74496296 subRun: 1 event: 55200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  19-Nov-2024 06:29:53 CET run: 74496296 subRun: 1 event: 55200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  19-Nov-2024 06:29:53 CET run: 74496296 subRun: 1 event: 55200
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 358, 272, 271
minwire 0: 1307
minwire 1: 1020
minwire 2: 1415
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.212498, 0.0641777, 0.723324, 
Output 1: 0.0201972, 0.049945, 0.216389, 0.713469, 
Output 2: 0.266913, 0.733087, 

612
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2275!
19-Nov-2024 06:30:04 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496296_551_20231201T212748Z_gen_g4_detsim_hitreco__20240508T045246Z_reco2_reco_data_2024-11-19T_033641Z.root"
19-Nov-2024 06:30:04 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/70/ea/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496296_551_20231201T212748Z_gen_g4_detsim_hitreco__20240508T045246Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.133702       65.5657       869.656       19.3394       128.708        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00356413     0.0206066     0.0596311     0.0196938    0.00766985       100    
reco:linecluster:LineCluster                          0.00421762     0.284861       15.0827      0.0555966      1.5045         100    
reco:trajcluster:TrajCluster                          0.00258466      1.38913       45.9814      0.0287195      5.19381        100    
reco:pandora:StandardPandora                           0.0228173      5.35326       88.9558       3.08912       10.1602        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000274666    0.0226335     0.470702     0.00397018     0.059755        100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000459416    0.0260614     0.213669     0.00614773     0.0429145       100    
reco:pandoracalo:Calorimetry                          0.000212903    0.0558308      1.53432     0.00263554     0.205967        100    
reco:pandorapid:Chi2ParticleID                        7.0895e-05    0.000514333   0.00721151    0.000343105   0.000867764      100    
reco:trkshowersplit:TrackShowerHits                   0.000269671    0.0368622     0.750726     0.00161592     0.105608        100    
reco:pmtrack:PMAlgTrackMaker                          0.00272398      5.0318        115.352      0.346211       14.6981        100    
reco:pmtrackcalo:Calorimetry                          0.000218657    0.0384097      1.04637     0.00215068     0.141732        100    
reco:pmtrackpid:Chi2ParticleID                         6.29e-05     0.000962125    0.0179971    0.000342957    0.0020684       100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000674595     5.62666       141.729      0.543498       16.7744        100    
reco:pmtrajfitcalo:Calorimetry                        0.000262664    0.0631638      1.94989     0.00306509     0.242586        100    
reco:pmtrajfitpid:Chi2ParticleID                      9.3812e-05    0.00046662    0.00202042    0.000354961   0.000357451      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00328748      5.80767       105.188      0.312916       15.5641        100    
reco:pmtracktccalo:Calorimetry                        0.000235629    0.0223929     0.519751     0.00187807     0.0663469       100    
reco:pmtracktcpid:Chi2ParticleID                      7.0923e-05    0.00121037     0.0245764    0.000328893   0.00300747       100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000686082     3.42226       58.9919      0.519604       7.98731        100    
reco:pmtrajfittccalo:Calorimetry                      0.000222652    0.0506947      1.33036     0.00255473     0.177579        100    
reco:pmtrajfittcpid:Chi2ParticleID                     8.185e-05    0.000919083   0.00777893    0.000395538   0.00132171       100    
reco:blurredcluster:BlurredClustering                 0.00028421      2.11042       14.7175      0.885012       2.75185        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000406395     30.8753       289.941       6.76983       56.3146        100    
reco:emshower:EMShower                                0.000583195    0.068818      0.488942      0.0245973     0.103098        100    
reco:cvnmap:CVNMapper                                 6.0971e-05     0.0283265      0.10994      0.0296396     0.0220942       100    
reco:cvneva:CVNEvaluator                              2.9145e-05      1.53854       3.70245       1.80934       1.03365        100    
reco:energyrecnumu:EnergyReco                         0.000656154    0.173802       8.63133     0.00918726     0.896156        100    
reco:energyrecnue:EnergyReco                          0.000278454   0.00651974     0.100512     0.00131873     0.0153057       100    
reco:energyrecnc:EnergyReco                           0.00277044    0.00956287     0.0988404    0.00470792     0.0150293       100    
reco:energyrecnumurange:EnergyReco                    0.000563968    0.148144       7.49762     0.00456209     0.781931        100    
reco:energyrecnumumcs:EnergyReco                      0.00281834     0.166033       8.79945     0.00637315     0.907899        100    
reco:opdec:Deconvolution                              0.00449873      0.30313       0.62601      0.288533      0.120917        100    
reco:ophitspe:OpHitFinderDeco                         0.00552262      2.36452       4.52291       2.28952      0.983758        100    
reco:opflash:OpFlashFinder                            0.000420981   0.00329292     0.0173984    0.00275079    0.00233234       100    
reco:rns:RandomNumberSaver                             2.945e-05    0.000124308   0.00272969    8.5588e-05    0.000292407      100    
reco:anglereconue:NuAngularReco                       0.000229617   0.00504055     0.0800373    0.000926011    0.0123469       100    
reco:anglereconumu:NuAngularReco                      0.000140174   0.00430046     0.0672968    0.000583354    0.010747        100    
reco:anglereconuepfps:NuAngularReco                    0.0001861    0.00484569     0.078289     0.000830395    0.0116563       100    
reco:anglereconumupfps:NuAngularReco                  0.000161895   0.00477311     0.0899312    0.000786453    0.012361        100    
[art]:TriggerResults:TriggerResultInserter            1.3952e-05    3.04694e-05   0.000202522   2.75745e-05   1.88998e-05      100    
end_path:out1:RootOutput                               3.786e-06    1.29096e-05   6.4637e-05    1.26115e-05   6.43681e-06      100    
end_path:out1:RootOutput(write)                        0.0303548     0.489216       1.99499      0.457319      0.356871        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 10763 MB
  Peak resident set size usage (VmHWM): 2411.87 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496296_551_20231201T212748Z_gen_g4_detsim_hitreco__20240508T045246Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496296_551_20231201T212748Z_gen_g4_detsim_hitreco__20240508T045246Z_reco2_reco_2024-11-19T_033641Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496296_551_20231201T212748Z_gen_g4_detsim_hitreco__20240508T045246Z_reco2_reco_data_2024-11-19T_033641Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work