Jobsub ID 296233.1@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	296233.1@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4143
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-19 07:11:11
Site	CZ_FZU
Entry	Nova_CZ_FZU_osgce2
Last heartbeat	2024-11-19 08:54:05
From worker node	Hostname	mikan44.farm.particle.cz
	cpuinfo	Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	172800 (48 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-19 07:12:08
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496296_551_20231201T212748Z_gen_g4_detsim_hitreco__20240508T045246Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (6090s)
	CPU time	1h (5768s = 94%)
Outputting started	2024-11-19 08:53:38
Output files
Finished	2024-11-19 08:54:05
List job events Wrapper job log
Jobscript log (last 10,000 characters)

e  19-Nov-2024 09:53:14 CET run: 74496296 subRun: 1 event: 55200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  19-Nov-2024 09:53:14 CET run: 74496296 subRun: 1 event: 55200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  19-Nov-2024 09:53:28 CET run: 74496296 subRun: 1 event: 55200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  19-Nov-2024 09:53:28 CET run: 74496296 subRun: 1 event: 55200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  19-Nov-2024 09:53:28 CET run: 74496296 subRun: 1 event: 55200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  19-Nov-2024 09:53:28 CET run: 74496296 subRun: 1 event: 55200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  19-Nov-2024 09:53:28 CET run: 74496296 subRun: 1 event: 55200
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 358, 272, 271
minwire 0: 1307
minwire 1: 1020
minwire 2: 1415
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.212498, 0.0641777, 0.723324, 
Output 1: 0.0201972, 0.049945, 0.216389, 0.713469, 
Output 2: 0.266913, 0.733087, 

612
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2275!
19-Nov-2024 09:53:36 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496296_551_20231201T212748Z_gen_g4_detsim_hitreco__20240508T045246Z_reco2_reco_data_2024-11-19T_071219Z.root"
19-Nov-2024 09:53:36 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/70/ea/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496296_551_20231201T212748Z_gen_g4_detsim_hitreco__20240508T045246Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.116201       58.9597       896.318       18.1922       118.537        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00259406     0.0210618     0.0538916     0.0195245    0.00853716       100    
reco:linecluster:LineCluster                          0.00220667     0.312839       16.2854      0.0604068      1.6248         100    
reco:trajcluster:TrajCluster                          0.00151488      1.39039       45.2371      0.0258506      5.14798        100    
reco:pandora:StandardPandora                           0.0261717      4.67672       95.354        2.60674       10.0069        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000274454    0.0167371     0.307828      0.0026947     0.0395941       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000398811     0.0246        0.28209     0.00547157     0.0447269       100    
reco:pandoracalo:Calorimetry                          0.000230549    0.0521081      1.61985     0.00207276     0.193004        100    
reco:pandorapid:Chi2ParticleID                        8.2453e-05    0.00035157    0.00295535    0.000260709   0.000343138      100    
reco:trkshowersplit:TrackShowerHits                   0.000145192    0.0425499      1.26475     0.00142257     0.144854        100    
reco:pmtrack:PMAlgTrackMaker                          0.00254177      5.33854       127.569      0.426352       15.6041        100    
reco:pmtrackcalo:Calorimetry                          0.000168751    0.0243439      0.49092     0.00158926     0.0661356       100    
reco:pmtrackpid:Chi2ParticleID                        7.1167e-05    0.000857311    0.0226031    0.000295833   0.00235017       100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000435127     5.49222       147.445      0.624159       16.7065        100    
reco:pmtrajfitcalo:Calorimetry                        0.000159604    0.0550843      1.85582     0.00304107     0.211529        100    
reco:pmtrajfitpid:Chi2ParticleID                      7.4492e-05    0.000485339   0.00559049    0.000302792   0.000744151      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00239977      6.11447       119.517      0.363045       16.4114        100    
reco:pmtracktccalo:Calorimetry                        0.000174492    0.0192064      0.32413     0.00160127     0.0489268       100    
reco:pmtracktcpid:Chi2ParticleID                      7.3759e-05    0.000615248   0.00484718    0.000279303   0.000835906      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000445913     3.63893       62.4797      0.550498       8.28224        100    
reco:pmtrajfittccalo:Calorimetry                      0.000152089    0.0443687      1.02351     0.00221574     0.141688        100    
reco:pmtrajfittcpid:Chi2ParticleID                    6.8233e-05    0.000845065   0.00742612    0.000330705   0.00129168       100    
reco:blurredcluster:BlurredClustering                 0.000166374     2.03157       12.3884      0.762726       2.61353        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000273672     24.4529       255.408       5.58637       44.2209        100    
reco:emshower:EMShower                                0.000370252    0.0682282     0.535557      0.0213612     0.101993        100    
reco:cvnmap:CVNMapper                                 4.8232e-05     0.0285696     0.142326      0.026791      0.0271764       100    
reco:cvneva:CVNEvaluator                              3.2347e-05      1.33072        2.277        1.74289      0.814725        100    
reco:energyrecnumu:EnergyReco                         0.000529452    0.113482       3.48732     0.00846016      0.40758        100    
reco:energyrecnue:EnergyReco                          0.00031702    0.00607515     0.0970194    0.00141138     0.0129721       100    
reco:energyrecnc:EnergyReco                           0.00299317     0.0096212     0.0816874    0.00459855     0.0113064       100    
reco:energyrecnumurange:EnergyReco                    0.000363607    0.0986736      3.48623     0.00396057     0.403063        100    
reco:energyrecnumumcs:EnergyReco                      0.00246207     0.0984367      3.3281      0.00669169     0.385001        100    
reco:opdec:Deconvolution                              0.00355602     0.323665      0.629647      0.303601       0.12714        100    
reco:ophitspe:OpHitFinderDeco                         0.00461278      2.62091       4.70897       2.53064       1.03986        100    
reco:opflash:OpFlashFinder                            0.000205004   0.00317957     0.0206958    0.00275707    0.00266103       100    
reco:rns:RandomNumberSaver                            2.7242e-05    7.55162e-05   0.000559878   7.15405e-05   5.40764e-05      100    
reco:anglereconue:NuAngularReco                       0.000266048   0.00406658     0.0465631    0.000903047   0.00843134       100    
reco:anglereconumu:NuAngularReco                      0.000167794   0.00308317     0.0433329    0.00048566    0.00705952       100    
reco:anglereconuepfps:NuAngularReco                   0.000256622   0.00339354     0.0426623    0.000790811   0.00703224       100    
reco:anglereconumupfps:NuAngularReco                  0.000245123   0.00327554     0.0426806    0.000657881   0.00709549       100    
[art]:TriggerResults:TriggerResultInserter            1.6401e-05    3.04862e-05   0.000107585   2.8768e-05    1.05084e-05      100    
end_path:out1:RootOutput                               4.25e-06     0.00011071     0.0100964    9.7605e-06     0.0010036       100    
end_path:out1:RootOutput(write)                        0.0212368     0.488338       1.5204       0.460804      0.333803        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 10763.7 MB
  Peak resident set size usage (VmHWM): 2441.83 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496296_551_20231201T212748Z_gen_g4_detsim_hitreco__20240508T045246Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496296_551_20231201T212748Z_gen_g4_detsim_hitreco__20240508T045246Z_reco2_reco_2024-11-19T_071219Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74496296_551_20231201T212748Z_gen_g4_detsim_hitreco__20240508T045246Z_reco2_reco_data_2024-11-19T_071219Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work