Jobsub ID 301747.3@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301747.3@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:06:17
Site	CZ_FZU
Entry	Nova_CZ_FZU_osgce2
Last heartbeat	2024-11-23 07:35:15
From worker node	Hostname	mff2104.farm.particle.cz
	cpuinfo	AMD EPYC 7352 24-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	172800 (48 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:08:19
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1497_386_20231027T144707Z_gen_g4_detsim_hitreco__20240503T040206Z_reco2.root
Jobscript	Exit code	0
	Real time	2h (8801s)
	CPU time	2h (8596s = 97%)
Outputting started	2024-11-23 07:35:01
Output files
Finished	2024-11-23 07:35:15
List job events Wrapper job log
Jobscript log (last 10,000 characters)

the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 08:34:54 CET run: 1497 subRun: 1 event: 77400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:34:54 CET run: 1497 subRun: 1 event: 77400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:34:54 CET run: 1497 subRun: 1 event: 77400
1st wire C:0 T:8 P:2 W:804 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:34:54 CET run: 1497 subRun: 1 event: 77400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:34:54 CET run: 1497 subRun: 1 event: 77400
0 enabled hits in AverageDist2 calculation.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:34:54 CET run: 1497 subRun: 1 event: 77400
Track empty.
%MSG
%MSG-e pma::Track3D:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 08:34:54 CET run: 1497 subRun: 1 event: 77400
TuneFullTree failed.
%MSG
Boundary wire vector sizes: 109, 115, 87
minwire 0: 1223
minwire 1: 1707
minwire 2: 941
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.103005, 0.000494928, 0.8965, 
Output 1: 0.154732, 0.620979, 0.195357, 0.0289315, 
Output 2: 0.937112, 0.0628883, 

541
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1866!
23-Nov-2024 08:35:00 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1497_386_20231027T144707Z_gen_g4_detsim_hitreco__20240503T040206Z_reco2_reco_data_2024-11-23T_050849Z.root"
23-Nov-2024 08:35:00 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/18/31/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1497_386_20231027T144707Z_gen_g4_detsim_hitreco__20240503T040206Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.104146       42.2494       851.992       13.2148       103.478        200    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00151293     0.0226718     0.094682      0.0202268     0.0109766       200    
reco:linecluster:LineCluster                          0.00368353     0.455697       39.2709      0.0701941      3.13514        200    
reco:trajcluster:TrajCluster                          0.00181849      1.71577       70.4898      0.027591       8.11316        200    
reco:pandora:StandardPandora                           0.0189861      4.40596       222.052       1.72735       21.1416        200    
reco:pandoraTrack:LArPandoraTrackCreation             0.000139777   0.00888956     0.177389     0.00208326     0.0223396       200    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000214289    0.017633      0.353348     0.00411679     0.0429606       200    
reco:pandoracalo:Calorimetry                          0.00010978     0.0250999     0.784497     0.00147565     0.094503        200    
reco:pandorapid:Chi2ParticleID                        4.1168e-05    0.000246448   0.00753583    0.00016518    0.000544549      200    
reco:trkshowersplit:TrackShowerHits                   0.000108537    0.0327866      1.31731     0.00118425     0.134446        200    
reco:pmtrack:PMAlgTrackMaker                          0.00138229      3.81631       86.5541      0.321525       12.2867        200    
reco:pmtrackcalo:Calorimetry                          6.6296e-05     0.0183004     0.740255     0.00139561     0.0736015       200    
reco:pmtrackpid:Chi2ParticleID                        2.7593e-05    0.00040858    0.00446128    0.000200713   0.000632138      200    
reco:pmtrajfit:PMAlgTrajFitter                        0.000201865     3.1099        88.4362      0.426693       9.27747        200    
reco:pmtrajfitcalo:Calorimetry                        6.0545e-05     0.0257331     0.804776     0.00218905     0.0967027       200    
reco:pmtrajfitpid:Chi2ParticleID                       2.62e-05     0.000270524   0.00427715    0.00019662    0.000354724      200    
reco:pmtracktc:PMAlgTrackMaker                        0.00108109      4.84012       101.032      0.377095       14.9005        200    
reco:pmtracktccalo:Calorimetry                        6.0937e-05     0.0128016      0.48461      0.001571      0.0473676       200    
reco:pmtracktcpid:Chi2ParticleID                      2.6821e-05    0.00039026    0.00426327    0.000198203   0.000584059      200    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000182708     2.51109       56.0689      0.451692       6.8126         200    
reco:pmtrajfittccalo:Calorimetry                      6.0506e-05     0.0244155     0.797579     0.00186794     0.0915807       200    
reco:pmtrajfittcpid:Chi2ParticleID                    2.6351e-05    0.000528397   0.00521032    0.000226528   0.000840351      200    
reco:blurredcluster:BlurredClustering                 0.00013608      1.40867       11.6624      0.690422       2.00285        200    
reco:emtrkmichelid:EmTrackMichelId                    0.000116873     15.9864       218.161       4.63245       32.5168        200    
reco:emshower:EMShower                                0.000162359    0.0511293     0.872151      0.0114342     0.0956925       200    
reco:cvnmap:CVNMapper                                 1.9037e-05     0.0166308     0.0922998     0.0159753     0.0155672       200    
reco:cvneva:CVNEvaluator                              1.7834e-05      1.06394       3.00976       1.03751      0.752559        200    
reco:energyrecnumu:EnergyReco                         0.000368302    0.0558207      1.75244     0.00839072      0.19023        200    
reco:energyrecnue:EnergyReco                          0.000137723   0.00335404     0.0616436    0.000753834   0.00753355       200    
reco:energyrecnc:EnergyReco                           0.00196049    0.00691632     0.0657348    0.00388581    0.00833485       200    
reco:energyrecnumurange:EnergyReco                    0.000141049    0.0466465      1.55428     0.00385705      0.17663        200    
reco:energyrecnumumcs:EnergyReco                      0.00131233     0.0481393      1.56996     0.00614448     0.176056        200    
reco:opdec:Deconvolution                              0.00525162     0.253672       1.10337      0.239766      0.143958        200    
reco:ophitspe:OpHitFinderDeco                         0.00438699      1.93186       6.73768       1.82855       1.00246        200    
reco:opflash:OpFlashFinder                            0.000127894   0.00177874    0.00724896    0.00167822    0.00087988       200    
reco:rns:RandomNumberSaver                            2.0639e-05    4.74632e-05   0.000456642   4.6083e-05    3.29515e-05      200    
reco:anglereconue:NuAngularReco                        9.432e-05    0.00226974      0.05689     0.000497419   0.00661112       200    
reco:anglereconumu:NuAngularReco                      6.6748e-05    0.00196374     0.0562586    0.000270105   0.00607732       200    
reco:anglereconuepfps:NuAngularReco                   0.000122945   0.00210317     0.0561259    0.000469972    0.0060619       200    
reco:anglereconumupfps:NuAngularReco                   9.968e-05    0.00205717     0.0571562    0.000357212   0.00623314       200    
[art]:TriggerResults:TriggerResultInserter             9.228e-06    1.59386e-05   6.7528e-05    1.54645e-05   6.06261e-06      200    
end_path:out1:RootOutput                               3.306e-06    7.31341e-06   2.9266e-05    7.6845e-06    3.40171e-06      200    
end_path:out1:RootOutput(write)                        0.0124478     0.318207       1.66151      0.308733      0.243358        200    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 17653.2 MB
  Peak resident set size usage (VmHWM): 2665.29 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1497_386_20231027T144707Z_gen_g4_detsim_hitreco__20240503T040206Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1497_386_20231027T144707Z_gen_g4_detsim_hitreco__20240503T040206Z_reco2_reco_2024-11-23T_050849Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1497_386_20231027T144707Z_gen_g4_detsim_hitreco__20240503T040206Z_reco2_reco_data_2024-11-23T_050849Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work