Jobsub ID 301640.72@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301640.72@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 02:11:22
Site	UK_Lancaster
Entry	UBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat	2024-11-23 02:59:38
From worker node	Hostname	comp21-00
	cpuinfo	Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 02:19:14
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420351_864_20231204T044618Z_gen_g4_detsim_hitreco__20240509T200934Z_reco2.root
Jobscript	Exit code	0
	Real time	40m (2409s)
	CPU time	37m (2254s = 93%)
Outputting started	2024-11-23 02:59:23
Output files
Finished	2024-11-23 02:59:38
List job events Wrapper job log
Jobscript log (last 10,000 characters)

ov-2024 02:59:16 GMT run: 6420351 subRun: 1 event: 86500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 02:59:16 GMT run: 6420351 subRun: 1 event: 86500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 02:59:16 GMT run: 6420351 subRun: 1 event: 86500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 02:59:16 GMT run: 6420351 subRun: 1 event: 86500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 02:59:19 GMT run: 6420351 subRun: 1 event: 86500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 02:59:19 GMT run: 6420351 subRun: 1 event: 86500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 02:59:19 GMT run: 6420351 subRun: 1 event: 86500
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 101, 134, 92
minwire 0: 2480
minwire 1: 137
minwire 2: 2789
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.52207, 0.451658, 0.0262715, 
Output 1: 0.175967, 0.780577, 0.0426826, 0.000773511, 
Output 2: 0.982196, 0.0178044, 

282
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 768!
23-Nov-2024 02:59:22 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420351_864_20231204T044618Z_gen_g4_detsim_hitreco__20240509T200934Z_reco2_reco_data_2024-11-23T_021918Z.root"
23-Nov-2024 02:59:22 GMT  Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/26/bb/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420351_864_20231204T044618Z_gen_g4_detsim_hitreco__20240509T200934Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.503588       22.4173       278.932       10.0352       39.3672        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0143376     0.0358723     0.0705015     0.0326842     0.0119686       100    
reco:linecluster:LineCluster                           0.082884      0.352104       11.1904      0.157274       1.16589        100    
reco:trajcluster:TrajCluster                           0.0419234     0.604974       22.8307      0.0646156      2.59096        100    
reco:pandora:StandardPandora                           0.0569904      1.83109       47.4836       1.12015       4.69028        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000119919   0.00444013     0.0548485    0.00147331    0.00851536       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000151815    0.0128243      0.46433     0.00234018     0.0476178       100    
reco:pandoracalo:Calorimetry                          9.3762e-05     0.0102442     0.300564     0.00107358     0.035365        100    
reco:pandorapid:Chi2ParticleID                         3.28e-05     0.000132849   0.00110932    9.78835e-05   0.000122886      100    
reco:trkshowersplit:TrackShowerHits                    8.398e-05     0.013731      0.395252     0.000909782    0.0535661       100    
reco:pmtrack:PMAlgTrackMaker                          0.00105178      1.33728       21.5668      0.175086       3.47739        100    
reco:pmtrackcalo:Calorimetry                          6.9595e-05    0.00549744     0.0901349    0.000895746    0.0148334       100    
reco:pmtrackpid:Chi2ParticleID                        2.8763e-05    0.000202556    0.0015535    0.000131846   0.000257287      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000173775     1.23795       18.779       0.256013       2.91897        100    
reco:pmtrajfitcalo:Calorimetry                        5.5971e-05     0.0111576     0.314827     0.00159427     0.0374386       100    
reco:pmtrajfitpid:Chi2ParticleID                      2.5682e-05    0.000151744   0.000568632   0.000135244   0.000104232      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00096181      1.55298       24.3814      0.211968       4.12918        100    
reco:pmtracktccalo:Calorimetry                        6.4011e-05     0.004834      0.0622525    0.000915927    0.0109705       100    
reco:pmtracktcpid:Chi2ParticleID                       2.581e-05    0.000192961   0.00111072    0.000126027   0.000210137      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000147925    0.911894       14.3677      0.305266       1.99105        100    
reco:pmtrajfittccalo:Calorimetry                      5.6771e-05    0.00964506     0.239665     0.00126325     0.0311537       100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.3412e-05    0.000261055   0.00229547    0.000155686   0.000367482      100    
reco:blurredcluster:BlurredClustering                 8.0284e-05      1.07044       7.70124       0.57946       1.44779        100    
reco:emtrkmichelid:EmTrackMichelId                     0.0001318      10.2565       105.876       3.9092        18.2326        100    
reco:emshower:EMShower                                0.00021235      0.02014      0.219897     0.00420143     0.0354512       100    
reco:cvnmap:CVNMapper                                 1.9217e-05     0.0129097     0.0379529     0.0130772    0.00917203       100    
reco:cvneva:CVNEvaluator                              1.4658e-05     0.541115      0.824596      0.700829      0.313653        100    
reco:energyrecnumu:EnergyReco                         0.000256211    0.0384985      1.04911     0.00463445      0.13605        100    
reco:energyrecnue:EnergyReco                          0.000128948   0.00147929     0.0173489    0.000487248   0.00274336       100    
reco:energyrecnc:EnergyReco                           0.00166663    0.00306291     0.0200543    0.00206136    0.00285203       100    
reco:energyrecnumurange:EnergyReco                    0.000187419    0.0355691      1.02042     0.00200856     0.132405        100    
reco:energyrecnumumcs:EnergyReco                      0.00165493     0.0366692      1.01775      0.0024424     0.132631        100    
reco:opdec:Deconvolution                               0.0670657     0.259669      0.425978      0.258041      0.0765081       100    
reco:ophitspe:OpHitFinderDeco                          0.122502       2.00283       3.83943       2.01732       0.79759        100    
reco:opflash:OpFlashFinder                            0.000237982   0.00113164    0.00387025    0.00109434    0.000524111      100    
reco:rns:RandomNumberSaver                            1.6568e-05    3.02726e-05   0.000220211   2.8398e-05    2.10767e-05      100    
reco:anglereconue:NuAngularReco                       0.000132687   0.00111564     0.0141555    0.000331027   0.00227007       100    
reco:anglereconumu:NuAngularReco                       6.367e-05    0.000953774    0.0131235    0.00020032    0.00216301       100    
reco:anglereconuepfps:NuAngularReco                   0.000104349   0.00105964     0.0133632    0.000293521   0.00218389       100    
reco:anglereconumupfps:NuAngularReco                  8.7444e-05    0.00102381     0.0132926    0.000264973   0.00219463       100    
[art]:TriggerResults:TriggerResultInserter             8.427e-06    1.08295e-05   3.8623e-05    1.0071e-05    3.59682e-06      100    
end_path:out1:RootOutput                               2.045e-06    4.6596e-06    1.7809e-05     4.808e-06    2.41348e-06      100    
end_path:out1:RootOutput(write)                        0.0133613     0.194148      0.638545      0.197153      0.119083        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9650.87 MB
  Peak resident set size usage (VmHWM): 2496.48 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420351_864_20231204T044618Z_gen_g4_detsim_hitreco__20240509T200934Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420351_864_20231204T044618Z_gen_g4_detsim_hitreco__20240509T200934Z_reco2_reco_2024-11-23T_021918Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6420351_864_20231204T044618Z_gen_g4_detsim_hitreco__20240509T200934Z_reco2_reco_data_2024-11-23T_021918Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work