Jobsub ID 108522.66@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108522.66@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-22 22:48:20
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce515
Last heartbeat	2024-11-23 00:13:33
From worker node	Hostname	b9g37p9661.cern.ch
	cpuinfo	Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	stalled
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-22 23:00:49
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50573344_691_20231203T083206Z_gen_g4_detsim_hitreco__20240508T040111Z_reco2.root
Outputting started	2024-11-23 00:13:33
Output files
Finished	2024-11-23 00:34:31
List job events Wrapper job log
Jobscript log (last 10,000 characters)

subRun: 1 event: 69200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 01:13:21 CET run: 50573344 subRun: 1 event: 69200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 01:13:21 CET run: 50573344 subRun: 1 event: 69200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 01:13:21 CET run: 50573344 subRun: 1 event: 69200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 01:13:21 CET run: 50573344 subRun: 1 event: 69200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 01:13:21 CET run: 50573344 subRun: 1 event: 69200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 01:13:22 CET run: 50573344 subRun: 1 event: 69200
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 63, 92, 92
minwire 0: 2405
minwire 1: 135
minwire 2: 2713
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0992168, 0.551566, 0.349217, 
Output 1: 0.981977, 0.0170468, 0.000755203, 0.000220915, 
Output 2: 0.951571, 0.0484287, 

170
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 392!
23-Nov-2024 01:13:31 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50573344_691_20231203T083206Z_gen_g4_detsim_hitreco__20240508T040111Z_reco2_reco_data_2024-11-22T_230055Z.root"
23-Nov-2024 01:13:31 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/45/6d/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50573344_691_20231203T083206Z_gen_g4_detsim_hitreco__20240508T040111Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.595513       40.3607       939.138       14.9068       111.749        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.017935      0.0416461     0.131599      0.0400288     0.0174937       100    
reco:linecluster:LineCluster                           0.0950621     0.546178       24.4241      0.187493       2.55064        100    
reco:trajcluster:TrajCluster                           0.0478458      1.54627       79.0384      0.0628126      8.50092        100    
reco:pandora:StandardPandora                           0.0659905      3.87651       104.389       2.33565       10.764         100    
reco:pandoraTrack:LArPandoraTrackCreation             0.00018589     0.0123432     0.316635     0.00227168     0.0417315       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000237891    0.0239202      1.04575     0.00498969     0.105337        100    
reco:pandoracalo:Calorimetry                          0.000147388    0.0421279      1.89955     0.00141554     0.224314        100    
reco:pandorapid:Chi2ParticleID                        5.3947e-05    0.000240993   0.00127969    0.00019821    0.000196715      100    
reco:trkshowersplit:TrackShowerHits                   8.5284e-05     0.0434221      2.45964     0.00101718     0.255945        100    
reco:pmtrack:PMAlgTrackMaker                           0.0018826      3.09065       90.9082      0.150353       11.6832        100    
reco:pmtrackcalo:Calorimetry                          0.000136991    0.0157223     0.444715     0.00120969     0.0604173       100    
reco:pmtrackpid:Chi2ParticleID                         5.197e-05    0.000380088   0.00464487    0.000212761   0.000626773      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000324279     3.56395       145.03       0.392807       15.7951        100    
reco:pmtrajfitcalo:Calorimetry                        0.000104778    0.0396441      1.62134     0.00212305      0.19314        100    
reco:pmtrajfitpid:Chi2ParticleID                      4.3555e-05    0.000293757    0.0012978    0.000216862   0.000256971      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00186019      3.92097       129.232      0.303499       15.6807        100    
reco:pmtracktccalo:Calorimetry                        0.000113214    0.0122641     0.286763     0.00138001     0.0425417       100    
reco:pmtracktcpid:Chi2ParticleID                      4.5086e-05    0.000383752   0.00419021    0.000216711   0.000577588      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000366656     2.37569       69.6523      0.395064       7.96173        100    
reco:pmtrajfittccalo:Calorimetry                      0.000142883    0.0296635      1.28387     0.00161775     0.140723        100    
reco:pmtrajfittcpid:Chi2ParticleID                    6.1376e-05    0.000498018   0.00556171    0.000235285   0.000865276      100    
reco:blurredcluster:BlurredClustering                 0.000125566     1.30546       12.4882      0.723195       1.91624        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000226833     15.7208       265.013       4.21732       37.9679        100    
reco:emshower:EMShower                                0.000308202    0.0412858     0.542729     0.00469023     0.0937242       100    
reco:cvnmap:CVNMapper                                 3.5346e-05     0.0190607     0.0564866     0.0253276     0.015453        100    
reco:cvneva:CVNEvaluator                              2.2408e-05     0.946647       1.74818       1.37113      0.721881        100    
reco:energyrecnumu:EnergyReco                         0.000446461    0.061519      0.794326      0.0061287     0.164052        100    
reco:energyrecnue:EnergyReco                          0.000221269   0.00437144     0.105913     0.00110746     0.0138672       100    
reco:energyrecnc:EnergyReco                            0.002843      0.0071252     0.103783     0.00422613     0.0133537       100    
reco:energyrecnumurange:EnergyReco                    0.000312813    0.0576487     0.827192     0.00394604     0.163791        100    
reco:energyrecnumumcs:EnergyReco                      0.00208517     0.0609479     0.955277     0.00459945     0.169842        100    
reco:opdec:Deconvolution                               0.0515684     0.348243      0.678169      0.352514      0.133758        100    
reco:ophitspe:OpHitFinderDeco                          0.143036       2.22497       5.07692       2.13629       1.09866        100    
reco:opflash:OpFlashFinder                            0.000198939   0.00230135    0.00845091    0.00210957    0.00147112       100    
reco:rns:RandomNumberSaver                            2.2339e-05    5.60941e-05   0.000517624   5.23275e-05   4.80523e-05      100    
reco:anglereconue:NuAngularReco                       0.000183723   0.00323962     0.0958009    0.000572066    0.0122411       100    
reco:anglereconumu:NuAngularReco                      0.000115321   0.00304902     0.107107     0.000320385    0.0130279       100    
reco:anglereconuepfps:NuAngularReco                   0.000168833   0.00311755     0.0875586    0.000551909    0.0115114       100    
reco:anglereconumupfps:NuAngularReco                  0.000149597   0.00300597     0.0880599    0.000414444    0.0115797       100    
[art]:TriggerResults:TriggerResultInserter            1.3405e-05    2.1821e-05    8.8157e-05    2.0823e-05    8.32637e-06      100    
end_path:out1:RootOutput                               3.097e-06    9.11197e-06   3.1457e-05     9.025e-06    3.82548e-06      100    
end_path:out1:RootOutput(write)                        0.0171534     0.362078       2.0312       0.351768      0.320299        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9588.18 MB
  Peak resident set size usage (VmHWM): 2386.67 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50573344_691_20231203T083206Z_gen_g4_detsim_hitreco__20240508T040111Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50573344_691_20231203T083206Z_gen_g4_detsim_hitreco__20240508T040111Z_reco2_reco_2024-11-22T_230055Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50573344_691_20231203T083206Z_gen_g4_detsim_hitreco__20240508T040111Z_reco2_reco_data_2024-11-22T_230055Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work