Jobsub ID 108593.81@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108593.81@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-22 23:25:00
Site	CA_SFU
Entry	DUNE_CA_SFU_lcg-ce3
Last heartbeat	2024-11-23 00:59:07
From worker node	Hostname	cdr52.int.cedar.computecanada.ca
	cpuinfo	Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	84598 (23 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-22 23:36:03
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577113_459_20231204T012846Z_gen_g4_detsim_hitreco__20240509T215655Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (4969s)
	CPU time	1h (4059s = 81%)
Outputting started	2024-11-23 00:58:53
Output files
Finished	2024-11-23 00:59:07
List job events Wrapper job log
Jobscript log (last 10,000 characters)

16:58:23 PST run: 50577113 subRun: 1 event: 46000
Hit sorting problem.
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 16:58:24 PST run: 50577113 subRun: 1 event: 46000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 16:58:25 PST run: 50577113 subRun: 1 event: 46000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 16:58:25 PST run: 50577113 subRun: 1 event: 46000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 16:58:25 PST run: 50577113 subRun: 1 event: 46000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 16:58:25 PST run: 50577113 subRun: 1 event: 46000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 16:58:26 PST run: 50577113 subRun: 1 event: 46000
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 511, 822, 427
minwire 0: 1849
minwire 1: 569
minwire 2: 1759
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.00262232, 0.000110466, 0.997267, 
Output 1: 0.494324, 0.378061, 0.0903284, 0.0372859, 
Output 2: 0.885553, 0.114447, 

623
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2210!
22-Nov-2024 16:58:52 PST  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577113_459_20231204T012846Z_gen_g4_detsim_hitreco__20240509T215655Z_reco2_reco_data_2024-11-22T_233614Z.root"
22-Nov-2024 16:58:52 PST  Closed input file "root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/fardet-hd/39/17/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577113_459_20231204T012846Z_gen_g4_detsim_hitreco__20240509T215655Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.12846       42.2353       1196.4        14.692        123.192        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0736419     0.157157      0.248612      0.164943      0.0424637       100    
reco:linecluster:LineCluster                           0.435115      0.974769       15.8133      0.714941       1.71804        100    
reco:trajcluster:TrajCluster                            0.21763       1.35526       61.1363      0.234318       6.47077        100    
reco:pandora:StandardPandora                           0.232703       6.38241       415.48        1.91194       41.1355        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000182909    0.0130214     0.661987     0.00225052     0.0660456       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.00025279     0.0262182     0.402098     0.00435488     0.0641677       100    
reco:pandoracalo:Calorimetry                          0.000139548    0.0314532      1.29072     0.00171318     0.138507        100    
reco:pandorapid:Chi2ParticleID                        5.6357e-05    0.000209512   0.00149221    0.000168996   0.000176916      100    
reco:trkshowersplit:TrackShowerHits                   8.4415e-05     0.0252362      1.3888      0.00127536     0.142506        100    
reco:pmtrack:PMAlgTrackMaker                          0.00171138      3.01683       94.3893      0.228036       10.4803        100    
reco:pmtrackcalo:Calorimetry                          0.000119941    0.0218001      1.29197      0.0014387      0.12933        100    
reco:pmtrackpid:Chi2ParticleID                         4.823e-05    0.000392067   0.00554166    0.000211203   0.000643501      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000282634     2.53361       74.9709      0.373703       8.45318        100    
reco:pmtrajfitcalo:Calorimetry                        0.000113065    0.033916       1.4349      0.00207089     0.152093        100    
reco:pmtrajfitpid:Chi2ParticleID                      4.7905e-05    0.000248242   0.00228564    0.000184694   0.000256911      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00153838      3.4787        115.438      0.321539       12.665         100    
reco:pmtracktccalo:Calorimetry                        0.000119149    0.0144663     0.630988     0.00142274     0.0640229       100    
reco:pmtracktcpid:Chi2ParticleID                      4.8594e-05    0.000358081   0.00432142    0.000223457   0.000498417      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000276381     1.87906       52.0683      0.412063       5.57965        100    
reco:pmtrajfittccalo:Calorimetry                      0.000114518    0.0291036      1.70227     0.00178139     0.170949        100    
reco:pmtrajfittcpid:Chi2ParticleID                    4.7655e-05    0.00045908    0.00716033    0.000236945   0.000812952      100    
reco:blurredcluster:BlurredClustering                 9.2235e-05      1.68575       12.6099      0.958246       2.07897        100    
reco:emtrkmichelid:EmTrackMichelId                    0.00015536       15.81        312.324       4.85156       35.2604        100    
reco:emshower:EMShower                                0.000232155    0.0411963     0.386725     0.00874782     0.0713564       100    
reco:cvnmap:CVNMapper                                 3.0272e-05     0.015686      0.0564621     0.0145672     0.0117814       100    
reco:cvneva:CVNEvaluator                              2.4974e-05     0.879994       2.9107        1.14205      0.539555        100    
reco:energyrecnumu:EnergyReco                         0.000367075    0.117935       7.81127     0.00560749     0.780849        100    
reco:energyrecnue:EnergyReco                          0.000232333   0.00428726     0.206397     0.000807036    0.0205828       100    
reco:energyrecnc:EnergyReco                           0.00298752    0.00707003     0.214901     0.00348216     0.0211742       100    
reco:energyrecnumurange:EnergyReco                    0.000238575    0.116911       7.91103     0.00328947     0.790806        100    
reco:energyrecnumumcs:EnergyReco                      0.00139941     0.119103       7.96464     0.00425963     0.795889        100    
reco:opdec:Deconvolution                                0.22219      0.781747       1.22495      0.772245      0.205033        100    
reco:ophitspe:OpHitFinderDeco                          0.681266        2.343        3.73487       2.41104      0.686523        100    
reco:opflash:OpFlashFinder                            0.000236176   0.00172352     0.0045943     0.001718     0.000748284      100    
reco:rns:RandomNumberSaver                            2.2286e-05    3.79464e-05   0.00033738    3.19005e-05   3.17692e-05      100    
reco:anglereconue:NuAngularReco                       0.000182591   0.00340253     0.194207     0.00043121     0.0193483       100    
reco:anglereconumu:NuAngularReco                      0.000117627    0.003252      0.191266     0.000310167    0.0190686       100    
reco:anglereconuepfps:NuAngularReco                   0.00017632    0.00341093     0.190987     0.000454471    0.0190314       100    
reco:anglereconumupfps:NuAngularReco                  0.000169585   0.00337864     0.192219     0.000406225    0.0191588       100    
[art]:TriggerResults:TriggerResultInserter            1.4262e-05    1.69586e-05    5.545e-05    1.58785e-05   4.40494e-06      100    
end_path:out1:RootOutput                               3.102e-06    5.97544e-06   1.9393e-05     4.667e-06    2.9367e-06       100    
end_path:out1:RootOutput(write)                        0.0128379     0.320575       1.80545      0.315343      0.228734        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8423.11 MB
  Peak resident set size usage (VmHWM): 2437.89 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577113_459_20231204T012846Z_gen_g4_detsim_hitreco__20240509T215655Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577113_459_20231204T012846Z_gen_g4_detsim_hitreco__20240509T215655Z_reco2_reco_2024-11-22T_233614Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50577113_459_20231204T012846Z_gen_g4_detsim_hitreco__20240509T215655Z_reco2_reco_data_2024-11-22T_233614Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work