Jobsub ID 108654.35@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108654.35@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 01:04:45
Site	US_UChicago
Entry	Engage_US_MWT2_iut2_condce_mcore
Last heartbeat	2024-11-23 02:58:55
From worker node	Hostname	iut2-c276.iu.edu
	cpuinfo	Intel(R) Xeon(R) Gold 6148 CPU @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	86400 (24 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 01:23:35
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419482_1_20231203T163423Z_gen_g4_detsim_hitreco__20240509T213022Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (5698s)
	CPU time	1h (5300s = 93%)
Outputting started	2024-11-23 02:58:34
Output files
Finished	2024-11-23 02:58:55
List job events Wrapper job log
Jobscript log (last 10,000 characters)

: 200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 21:57:52 EST run: 6419482 subRun: 1 event: 200
1st wire C:0 T:20 P:2 W:952 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 21:57:52 EST run: 6419482 subRun: 1 event: 200
1st wire C:0 T:20 P:2 W:985 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 21:57:53 EST run: 6419482 subRun: 1 event: 200
1st wire C:0 T:22 P:2 W:911 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 21:57:53 EST run: 6419482 subRun: 1 event: 200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 21:57:53 EST run: 6419482 subRun: 1 event: 200
1st wire C:0 T:23 P:2 W:751 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 21:57:54 EST run: 6419482 subRun: 1 event: 200
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 660, 726, 210
minwire 0: 2567
minwire 1: 321
minwire 2: 2682
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.00170013, 0.000119435, 0.998181, 
Output 1: 0.446105, 0.385681, 0.115571, 0.0526427, 
Output 2: 0.851149, 0.148851, 

526
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1816!
22-Nov-2024 21:58:32 EST  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419482_1_20231203T163423Z_gen_g4_detsim_hitreco__20240509T213022Z_reco2_reco_data_2024-11-23T_012341Z.root"
22-Nov-2024 21:58:32 EST  Closed input file "root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/fardet-hd/32/fe/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419482_1_20231203T163423Z_gen_g4_detsim_hitreco__20240509T213022Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.956172       53.1166       970.75        17.249        135.545        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.027757      0.088909      0.345144      0.0861264     0.0445104       100    
reco:linecluster:LineCluster                           0.176399      0.780206       40.1411       0.29795       3.98971        100    
reco:trajcluster:TrajCluster                           0.0771632      2.29621       111.361      0.0923737      13.0843        100    
reco:pandora:StandardPandora                           0.107611       3.80913       57.4265       2.10449       7.87716        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000241086    0.0148492     0.314945     0.00234168     0.0440449       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000313436    0.0366243     0.981457     0.00421896     0.125864        100    
reco:pandoracalo:Calorimetry                          0.000161488     0.03931       1.42071     0.00153817     0.182821        100    
reco:pandorapid:Chi2ParticleID                        6.2586e-05    0.000324013   0.00368031    0.000241053   0.000402737      100    
reco:trkshowersplit:TrackShowerHits                   0.000200169    0.0389234      1.12378     0.00127999     0.171408        100    
reco:pmtrack:PMAlgTrackMaker                          0.00201621      4.90199       182.05       0.315833       20.6575        100    
reco:pmtrackcalo:Calorimetry                          0.000146869    0.0226314     0.552484     0.00126123     0.0830941       100    
reco:pmtrackpid:Chi2ParticleID                        5.2918e-05    0.00051244    0.00861864    0.000239118    0.0010408       100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000377786     3.05055       70.337       0.344564       10.4075        100    
reco:pmtrajfitcalo:Calorimetry                        0.000104703    0.0413654      1.48012     0.00191618     0.186687        100    
reco:pmtrajfitpid:Chi2ParticleID                      4.2714e-05    0.000348569   0.00178761    0.000258285   0.000316482      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00172062      5.43277       179.921      0.335686       22.143         100    
reco:pmtracktccalo:Calorimetry                        0.000134754    0.0177804     0.433118     0.00140649     0.0598959       100    
reco:pmtracktcpid:Chi2ParticleID                      5.3259e-05    0.000470695    0.0045769    0.000262005   0.000702113      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000368623     2.91548       47.6876      0.431373       8.27768        100    
reco:pmtrajfittccalo:Calorimetry                      0.000117802    0.0331073     0.954486     0.00177212     0.137174        100    
reco:pmtrajfittcpid:Chi2ParticleID                    4.4483e-05    0.000580602   0.00695588    0.000261568   0.00101967       100    
reco:blurredcluster:BlurredClustering                 0.000144691     1.91355       17.1132       0.80151        2.854         100    
reco:emtrkmichelid:EmTrackMichelId                    0.000201616     21.0214       284.727       4.72346       49.6446        100    
reco:emshower:EMShower                                0.00028322     0.060729      0.738984     0.00489267     0.122955        100    
reco:cvnmap:CVNMapper                                 3.7132e-05     0.0303978     0.114773      0.0214582     0.0283065       100    
reco:cvneva:CVNEvaluator                              2.7572e-05      1.0922        1.96234       1.47467      0.760689        100    
reco:energyrecnumu:EnergyReco                         0.000529644    0.127832       6.38592     0.00792383     0.657863        100    
reco:energyrecnue:EnergyReco                          0.000259353   0.00460974     0.0764338    0.00116971      0.01124        100    
reco:energyrecnc:EnergyReco                           0.00319594     0.0079514     0.080002     0.00470422     0.0112579       100    
reco:energyrecnumurange:EnergyReco                    0.000437691    0.123478       6.35534     0.00440677     0.654134        100    
reco:energyrecnumumcs:EnergyReco                      0.00349745     0.125512       6.38475     0.00592412     0.655673        100    
reco:opdec:Deconvolution                               0.105303      0.516748      0.972274      0.479919      0.191712        100    
reco:ophitspe:OpHitFinderDeco                          0.230558       4.15798       8.84427       3.98238       1.84161        100    
reco:opflash:OpFlashFinder                            0.000300854   0.00244848    0.00770048    0.00230653    0.00118737       100    
reco:rns:RandomNumberSaver                            3.0907e-05    6.96523e-05   0.000526873   6.0285e-05    5.34752e-05      100    
reco:anglereconue:NuAngularReco                       0.00025132    0.00369838     0.085043     0.000782223    0.0109958       100    
reco:anglereconumu:NuAngularReco                      0.00012339    0.00327404     0.084474     0.000394045    0.0108386       100    
reco:anglereconuepfps:NuAngularReco                   0.000193859   0.00352882     0.085047     0.000655901    0.010936        100    
reco:anglereconumupfps:NuAngularReco                  0.000179539    0.0034366     0.0847558    0.000531792    0.0109307       100    
[art]:TriggerResults:TriggerResultInserter            1.3815e-05    2.50724e-05   0.000101738   2.44115e-05   1.05065e-05      100    
end_path:out1:RootOutput                               3.506e-06    8.83837e-06   3.5294e-05     8.285e-06    4.8868e-06       100    
end_path:out1:RootOutput(write)                        0.0222182      0.39195       2.01123      0.366494       0.33325        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 15129 MB
  Peak resident set size usage (VmHWM): 2384.31 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419482_1_20231203T163423Z_gen_g4_detsim_hitreco__20240509T213022Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419482_1_20231203T163423Z_gen_g4_detsim_hitreco__20240509T213022Z_reco2_reco_2024-11-23T_012341Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6419482_1_20231203T163423Z_gen_g4_detsim_hitreco__20240509T213022Z_reco2_reco_data_2024-11-23T_012341Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work