Jobsub ID 301628.2@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301628.2@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 01:47:02
Site	US_UChicago
Entry	Engage_US_MWT2_uiuc_gk02_condce_mcore
Last heartbeat	2024-11-23 02:54:20
From worker node	Hostname	iut2-c267.iu.edu
	cpuinfo	Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	86400 (24 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 01:47:40
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406796_962_20231202T140203Z_gen_g4_detsim_hitreco__20240507T223820Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (3980s)
	CPU time	1h (3665s = 92%)
Outputting started	2024-11-23 02:54:01
Output files
Finished	2024-11-23 02:54:20
List job events Wrapper job log
Jobscript log (last 10,000 characters)

06796 subRun: 1 event: 96300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 21:53:43 EST run: 6406796 subRun: 1 event: 96300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 21:53:43 EST run: 6406796 subRun: 1 event: 96300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 21:53:43 EST run: 6406796 subRun: 1 event: 96300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 21:53:43 EST run: 6406796 subRun: 1 event: 96300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 21:53:43 EST run: 6406796 subRun: 1 event: 96300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 21:53:43 EST run: 6406796 subRun: 1 event: 96300
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 151, 323, 227
minwire 0: 937
minwire 1: 1626
minwire 2: 745
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0078207, 0.000358363, 0.991821, 
Output 1: 0.802974, 0.173733, 0.0194228, 0.00386959, 
Output 2: 0.958512, 0.0414875, 

536
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1703!
22-Nov-2024 21:53:58 EST  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406796_962_20231202T140203Z_gen_g4_detsim_hitreco__20240507T223820Z_reco2_reco_data_2024-11-23T_014745Z.root"
22-Nov-2024 21:53:58 EST  Closed input file "root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/fardet-hd/bb/a5/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406796_962_20231202T140203Z_gen_g4_detsim_hitreco__20240507T223820Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              4.22278       36.2671       625.713       18.2265       70.1286        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0274542      0.11366      0.315632      0.105794      0.0595995       100    
reco:linecluster:LineCluster                           0.248695      0.509093       15.275       0.294936       1.5102         100    
reco:trajcluster:TrajCluster                           0.0780955      1.12955       74.967        0.10204       7.47573        100    
reco:pandora:StandardPandora                            1.75378       3.04256       28.8508       2.46868       2.92503        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000237238   0.00906718     0.170408     0.00369902     0.0195619       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000284922    0.0214236     0.746861     0.00626254     0.0765394       100    
reco:pandoracalo:Calorimetry                          0.000166212    0.0204788     0.653416     0.00227389     0.0755237       100    
reco:pandorapid:Chi2ParticleID                         5.614e-05    0.000298222   0.00302511    0.000261411   0.000307857      100    
reco:trkshowersplit:TrackShowerHits                   0.000230962    0.0217282      1.01532     0.00165152     0.105117        100    
reco:pmtrack:PMAlgTrackMaker                          0.00191774      2.77545       85.1097      0.404405       9.54393        100    
reco:pmtrackcalo:Calorimetry                          0.000151817    0.0133225     0.654606     0.00187744     0.0660305       100    
reco:pmtrackpid:Chi2ParticleID                        5.5913e-05    0.000440353   0.00634278    0.000270212   0.000680788      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000338364     2.18444       53.348       0.725473       5.89983        100    
reco:pmtrajfitcalo:Calorimetry                        0.000115378    0.0226845     0.796866     0.00275861     0.086653        100    
reco:pmtrajfitpid:Chi2ParticleID                      4.5728e-05    0.000335263   0.00166243    0.000276463   0.000245701      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00197801      2.91909       98.6055      0.421395       10.5203        100    
reco:pmtracktccalo:Calorimetry                        0.000130532   0.00975664     0.371083     0.00217932     0.0379383       100    
reco:pmtracktcpid:Chi2ParticleID                      6.8882e-05    0.000433333   0.00414682    0.000292667   0.000489028      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000408779     1.75691       26.8435      0.586177       3.63598        100    
reco:pmtrajfittccalo:Calorimetry                      0.000141883    0.0151578      0.45779     0.00237591     0.0515898       100    
reco:pmtrajfittcpid:Chi2ParticleID                    5.3507e-05    0.000514606   0.00356577    0.00030459    0.000623773      100    
reco:blurredcluster:BlurredClustering                  0.0213167      1.6171        11.0919       1.08236       1.80302        100    
reco:emtrkmichelid:EmTrackMichelId                     0.398033       15.194        216.531       6.52898       28.1476        100    
reco:emshower:EMShower                                0.000342863    0.0613613     0.624737      0.0222014     0.0974084       100    
reco:cvnmap:CVNMapper                                 4.2497e-05     0.0270104     0.0698803     0.0329593     0.0175987       100    
reco:cvneva:CVNEvaluator                              2.6942e-05      1.19213       1.97964       1.53206      0.644309        100    
reco:energyrecnumu:EnergyReco                         0.00070244     0.0255376     0.468513     0.00863571     0.060871        100    
reco:energyrecnue:EnergyReco                          0.000259613   0.00309284     0.0631886    0.00121097    0.00690879       100    
reco:energyrecnc:EnergyReco                            0.0027445    0.00604857     0.064335     0.00427413    0.00671098       100    
reco:energyrecnumurange:EnergyReco                    0.000496222    0.0215899     0.464819     0.00402914     0.0594696       100    
reco:energyrecnumumcs:EnergyReco                      0.00220576     0.0238731     0.466644      0.0060595     0.0596828       100    
reco:opdec:Deconvolution                               0.181774      0.512423       1.14889       0.46496      0.213931        100    
reco:ophitspe:OpHitFinderDeco                          0.718794       2.5933        4.58772       2.58686      0.872159        100    
reco:opflash:OpFlashFinder                            0.00063694    0.00273294    0.00695669     0.0026169    0.00101195       100    
reco:rns:RandomNumberSaver                            3.4453e-05    6.78151e-05   0.000387835   6.2794e-05    3.59689e-05      100    
reco:anglereconue:NuAngularReco                       0.000281901   0.00229952     0.0528583    0.000922904   0.00570415       100    
reco:anglereconumu:NuAngularReco                      0.000152836   0.00194052     0.0525169    0.000513672   0.00565613       100    
reco:anglereconuepfps:NuAngularReco                   0.000229851   0.00221092     0.0537867    0.000768007   0.00577138       100    
reco:anglereconumupfps:NuAngularReco                  0.000219253   0.00211374     0.053229     0.000603533   0.00577675       100    
[art]:TriggerResults:TriggerResultInserter            1.4681e-05    2.47913e-05   6.1493e-05    2.33395e-05   6.73113e-06      100    
end_path:out1:RootOutput                               4.093e-06    8.84974e-06   3.0885e-05    8.3145e-06    3.54057e-06      100    
end_path:out1:RootOutput(write)                        0.0450707     0.408518       1.58703      0.421392      0.230176        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9381.98 MB
  Peak resident set size usage (VmHWM): 2180.76 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406796_962_20231202T140203Z_gen_g4_detsim_hitreco__20240507T223820Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406796_962_20231202T140203Z_gen_g4_detsim_hitreco__20240507T223820Z_reco2_reco_2024-11-23T_014745Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406796_962_20231202T140203Z_gen_g4_detsim_hitreco__20240507T223820Z_reco2_reco_data_2024-11-23T_014745Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work