Jobsub ID 108671.165@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108671.165@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 01:36:08
Site	US_UChicago
Entry	Engage_US_MWT2_iut2_condce_mcore
Last heartbeat	2024-11-23 03:00:57
From worker node	Hostname	iut2-c377.iu.edu
	cpuinfo	AMD EPYC 7302 16-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	86400 (24 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 01:58:42
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66004350_1186_20231202T161111Z_gen_g4_detsim_hitreco__20240508T035137Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (3717s)
	CPU time	55m (3321s = 89%)
Outputting started	2024-11-23 03:00:40
Output files
Finished	2024-11-23 03:00:57
List job events Wrapper job log
Jobscript log (last 10,000 characters)

omparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 22:00:30 EST run: 66004350 subRun: 1 event: 118700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 22:00:30 EST run: 66004350 subRun: 1 event: 118700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 22:00:31 EST run: 66004350 subRun: 1 event: 118700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 22:00:31 EST run: 66004350 subRun: 1 event: 118700
1st wire C:0 T:17 P:2 W:567 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 22:00:31 EST run: 66004350 subRun: 1 event: 118700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 22:00:31 EST run: 66004350 subRun: 1 event: 118700
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 92, 58, 69
minwire 0: 1976
minwire 1: 910
minwire 2: 1944
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.143379, 0.00119679, 0.855424, 
Output 1: 0.309934, 0.669575, 0.0200541, 0.000437182, 
Output 2: 0.851908, 0.148092, 

490
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1628!
22-Nov-2024 22:00:38 EST  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66004350_1186_20231202T161111Z_gen_g4_detsim_hitreco__20240508T035137Z_reco2_reco_data_2024-11-23T_015847Z.root"
22-Nov-2024 22:00:38 EST  Closed input file "root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/fardet-hd/c8/d4/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66004350_1186_20231202T161111Z_gen_g4_detsim_hitreco__20240508T035137Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.852668       33.3226       590.822       14.2939       65.9982        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0268283     0.0985971     0.279685      0.0934007     0.0398793       100    
reco:linecluster:LineCluster                           0.154921      0.466272       12.3737      0.296897       1.22043        100    
reco:trajcluster:TrajCluster                           0.0777295      1.1069        65.7562      0.110234       6.59328        100    
reco:pandora:StandardPandora                           0.0879563      2.53636       50.0076       1.59265       5.08523        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000164189   0.00723627     0.121249     0.00217506     0.0141795       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000232317    0.0166413     0.245605     0.00526544     0.0336123       100    
reco:pandoracalo:Calorimetry                          0.000139743    0.0203406     0.784836     0.00259372     0.082841        100    
reco:pandorapid:Chi2ParticleID                        4.4764e-05    0.000214572    0.0014506    0.000167119   0.000177781      100    
reco:trkshowersplit:TrackShowerHits                   0.000158348    0.0242708      1.04234     0.00166777     0.108446        100    
reco:pmtrack:PMAlgTrackMaker                          0.00133916      2.57295       58.5154      0.475987       6.85852        100    
reco:pmtrackcalo:Calorimetry                          7.7054e-05     0.0104161     0.244953     0.00171473     0.0291697       100    
reco:pmtrackpid:Chi2ParticleID                        3.3773e-05    0.000361694   0.00247939    0.00019946    0.00044571       100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000248717     2.51018       99.4538      0.484269       10.0395        100    
reco:pmtrajfitcalo:Calorimetry                        8.1633e-05     0.0216388     0.880607     0.00311468     0.0903671       100    
reco:pmtrajfitpid:Chi2ParticleID                      3.3693e-05    0.000262992   0.000871939   0.000201579   0.00018733       100    
reco:pmtracktc:PMAlgTrackMaker                        0.00124556      3.05266       77.4212      0.517195       8.8518         100    
reco:pmtracktccalo:Calorimetry                        7.4601e-05    0.00820583     0.114902     0.00170349     0.0185523       100    
reco:pmtracktcpid:Chi2ParticleID                      3.2121e-05    0.000356027   0.00294867    0.000209504   0.000463334      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000227297     1.89788       35.9999      0.601458       4.23742        100    
reco:pmtrajfittccalo:Calorimetry                      8.5731e-05     0.0165736     0.539286     0.00219223     0.0582042       100    
reco:pmtrajfittcpid:Chi2ParticleID                     3.688e-05    0.000463413    0.0045857    0.000241704   0.000641931      100    
reco:blurredcluster:BlurredClustering                 0.000126959     1.29785       10.5798      0.630575       1.70744        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000136546     13.8799       168.431       5.52203       23.2255        100    
reco:emshower:EMShower                                0.000201579    0.0436004     0.488282      0.0157736     0.0704917       100    
reco:cvnmap:CVNMapper                                 2.4606e-05     0.0138871     0.0421849     0.0150643    0.00914801       100    
reco:cvneva:CVNEvaluator                              1.9156e-05     0.767508       1.40476       0.92588      0.407967        100    
reco:energyrecnumu:EnergyReco                         0.000478789     0.03286      0.552751     0.00587051     0.087195        100    
reco:energyrecnue:EnergyReco                          0.000142468   0.00234557     0.0351433    0.00102418    0.00434731       100    
reco:energyrecnc:EnergyReco                            0.0018552    0.00489355     0.0409826    0.00353686    0.00466659       100    
reco:energyrecnumurange:EnergyReco                    0.000261491    0.0299439     0.581021     0.00325794     0.0870964       100    
reco:energyrecnumumcs:EnergyReco                      0.00189542     0.0310735     0.514691     0.00511671     0.0829019       100    
reco:opdec:Deconvolution                               0.108498      0.415082      0.785252      0.379219      0.149409        100    
reco:ophitspe:OpHitFinderDeco                          0.234386       2.1266        4.19059       2.13429      0.806374        100    
reco:opflash:OpFlashFinder                            0.000193103   0.00180056    0.00583735    0.00167512    0.000767601      100    
reco:rns:RandomNumberSaver                            2.3525e-05    4.90622e-05   0.000395654   4.4884e-05    3.58971e-05      100    
reco:anglereconue:NuAngularReco                       0.00020742    0.00184415     0.0296936    0.000615593   0.00356998       100    
reco:anglereconumu:NuAngularReco                      9.5039e-05    0.00147911     0.0299881    0.000287801   0.00363109       100    
reco:anglereconuepfps:NuAngularReco                   0.000153569    0.0016413     0.0296083    0.000460274    0.0035072       100    
reco:anglereconumupfps:NuAngularReco                  0.000120015    0.0015407     0.0296882    0.000376537   0.00349369       100    
[art]:TriggerResults:TriggerResultInserter             9.258e-06    1.37997e-05    5.809e-05    1.2764e-05    5.51869e-06      100    
end_path:out1:RootOutput                               3.677e-06    7.67863e-06   2.3794e-05     7.384e-06    2.55533e-06      100    
end_path:out1:RootOutput(write)                        0.013512      0.297448       1.30458      0.301942      0.190794        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 12952.5 MB
  Peak resident set size usage (VmHWM): 2435.09 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66004350_1186_20231202T161111Z_gen_g4_detsim_hitreco__20240508T035137Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66004350_1186_20231202T161111Z_gen_g4_detsim_hitreco__20240508T035137Z_reco2_reco_2024-11-23T_015847Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66004350_1186_20231202T161111Z_gen_g4_detsim_hitreco__20240508T035137Z_reco2_reco_data_2024-11-23T_015847Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work