Jobsub ID 108774.15@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108774.15@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:54:30
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce512
Last heartbeat	2024-11-23 05:45:30
From worker node	Hostname	b9p08p3442.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:56:02
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6432497_16_20231205T113231Z_gen_g4_detsim_hitreco__20240510T034142Z_reco2.root
Jobscript	Exit code	0
	Real time	49m (2950s)
	CPU time	50m (3029s = 102%)
Outputting started	2024-11-23 05:45:12
Output files
Finished	2024-11-23 05:45:30
List job events Wrapper job log
Jobscript log (last 10,000 characters)

-Nov-2024 06:45:05 CET run: 6432497 subRun: 1 event: 1700
1st wire C:0 T:6 P:2 W:677 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:45:05 CET run: 6432497 subRun: 1 event: 1700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:45:05 CET run: 6432497 subRun: 1 event: 1700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:45:05 CET run: 6432497 subRun: 1 event: 1700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:45:06 CET run: 6432497 subRun: 1 event: 1700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:45:06 CET run: 6432497 subRun: 1 event: 1700
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:45:06 CET run: 6432497 subRun: 1 event: 1700
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 84, 60, 49
minwire 0: 947
minwire 1: 2296
minwire 2: 640
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.081202, 0.512728, 0.40607, 
Output 1: 0.584848, 0.405113, 0.00944779, 0.000590616, 
Output 2: 0.933586, 0.0664142, 

395
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1183!
23-Nov-2024 06:45:11 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6432497_16_20231205T113231Z_gen_g4_detsim_hitreco__20240510T034142Z_reco2_reco_data_2024-11-23T_045609Z.root"
23-Nov-2024 06:45:11 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/f7/e0/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6432497_16_20231205T113231Z_gen_g4_detsim_hitreco__20240510T034142Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.552684       27.5136       482.485       9.20906       72.989         100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0165569     0.0396889     0.0964383     0.0391985     0.0120696       100    
reco:linecluster:LineCluster                           0.0957817     0.510366       16.8422       0.16681       2.06027        100    
reco:trajcluster:TrajCluster                           0.0478921     0.923856       28.1353      0.0583561      4.10848        100    
reco:pandora:StandardPandora                           0.0600219      2.38513       47.0878       1.30232       5.79299        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000125101   0.00658814     0.195378     0.00139506     0.0227194       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000201821    0.0216282      1.1249      0.00285384     0.113138        100    
reco:pandoracalo:Calorimetry                          0.000111181    0.023632      0.801698     0.00110561     0.103411        100    
reco:pandorapid:Chi2ParticleID                        4.4881e-05    0.000181543   0.00125263    0.000134151   0.000188493      100    
reco:trkshowersplit:TrackShowerHits                    9.89e-05      0.0345194     0.908411     0.000715049    0.147254        100    
reco:pmtrack:PMAlgTrackMaker                          0.00109395      2.4999        66.8872      0.186599       9.57155        100    
reco:pmtrackcalo:Calorimetry                           6.997e-05     0.0116462      0.41861     0.000853845    0.0509047       100    
reco:pmtrackpid:Chi2ParticleID                         2.887e-05    0.000259094   0.00242576    0.000147346   0.000392885      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000181491     2.17278       59.8983      0.235773       7.82486        100    
reco:pmtrajfitcalo:Calorimetry                         6.978e-05     0.0212292     0.552377     0.00137613     0.082111        100    
reco:pmtrajfitpid:Chi2ParticleID                       2.56e-05     0.000195296   0.00101637    0.000145875   0.000170803      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000915216     3.33279       69.7699      0.227615       12.2214        100    
reco:pmtracktccalo:Calorimetry                        7.7471e-05    0.00818793     0.393962     0.000903436    0.0406165       100    
reco:pmtracktcpid:Chi2ParticleID                       2.72e-05     0.000251273   0.00296515    0.000154026   0.000373547      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000193411     1.56543       32.401       0.252385       5.26631        100    
reco:pmtrajfittccalo:Calorimetry                      7.0001e-05     0.0204564     0.644735     0.000936355    0.0916398       100    
reco:pmtrajfittcpid:Chi2ParticleID                     2.554e-05    0.00035883    0.00534961    0.000158941   0.000740045      100    
reco:blurredcluster:BlurredClustering                 0.000161011     1.02429       9.62844      0.507089       1.66529        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000150961     9.38737       154.957       2.60368       24.0885        100    
reco:emshower:EMShower                                0.000225212    0.0301235     0.214286     0.00616306     0.0445226       100    
reco:cvnmap:CVNMapper                                  2.741e-05     0.0126044     0.0514584     0.0145383     0.0100737       100    
reco:cvneva:CVNEvaluator                               1.719e-05     0.721089       1.99064      0.906126      0.470799        100    
reco:energyrecnumu:EnergyReco                         0.000276021    0.0679712      3.74269     0.00394704     0.390617        100    
reco:energyrecnue:EnergyReco                          0.000114361    0.0023633     0.0685058    0.000572828   0.00776258       100    
reco:energyrecnc:EnergyReco                           0.00173383    0.00503922     0.0720708    0.00342637    0.00809648       100    
reco:energyrecnumurange:EnergyReco                    0.00013469     0.076713       4.98732     0.00246125     0.507586        100    
reco:energyrecnumumcs:EnergyReco                      0.000812894    0.0791727      5.07574     0.00365654      0.51576        100    
reco:opdec:Deconvolution                               0.0543916      0.27679      0.457675      0.283912      0.0858338       100    
reco:ophitspe:OpHitFinderDeco                          0.146515       2.00343       4.05695       2.07042      0.836195        100    
reco:opflash:OpFlashFinder                            0.000168231   0.00149537    0.00407096    0.00146876    0.000664095      100    
reco:rns:RandomNumberSaver                            1.7771e-05    4.01569e-05   0.000286172   3.76455e-05   2.63851e-05      100    
reco:anglereconue:NuAngularReco                       0.000158461   0.00143352     0.0350647    0.000466988   0.00423094       100    
reco:anglereconumu:NuAngularReco                       6.298e-05    0.00116408     0.0340381    0.000217466   0.00408158       100    
reco:anglereconuepfps:NuAngularReco                   9.7641e-05     0.0013432     0.0340219    0.000391303   0.00409101       100    
reco:anglereconumupfps:NuAngularReco                   8.503e-05    0.00125129     0.0340073    0.000298437    0.0041097       100    
[art]:TriggerResults:TriggerResultInserter              8.4e-06     1.40856e-05   3.7921e-05    1.3535e-05    5.40213e-06      100    
end_path:out1:RootOutput                                2.6e-06     6.68917e-06   2.3171e-05     6.59e-06     2.47847e-06      100    
end_path:out1:RootOutput(write)                        0.0119725     0.238672      0.896082      0.245627      0.165875        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21818.4 MB
  Peak resident set size usage (VmHWM): 2390.56 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6432497_16_20231205T113231Z_gen_g4_detsim_hitreco__20240510T034142Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6432497_16_20231205T113231Z_gen_g4_detsim_hitreco__20240510T034142Z_reco2_reco_2024-11-23T_045609Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6432497_16_20231205T113231Z_gen_g4_detsim_hitreco__20240510T034142Z_reco2_reco_data_2024-11-23T_045609Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work