Jobsub ID 108742.59@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108742.59@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:30:09
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce512
Last heartbeat	2024-11-23 05:11:10
From worker node	Hostname	b9p10p6463.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:40:59
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_678_20231203T051952Z_gen_g4_detsim_hitreco__20240508T050851Z_reco2.root
Jobscript	Exit code	0
	Real time	29m (1794s)
	CPU time	35m (2151s = 119%)
Outputting started	2024-11-23 05:10:54
Output files
Finished	2024-11-23 05:11:10
List job events Wrapper job log
Jobscript log (last 10,000 characters)

ginModule  23-Nov-2024 06:10:45 CET run: 6407968 subRun: 1 event: 67899
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:10:45 CET run: 6407968 subRun: 1 event: 67899
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:10:45 CET run: 6407968 subRun: 1 event: 67899
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:10:48 CET run: 6407968 subRun: 1 event: 67899
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 163, 199, 175
minwire 0: 484
minwire 1: 2043
minwire 2: 335
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.936473, 0.0575421, 0.00598439, 
Output 1: 0.957157, 0.0411573, 0.00141244, 0.000272798, 
Output 2: 0.990888, 0.00911203, 

370
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 986!
Begin processing the 100th record. run: 6407968 subRun: 1 event: 67900 at 23-Nov-2024 06:10:51 CET
> Running Algorithm: Alg0001, LArPreProcessing
> Running Algorithm: Alg0002, LArDLMaster
> Running Algorithm: Alg0003, LArNeutrinoEventValidation
---RAW-MATCHING-OUTPUT--------------------------------------------------------------------------
Failure in algorithm Alg0003, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
105
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 142!
23-Nov-2024 06:10:54 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_678_20231203T051952Z_gen_g4_detsim_hitreco__20240508T050851Z_reco2_reco_data_2024-11-23T_044104Z.root"
23-Nov-2024 06:10:54 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/dd/e6/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_678_20231203T051952Z_gen_g4_detsim_hitreco__20240508T050851Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.529535       16.0405       335.675       8.01644       36.0901        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0170067     0.039588      0.0679345     0.0385302     0.010431        100    
reco:linecluster:LineCluster                           0.0948537     0.219369       3.73221      0.176073      0.355379        100    
reco:trajcluster:TrajCluster                           0.0478574      0.42318       27.8389      0.0586943      2.77048        100    
reco:pandora:StandardPandora                           0.0550672      1.50988       24.9538       1.05439       2.5272         100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000123401   0.00336069     0.0697723    0.00127616    0.00783519       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000168121   0.00702215     0.0690906    0.00241872     0.0134218       100    
reco:pandoracalo:Calorimetry                          9.5231e-05     0.0113576     0.617902     0.00091604     0.0641672       100    
reco:pandorapid:Chi2ParticleID                         3.615e-05    0.000141613   0.000899246   0.000108696   0.000116266      100    
reco:trkshowersplit:TrackShowerHits                   5.7291e-05    0.00943368      0.50067     0.000660014    0.0506219       100    
reco:pmtrack:PMAlgTrackMaker                          0.000975666     1.18744       47.5538      0.195119       4.89702        100    
reco:pmtrackcalo:Calorimetry                           5.871e-05    0.00623384     0.275043     0.000839805    0.0302689       100    
reco:pmtrackpid:Chi2ParticleID                         2.313e-05    0.000210781   0.00249298    0.00013368    0.000318816      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000170572    0.871214       18.6734      0.244042       2.17322        100    
reco:pmtrajfitcalo:Calorimetry                        5.8341e-05     0.0103678     0.470607     0.00117144     0.0499972       100    
reco:pmtrajfitpid:Chi2ParticleID                       2.361e-05    0.000150078   0.000647064   0.000134031   0.000102224      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000812785     1.51916       63.8583      0.217406       6.67128        100    
reco:pmtracktccalo:Calorimetry                        6.0601e-05    0.00392442     0.154194     0.000793945    0.0164432       100    
reco:pmtracktcpid:Chi2ParticleID                       2.408e-05    0.000181961   0.00194316    0.000133646   0.000222682      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000143511    0.793778       19.5232      0.249122       2.2011         100    
reco:pmtrajfittccalo:Calorimetry                       5.704e-05    0.00700425     0.354427     0.000965971    0.0360183       100    
reco:pmtrajfittcpid:Chi2ParticleID                     2.289e-05    0.000231838   0.00279828    0.000154791   0.000326933      100    
reco:blurredcluster:BlurredClustering                 0.000138841    0.710536       4.67359      0.311898      0.926566        100    
reco:emtrkmichelid:EmTrackMichelId                     9.407e-05      5.9673        111.695       2.25848       13.4375        100    
reco:emshower:EMShower                                0.000135881    0.0225878     0.206897     0.00515917     0.0386012       100    
reco:cvnmap:CVNMapper                                  1.715e-05    0.00805407     0.037508     0.00658436    0.00716135       100    
reco:cvneva:CVNEvaluator                               1.452e-05     0.601485       1.08976       0.76012      0.364793        100    
reco:energyrecnumu:EnergyReco                         0.000319982    0.0392313      2.69827     0.00332559     0.268892        100    
reco:energyrecnue:EnergyReco                          0.00012195    0.00141597     0.0344232    0.000408738   0.00368375       100    
reco:energyrecnc:EnergyReco                           0.00142249    0.00321628     0.0397269    0.00221591     0.0041426       100    
reco:energyrecnumurange:EnergyReco                    0.00012546     0.0331479      2.2708       0.0017836     0.226753        100    
reco:energyrecnumumcs:EnergyReco                      0.000844275    0.0296033      1.78838     0.00279618     0.179141        100    
reco:opdec:Deconvolution                               0.0492441     0.263422      0.609559      0.259882      0.0831074       100    
reco:ophitspe:OpHitFinderDeco                          0.146438       1.54114       3.87575       1.45474      0.725318        100    
reco:opflash:OpFlashFinder                            0.000139751   0.00119624    0.00432417    0.00112593    0.00058696       100    
reco:rns:RandomNumberSaver                             1.828e-05    3.80808e-05   0.000309442   3.49405e-05   2.99898e-05      100    
reco:anglereconue:NuAngularReco                       0.000165141   0.000859359    0.0177894    0.000362303   0.00195473       100    
reco:anglereconumu:NuAngularReco                       6.935e-05    0.000681704    0.0176446    0.000209587   0.00192565       100    
reco:anglereconuepfps:NuAngularReco                   0.000116801   0.000818833    0.0173315    0.000335462   0.00189639       100    
reco:anglereconumupfps:NuAngularReco                  8.7931e-05    0.000728447    0.0170085    0.000270512    0.0018658       100    
[art]:TriggerResults:TriggerResultInserter             7.99e-06     1.17995e-05    4.02e-05      1.042e-05    4.39925e-06      100    
end_path:out1:RootOutput                               3.42e-06     7.23776e-06   1.8171e-05    7.2855e-06    2.84057e-06      100    
end_path:out1:RootOutput(write)                       0.00988947     0.190031      0.860607      0.193674      0.119964        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21549.5 MB
  Peak resident set size usage (VmHWM): 2115.71 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_678_20231203T051952Z_gen_g4_detsim_hitreco__20240508T050851Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_678_20231203T051952Z_gen_g4_detsim_hitreco__20240508T050851Z_reco2_reco_2024-11-23T_044104Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6407968_678_20231203T051952Z_gen_g4_detsim_hitreco__20240508T050851Z_reco2_reco_data_2024-11-23T_044104Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work