Jobsub ID 108744.164@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108744.164@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:38:45
Site	CERN
Entry	CMSHTPC_T2_CH_CERN_ce512
Last heartbeat	2024-11-23 05:18:34
From worker node	Hostname	b9p05p9589.cern.ch
	cpuinfo	AMD EPYC 7543 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	171000 (47 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:44:25
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6410210_32_20231203T071004Z_gen_g4_detsim_hitreco__20240508T054348Z_reco2.root
Jobscript	Exit code	0
	Real time	33m (2031s)
	CPU time	38m (2300s = 113%)
Outputting started	2024-11-23 05:18:17
Output files
Finished	2024-11-23 05:18:34
List job events Wrapper job log
Jobscript log (last 10,000 characters)

e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:18:06 CET run: 6410210 subRun: 1 event: 3299
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:18:06 CET run: 6410210 subRun: 1 event: 3299
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:18:06 CET run: 6410210 subRun: 1 event: 3299
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:18:06 CET run: 6410210 subRun: 1 event: 3299
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 206, 185, 167
minwire 0: 1593
minwire 1: 949
minwire 2: 1320
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.516118, 0.420069, 0.0638128, 
Output 1: 0.142856, 0.585669, 0.235252, 0.0362238, 
Output 2: 0.569194, 0.430806, 

707
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2711!
Begin processing the 100th record. run: 6410210 subRun: 1 event: 3300 at 23-Nov-2024 06:18:10 CET
> Running Algorithm: Alg0001, LArPreProcessing
> Running Algorithm: Alg0002, LArDLMaster
> Running Algorithm: Alg0003, LArNeutrinoEventValidation
---RAW-MATCHING-OUTPUT--------------------------------------------------------------------------
Failure in algorithm Alg0003, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
287
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 762!
23-Nov-2024 06:18:15 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6410210_32_20231203T071004Z_gen_g4_detsim_hitreco__20240508T054348Z_reco2_reco_data_2024-11-23T_044430Z.root"
23-Nov-2024 06:18:15 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/8f/02/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6410210_32_20231203T071004Z_gen_g4_detsim_hitreco__20240508T054348Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.145348       19.3832       243.338       8.68741       32.5925        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00337951     0.0156102     0.061365      0.0157692     0.0088871       100    
reco:linecluster:LineCluster                           0.0167447     0.0986906      1.79382      0.044571       0.24058        100    
reco:trajcluster:TrajCluster                          0.00854048     0.285885       8.30423      0.0203384      1.06312        100    
reco:pandora:StandardPandora                           0.0161415      1.52297       16.2556       1.22552       1.63051        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000116092   0.00449085     0.106723     0.00160258     0.0113896       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000177184    0.0104537     0.160291     0.00354609     0.0235994       100    
reco:pandoracalo:Calorimetry                          9.2022e-05     0.0116459     0.433472     0.00108863     0.0462638       100    
reco:pandorapid:Chi2ParticleID                        4.1801e-05    0.000163232   0.000837648   0.000138373   0.000108965      100    
reco:trkshowersplit:TrackShowerHits                   7.2391e-05     0.0102792     0.283765     0.000839743    0.033899        100    
reco:pmtrack:PMAlgTrackMaker                          0.000970852     1.46579       33.206       0.190403       4.11115        100    
reco:pmtrackcalo:Calorimetry                          6.1071e-05    0.00685284     0.174243     0.000922776    0.0216071       100    
reco:pmtrackpid:Chi2ParticleID                         2.398e-05    0.000242867   0.00227978    0.000135333   0.000328881      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000154994     1.26469       25.4182      0.297421       3.07271        100    
reco:pmtrajfitcalo:Calorimetry                        5.5122e-05    0.00990899     0.254104     0.00137743     0.0302231       100    
reco:pmtrajfitpid:Chi2ParticleID                       2.303e-05    0.000183846   0.00088835    0.000151038   0.000143134      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00114846      1.80874       34.5678      0.205651       5.05339        100    
reco:pmtracktccalo:Calorimetry                        7.8912e-05    0.00487461     0.083872     0.000964991    0.0120757       100    
reco:pmtracktcpid:Chi2ParticleID                      3.2521e-05    0.000230233   0.00153358    0.000159828   0.00024439       100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000186684     1.23395       16.7006      0.287349       2.62772        100    
reco:pmtrajfittccalo:Calorimetry                      7.4851e-05    0.00835394     0.210596     0.00120588     0.0248099       100    
reco:pmtrajfittcpid:Chi2ParticleID                     3.287e-05    0.000296065   0.00207243    0.000164629   0.000363378      100    
reco:blurredcluster:BlurredClustering                 9.8402e-05     0.998696       7.4046       0.539516       1.26787        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000140654     7.51702       92.7818       2.55094       13.4884        100    
reco:emshower:EMShower                                0.000175943    0.0278086      0.22875     0.00822697     0.0424692       100    
reco:cvnmap:CVNMapper                                  2.227e-05    0.00931091     0.0320407    0.00688629     0.0081657       100    
reco:cvneva:CVNEvaluator                               1.69e-05      0.676179       1.13972       0.83715       0.41356        100    
reco:energyrecnumu:EnergyReco                         0.000289096    0.0251957     0.645951      0.0039748     0.0859253       100    
reco:energyrecnue:EnergyReco                          0.000116362   0.00158032     0.0221049    0.000513287   0.00287182       100    
reco:energyrecnc:EnergyReco                           0.00173419    0.00396361     0.0224207    0.00326483    0.00288582       100    
reco:energyrecnumurange:EnergyReco                    0.000173204    0.0221475     0.525802     0.00296055     0.0775927       100    
reco:energyrecnumumcs:EnergyReco                      0.000747287    0.0235157     0.489356     0.00382211     0.0757827       100    
reco:opdec:Deconvolution                               0.0148529     0.183362      0.336722      0.183287      0.0683376       100    
reco:ophitspe:OpHitFinderDeco                          0.0319248      1.89918       4.02181       1.82757      0.865634        100    
reco:opflash:OpFlashFinder                            0.000133673   0.00137394    0.00416112    0.00127211    0.000639024      100    
reco:rns:RandomNumberSaver                            1.7451e-05    3.80456e-05   0.000362628   3.4496e-05    3.39616e-05      100    
reco:anglereconue:NuAngularReco                       0.000145383   0.00105763     0.0167371    0.000379834   0.00206276       100    
reco:anglereconumu:NuAngularReco                      6.1472e-05    0.000853958    0.0170474    0.000200859   0.00205015       100    
reco:anglereconuepfps:NuAngularReco                   0.000102452   0.000998027    0.0161585    0.000380609   0.00199047       100    
reco:anglereconumupfps:NuAngularReco                  8.3392e-05    0.000910459    0.0162672    0.000279756   0.00198133       100    
[art]:TriggerResults:TriggerResultInserter             8.67e-06     1.31058e-05   4.2561e-05    1.24205e-05   4.74138e-06      100    
end_path:out1:RootOutput                               3.05e-06     6.46824e-06   2.4681e-05    6.3605e-06    3.2246e-06       100    
end_path:out1:RootOutput(write)                        0.0127847     0.223729      0.696887      0.228473      0.138017        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21719.3 MB
  Peak resident set size usage (VmHWM): 2285.65 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6410210_32_20231203T071004Z_gen_g4_detsim_hitreco__20240508T054348Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6410210_32_20231203T071004Z_gen_g4_detsim_hitreco__20240508T054348Z_reco2_reco_2024-11-23T_044430Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6410210_32_20231203T071004Z_gen_g4_detsim_hitreco__20240508T054348Z_reco2_reco_data_2024-11-23T_044430Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work