Jobsub ID 108724.18@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108724.18@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 03:47:42
Site	UK_RAL-PPD
Entry	CMSHTPC_T2_UK_SGrid_RALPP_hep207
Last heartbeat	2024-11-23 05:16:47
From worker node	Hostname	heplnc147.pp.rl.ac.uk
	cpuinfo	AMD EPYC 7452 32-Core Processor
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 03:53:09
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_892_20231203T123937Z_gen_g4_detsim_hitreco__20240509T201458Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (4998s)
	CPU time	1h (4903s = 98%)
Outputting started	2024-11-23 05:16:27
Output files
Finished	2024-11-23 05:16:47
List job events Wrapper job log
Jobscript log (last 10,000 characters)

IDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:16:16 GMT run: 6411871 subRun: 1 event: 89299
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 05:16:16 GMT run: 6411871 subRun: 1 event: 89299
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 891, 674, 574
minwire 0: 758
minwire 1: 1194
minwire 2: 741
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.602004, 0.20035, 0.197646, 
Output 1: 0.126505, 0.381115, 0.292695, 0.199685, 
Output 2: 0.027944, 0.972056, 

607
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2158!
Begin processing the 100th record. run: 6411871 subRun: 1 event: 89300 at 23-Nov-2024 05:16:20 GMT
> Running Algorithm: Alg0001, LArPreProcessing
> Running Algorithm: Alg0002, LArDLMaster
> Running Algorithm: Alg0003, LArNeutrinoEventValidation
---RAW-MATCHING-OUTPUT--------------------------------------------------------------------------
Failure in algorithm Alg0003, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 05:16:23 GMT run: 6411871 subRun: 1 event: 89300
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 05:16:23 GMT run: 6411871 subRun: 1 event: 89300
Comparing two wires in the same plane: return failure
%MSG
140
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 262!
23-Nov-2024 05:16:25 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_892_20231203T123937Z_gen_g4_detsim_hitreco__20240509T201458Z_reco2_reco_data_2024-11-23T_035316Z.root"
23-Nov-2024 05:16:25 GMT  Closed input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/be/ec/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_892_20231203T123937Z_gen_g4_detsim_hitreco__20240509T201458Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              0.88703       47.7213       583.632       13.8389       94.7449        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00508591     0.0290659     0.379382      0.0201418     0.0443888       100    
reco:linecluster:LineCluster                           0.0421779     0.498261       17.9715      0.0759282      1.86659        100    
reco:trajcluster:TrajCluster                           0.0115008      1.13872       31.814       0.0321249      4.12438        100    
reco:pandora:StandardPandora                           0.029737       3.78402       47.4875       1.85701       7.67255        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000174971    0.0119093     0.152626     0.00234587     0.0245267       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.00025952     0.020039       0.33639     0.00493336     0.0455884       100    
reco:pandoracalo:Calorimetry                          0.000135806    0.030937      0.722165      0.0015335      0.10299        100    
reco:pandorapid:Chi2ParticleID                        4.9163e-05    0.000292995   0.00501759    0.000178231   0.000514106      100    
reco:trkshowersplit:TrackShowerHits                   0.000145334    0.0377541     0.831306     0.00127792      0.11847        100    
reco:pmtrack:PMAlgTrackMaker                           0.0021284      4.04082       97.6709      0.305636       12.0022        100    
reco:pmtrackcalo:Calorimetry                          9.6983e-05     0.0202229     0.701401     0.00143791     0.0779287       100    
reco:pmtrackpid:Chi2ParticleID                        3.9415e-05    0.000519759   0.00589348    0.00020641    0.000873523      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000244863     3.1419        50.0183      0.359082       8.02423        100    
reco:pmtrajfitcalo:Calorimetry                        9.2815e-05     0.0318951      0.73816     0.00197591     0.107645        100    
reco:pmtrajfitpid:Chi2ParticleID                      4.0225e-05    0.000309253   0.00131059    0.000232374   0.000274989      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00167507      4.52741       93.4101      0.315929       12.7386        100    
reco:pmtracktccalo:Calorimetry                        9.9567e-05     0.0126176     0.235802     0.00147103     0.0334199       100    
reco:pmtracktcpid:Chi2ParticleID                      3.7882e-05    0.000458769   0.00257778    0.000223917   0.000571902      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000225416     2.58217       46.0272      0.508226       6.39383        100    
reco:pmtrajfittccalo:Calorimetry                      8.8226e-05     0.0251522     0.590697     0.00210361     0.0792338       100    
reco:pmtrajfittcpid:Chi2ParticleID                     3.758e-05    0.000635224   0.00869992    0.000269364   0.00108082       100    
reco:blurredcluster:BlurredClustering                 0.000176422     1.56082       9.18757       0.75609       2.05757        100    
reco:emtrkmichelid:EmTrackMichelId                     0.0390641      22.0979       230.48        5.60168       41.3666        100    
reco:emshower:EMShower                                0.000359168    0.0453305     0.392988     0.00936085     0.0769014       100    
reco:cvnmap:CVNMapper                                 2.9656e-05     0.0167311     0.078327      0.0169672     0.0134589       100    
reco:cvneva:CVNEvaluator                              2.1961e-05      1.12394       2.11279       1.37017       0.62428        100    
reco:energyrecnumu:EnergyReco                         0.000510524    0.106013       3.95207      0.0089345     0.436828        100    
reco:energyrecnue:EnergyReco                          0.000183115   0.00400344     0.0702283    0.00101093    0.00934021       100    
reco:energyrecnc:EnergyReco                           0.00207401    0.00677599     0.0694678    0.00388746    0.00923014       100    
reco:energyrecnumurange:EnergyReco                    0.000262756    0.0996889      3.63023     0.00381473     0.409128        100    
reco:energyrecnumumcs:EnergyReco                      0.00102197     0.103853       3.74153     0.00545514     0.421219        100    
reco:opdec:Deconvolution                               0.0384377     0.252381      0.755061      0.219084      0.127687        100    
reco:ophitspe:OpHitFinderDeco                          0.294628       2.02567       4.20896       1.97765      0.844348        100    
reco:opflash:OpFlashFinder                            0.000471901   0.00219882    0.00999412    0.00201131    0.00120005       100    
reco:rns:RandomNumberSaver                            2.0989e-05    5.72197e-05   0.000615793   4.45595e-05   6.35874e-05      100    
reco:anglereconue:NuAngularReco                       0.000205357   0.00276287     0.0380739    0.000610141   0.00622552       100    
reco:anglereconumu:NuAngularReco                      9.6863e-05    0.00242038     0.0361632    0.000288174   0.00613013       100    
reco:anglereconuepfps:NuAngularReco                   0.000162787   0.00265561     0.0382164    0.000502889    0.0062779       100    
reco:anglereconumupfps:NuAngularReco                  0.000136317   0.00256371     0.0371161    0.000395701   0.00622459       100    
[art]:TriggerResults:TriggerResultInserter             1.081e-05    1.91379e-05   0.000143621   1.6125e-05    1.45926e-05      100    
end_path:out1:RootOutput                               3.878e-06    1.05366e-05    5.317e-05    8.7765e-06    8.55938e-06      100    
end_path:out1:RootOutput(write)                        0.0255181     0.327073       1.40291      0.298211      0.235532        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21665.8 MB
  Peak resident set size usage (VmHWM): 2249.83 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_892_20231203T123937Z_gen_g4_detsim_hitreco__20240509T201458Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_892_20231203T123937Z_gen_g4_detsim_hitreco__20240509T201458Z_reco2_reco_2024-11-23T_035316Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6411871_892_20231203T123937Z_gen_g4_detsim_hitreco__20240509T201458Z_reco2_reco_data_2024-11-23T_035316Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work