Jobsub ID 301618.164@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301618.164@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 01:19:16
Site	UK_Lancaster
Entry	UBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat	2024-11-23 02:15:22
From worker node	Hostname	comp21-00
	cpuinfo	Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 01:23:44
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50640579_163_20231207T161420Z_gen_g4_detsim_hitreco__20240510T051118Z_reco2.root
Jobscript	Exit code	0
	Real time	51m (3083s)
	CPU time	45m (2757s = 89%)
Outputting started	2024-11-23 02:15:07
Output files
Finished	2024-11-23 02:15:22
List job events Wrapper job log
Jobscript log (last 10,000 characters)

ax tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.366895, 0.150907, 0.482198, 
Output 1: 0.175463, 0.749367, 0.0731487, 0.00202133, 
Output 2: 0.629757, 0.370243, 

390
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1009!
Begin processing the 100th record. run: 50640579 subRun: 1 event: 16400 at 23-Nov-2024 02:15:01 GMT
> Running Algorithm: Alg0001, LArPreProcessing
> Running Algorithm: Alg0002, LArDLMaster
> Running Algorithm: Alg0003, LArNeutrinoEventValidation
---RAW-MATCHING-OUTPUT--------------------------------------------------------------------------
Failure in algorithm Alg0003, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  23-Nov-2024 02:15:03 GMT run: 50640579 subRun: 1 event: 16400
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 02:15:03 GMT run: 50640579 subRun: 1 event: 16400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 02:15:03 GMT run: 50640579 subRun: 1 event: 16400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 02:15:03 GMT run: 50640579 subRun: 1 event: 16400
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 02:15:03 GMT run: 50640579 subRun: 1 event: 16400
Comparing two wires in the same plane: return failure
%MSG
334
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 802!
23-Nov-2024 02:15:06 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50640579_163_20231207T161420Z_gen_g4_detsim_hitreco__20240510T051118Z_reco2_reco_data_2024-11-23T_012349Z.root"
23-Nov-2024 02:15:06 GMT  Closed input file "root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/95/e8/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50640579_163_20231207T161420Z_gen_g4_detsim_hitreco__20240510T051118Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.26841       27.8891       454.974       8.42251       65.1863        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0111171     0.105075      0.314272      0.0808057     0.0734791       100    
reco:linecluster:LineCluster                           0.150164      0.633484       14.3504      0.296509       1.83881        100    
reco:trajcluster:TrajCluster                           0.0344665      1.29753       46.5033      0.106131       6.04373        100    
reco:pandora:StandardPandora                           0.954855       1.96565       32.9725       1.1259        4.16999        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000129707    0.0067942     0.158488     0.00102748     0.0214417       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000164113   0.00914952     0.230769     0.00207979     0.0257503       100    
reco:pandoracalo:Calorimetry                          9.7603e-05     0.018593      0.579478     0.000638201    0.077466        100    
reco:pandorapid:Chi2ParticleID                        3.5555e-05    0.00013979    0.00114559    0.000102242    0.0001508       100    
reco:trkshowersplit:TrackShowerHits                   9.1709e-05     0.0225084     0.735201     0.000476359    0.101901        100    
reco:pmtrack:PMAlgTrackMaker                          0.00103407      1.99951       55.8934      0.0884115      7.08785        100    
reco:pmtrackcalo:Calorimetry                          7.2174e-05     0.0114047     0.294573     0.000660458    0.0443302       100    
reco:pmtrackpid:Chi2ParticleID                        2.7366e-05    0.000236093   0.00265991    0.000110258   0.000393482      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000171556      1.853        61.5438       0.12455       7.14059        100    
reco:pmtrajfitcalo:Calorimetry                        5.7998e-05     0.0193948     0.640409     0.000776703    0.079805        100    
reco:pmtrajfitpid:Chi2ParticleID                       2.455e-05    0.00015398    0.000829486   0.000111889   0.000148054      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000936496     2.42617       51.1962      0.128529       7.87004        100    
reco:pmtracktccalo:Calorimetry                        6.4289e-05     0.0076232     0.200399     0.000740728    0.0280312       100    
reco:pmtracktcpid:Chi2ParticleID                      2.7203e-05    0.000231759    0.0021412    0.000114026   0.000357869      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000149911     1.39606       29.126       0.198163       4.11096        100    
reco:pmtrajfittccalo:Calorimetry                      5.9151e-05     0.0191308     0.564524     0.000778251    0.0791391       100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.4345e-05    0.000320183   0.00521728    0.000118965   0.000660526      100    
reco:blurredcluster:BlurredClustering                  0.0144347     0.914703       7.63123      0.366549       1.39436        100    
reco:emtrkmichelid:EmTrackMichelId                      0.24482       11.9668       153.934       2.55776       26.1122        100    
reco:emshower:EMShower                                0.000226362    0.0232169     0.325403     0.00555559     0.0471807       100    
reco:cvnmap:CVNMapper                                 1.9343e-05     0.0137909     0.0450424     0.0132314     0.0108086       100    
reco:cvneva:CVNEvaluator                              1.4366e-05     0.511216       1.48151      0.686631       0.33667        100    
reco:energyrecnumu:EnergyReco                         0.000266201    0.0413542      0.67228     0.00386862     0.114742        100    
reco:energyrecnue:EnergyReco                          0.000123386   0.00225307     0.0459317    0.000432306    0.0068749       100    
reco:energyrecnc:EnergyReco                           0.00167584    0.00386662     0.048997     0.00203366    0.00712528       100    
reco:energyrecnumurange:EnergyReco                    0.000173421    0.0390648     0.672237      0.001835      0.113881        100    
reco:energyrecnumumcs:EnergyReco                      0.000649682    0.0400479      0.67469     0.00234366     0.113558        100    
reco:opdec:Deconvolution                               0.114824      0.371433      0.825052      0.364536      0.144636        100    
reco:ophitspe:OpHitFinderDeco                          0.446806       1.96494       4.16983       1.79681      0.829917        100    
reco:opflash:OpFlashFinder                            0.000305762    0.0012076    0.00401007    0.00098514    0.00064885       100    
reco:rns:RandomNumberSaver                            1.7705e-05    4.3587e-05    0.000544683   3.2374e-05    7.21673e-05      100    
reco:anglereconue:NuAngularReco                       0.000137261   0.00182875     0.0418916    0.000282386   0.00601853       100    
reco:anglereconumu:NuAngularReco                      6.3439e-05    0.00161035     0.0408009    0.000130796   0.00593958       100    
reco:anglereconuepfps:NuAngularReco                   0.000105104   0.00172426     0.0412885    0.000257637   0.00600328       100    
reco:anglereconumupfps:NuAngularReco                  8.7356e-05    0.00165904     0.0410902    0.00019877    0.00592904       100    
[art]:TriggerResults:TriggerResultInserter             7.949e-06    1.10368e-05   4.0768e-05    1.01885e-05   3.82409e-06      100    
end_path:out1:RootOutput                               2.157e-06    5.25119e-06   1.7404e-05    5.5095e-06    2.77391e-06      100    
end_path:out1:RootOutput(write)                        0.0171086     0.194572      0.880919      0.185609      0.150101        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9602.08 MB
  Peak resident set size usage (VmHWM): 2453.52 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50640579_163_20231207T161420Z_gen_g4_detsim_hitreco__20240510T051118Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50640579_163_20231207T161420Z_gen_g4_detsim_hitreco__20240510T051118Z_reco2_reco_2024-11-23T_012349Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50640579_163_20231207T161420Z_gen_g4_detsim_hitreco__20240510T051118Z_reco2_reco_data_2024-11-23T_012349Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work