Jobsub ID 301744.40@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301744.40@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 04:56:37
Site	UK_Lancaster
Entry	UBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat	2024-11-23 07:22:41
From worker node	Hostname	comp23-20
	cpuinfo	Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 04:58:23
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207014_37_20231120T132529Z_gen_g4_detsim_hitreco__20240503T062229Z_reco2.root
Jobscript	Exit code	0
	Real time	2h (8641s)
	CPU time	2h (7709s = 89%)
Outputting started	2024-11-23 07:22:25
Output files
Finished	2024-11-23 07:22:41
List job events Wrapper job log
Jobscript log (last 10,000 characters)

on failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 5.73935e-05, 0.999845, 9.74028e-05, 
Output 1: 0.0045659, 0.989733, 0.00567865, 2.28822e-05, 
Output 2: 0.995108, 0.00489228, 

351
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1046!
Begin processing the 200th record. run: 6207014 subRun: 1 event: 7600 at 23-Nov-2024 07:22:20 GMT
> Running Algorithm: Alg0001, LArPreProcessing
> Running Algorithm: Alg0002, LArDLMaster
> Running Algorithm: Alg0003, LArNeutrinoEventValidation
---RAW-MATCHING-OUTPUT--------------------------------------------------------------------------
Failure in algorithm Alg0003, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  23-Nov-2024 07:22:22 GMT run: 6207014 subRun: 1 event: 7600
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  23-Nov-2024 07:22:22 GMT run: 6207014 subRun: 1 event: 7600
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:22:22 GMT run: 6207014 subRun: 1 event: 7600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:22:22 GMT run: 6207014 subRun: 1 event: 7600
Comparing two wires in the same plane: return failure
%MSG
110
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 159!
23-Nov-2024 07:22:24 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207014_37_20231120T132529Z_gen_g4_detsim_hitreco__20240503T062229Z_reco2_reco_data_2024-11-23T_045827Z.root"
23-Nov-2024 07:22:24 GMT  Closed input file "root://dune.dcache.nikhef.nl:1094/pnfs/nikhef.nl/data/dune/generic/rucio/fardet-hd/0f/5a/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207014_37_20231120T132529Z_gen_g4_detsim_hitreco__20240503T062229Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.583434       38.7326       1558.93       9.58661       139.848        200    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0144151     0.0415183     0.121706      0.0394931     0.0163314       200    
reco:linecluster:LineCluster                           0.082646       0.81257       52.1593      0.173707       4.45909        200    
reco:trajcluster:TrajCluster                           0.0416945      1.7255        72.748       0.0583645      9.08888        200    
reco:pandora:StandardPandora                           0.0610821      6.94642       885.343       1.11836       62.8702        200    
reco:pandoraTrack:LArPandoraTrackCreation             0.000138463   0.00969073     0.306975     0.00163949     0.0351424       200    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000169843    0.0101657     0.178008     0.00211677     0.0256961       200    
reco:pandoracalo:Calorimetry                          9.8491e-05     0.0490023      3.22088     0.000913904    0.286829        200    
reco:pandorapid:Chi2ParticleID                          3.5e-05     0.000139848   0.00126085     9.967e-05    0.000139525      200    
reco:trkshowersplit:TrackShowerHits                   9.3466e-05     0.028691       1.16614     0.00061343     0.139193        200    
reco:pmtrack:PMAlgTrackMaker                          0.00112794      2.95899       116.659      0.145009       12.8031        200    
reco:pmtrackcalo:Calorimetry                          7.8021e-05     0.0171867     0.630549     0.000719056    0.075683        200    
reco:pmtrackpid:Chi2ParticleID                        2.9623e-05    0.000277462   0.00424216    0.000121236   0.000553695      200    
reco:pmtrajfit:PMAlgTrajFitter                        0.000184554     2.4867        81.2808       0.18303       10.1335        200    
reco:pmtrajfitcalo:Calorimetry                         6.126e-05     0.050188       3.14049     0.00130268     0.286693        200    
reco:pmtrajfitpid:Chi2ParticleID                      2.6234e-05    0.00016492    0.000964258   0.000125724   0.000154235      200    
reco:pmtracktc:PMAlgTrackMaker                        0.00101239      3.40239       110.738      0.179063       14.0247        200    
reco:pmtracktccalo:Calorimetry                        7.0904e-05     0.009895      0.361676     0.000822662    0.039399        200    
reco:pmtracktcpid:Chi2ParticleID                      2.7479e-05    0.000257769   0.00278885    0.000126698   0.000437265      200    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000175596     1.8261        49.0053      0.181512       6.3502         200    
reco:pmtrajfittccalo:Calorimetry                      6.3814e-05     0.0287705      1.32017     0.000990576    0.141184        200    
reco:pmtrajfittcpid:Chi2ParticleID                    2.7074e-05    0.000340394    0.0045598    0.000135151   0.00067907       200    
reco:blurredcluster:BlurredClustering                 5.9492e-05      1.08349       10.7549      0.475083       1.6777         200    
reco:emtrkmichelid:EmTrackMichelId                    9.5768e-05      13.9722       266.457       3.14828       35.0169        200    
reco:emshower:EMShower                                0.000150302    0.0287538     0.371066     0.00525013     0.0592748       200    
reco:cvnmap:CVNMapper                                 1.7161e-05     0.0124122      0.0473       0.0127151     0.0105876       200    
reco:cvneva:CVNEvaluator                              1.4668e-05     0.541831       3.55384      0.753183      0.428689        200    
reco:energyrecnumu:EnergyReco                         0.000243802    0.0411173      1.71298     0.00567039     0.169105        200    
reco:energyrecnue:EnergyReco                          0.000127332   0.00288995     0.0734235    0.000541577   0.00979288       200    
reco:energyrecnc:EnergyReco                           0.00179092    0.00446134     0.0745998    0.00212076    0.00985016       200    
reco:energyrecnumurange:EnergyReco                    0.000191649    0.0380144      1.67537     0.00200767     0.166953        200    
reco:energyrecnumumcs:EnergyReco                      0.00101011     0.0392596      1.69107     0.00235107     0.168493        200    
reco:opdec:Deconvolution                               0.087966      0.295553      0.662662      0.286399      0.108465        200    
reco:ophitspe:OpHitFinderDeco                          0.124367       2.05476       4.44746       2.07415      0.896799        200    
reco:opflash:OpFlashFinder                            0.000201541   0.00124892    0.00305327    0.00121921    0.000547926      200    
reco:rns:RandomNumberSaver                            1.7584e-05    2.78273e-05   0.000220658   2.3785e-05    1.56758e-05      200    
reco:anglereconue:NuAngularReco                       0.000149581   0.00236883     0.0670632    0.000314836   0.00898736       200    
reco:anglereconumu:NuAngularReco                      6.5127e-05    0.00214558     0.0657613    0.000181731   0.00867389       200    
reco:anglereconuepfps:NuAngularReco                   0.000111232    0.0022437     0.066366     0.000272353   0.00872957       200    
reco:anglereconumupfps:NuAngularReco                  9.4448e-05     0.0022238     0.0665762    0.000240522   0.00875391       200    
[art]:TriggerResults:TriggerResultInserter             7.851e-06    1.04164e-05   3.7376e-05    9.6045e-06    2.85546e-06      200    
end_path:out1:RootOutput                               2.179e-06    4.14519e-06   3.2467e-05    2.9305e-06    2.86322e-06      200    
end_path:out1:RootOutput(write)                       0.00941366     0.200994       1.27183      0.194687      0.179595        200    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9933.42 MB
  Peak resident set size usage (VmHWM): 2760.24 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207014_37_20231120T132529Z_gen_g4_detsim_hitreco__20240503T062229Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207014_37_20231120T132529Z_gen_g4_detsim_hitreco__20240503T062229Z_reco2_reco_2024-11-23T_045827Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6207014_37_20231120T132529Z_gen_g4_detsim_hitreco__20240503T062229Z_reco2_reco_data_2024-11-23T_045827Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work