Jobsub ID 108830.39@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108830.39@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:33:22
Site	UK_RAL-PPD
Entry	CMSHTPC_T2_UK_SGrid_RALPP_hep208
Last heartbeat	2024-11-23 07:04:40
From worker node	Hostname	heplnc067.pp.rl.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:34:39
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66050399_600_20231207T225949Z_gen_g4_detsim_hitreco__20240510T050119Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (5371s)
	CPU time	1h (4939s = 91%)
Outputting started	2024-11-23 07:04:11
Output files
Finished	2024-11-23 07:04:35
List job events Wrapper job log
Jobscript log (last 10,000 characters)

alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.347794, 0.00722756, 0.644978, 
Output 1: 0.188673, 0.447089, 0.256448, 0.107789, 
Output 2: 0.0440435, 0.955956, 

790
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 3314!
Begin processing the 100th record. run: 66050399 subRun: 1 event: 60100 at 23-Nov-2024 07:04:01 GMT
> Running Algorithm: Alg0001, LArPreProcessing
> Running Algorithm: Alg0002, LArDLMaster
> Running Algorithm: Alg0003, LArNeutrinoEventValidation
---RAW-MATCHING-OUTPUT--------------------------------------------------------------------------
Failure in algorithm Alg0003, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  23-Nov-2024 07:04:05 GMT run: 66050399 subRun: 1 event: 60100
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  23-Nov-2024 07:04:05 GMT run: 66050399 subRun: 1 event: 60100
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 07:04:05 GMT run: 66050399 subRun: 1 event: 60100
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 07:04:05 GMT run: 66050399 subRun: 1 event: 60100
Comparing two wires in the same plane: return failure
%MSG
233
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 541!
23-Nov-2024 07:04:09 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66050399_600_20231207T225949Z_gen_g4_detsim_hitreco__20240510T050119Z_reco2_reco_data_2024-11-23T_053451Z.root"
23-Nov-2024 07:04:09 GMT  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/a6/36/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66050399_600_20231207T225949Z_gen_g4_detsim_hitreco__20240510T050119Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.974494       48.9418       504.426       18.5946       87.9518        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0253212     0.0591437      0.31138      0.0567648     0.0308205       100    
reco:linecluster:LineCluster                           0.154547       0.41238       3.11885      0.259206      0.475148        100    
reco:trajcluster:TrajCluster                           0.0661604      1.00991       23.296       0.0969444      3.19122        100    
reco:pandora:StandardPandora                           0.116992        4.777        31.611        3.53063       4.75333        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000285897    0.0134319     0.205037     0.00352289     0.0309378       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000346808    0.0223197     0.528536     0.00710022     0.0577647       100    
reco:pandoracalo:Calorimetry                          0.000235979    0.0328617     0.583668     0.00241383     0.0954059       100    
reco:pandorapid:Chi2ParticleID                        8.3047e-05    0.000370326   0.00318162    0.000295897   0.000351408      100    
reco:trkshowersplit:TrackShowerHits                   0.00029149     0.0305279     0.751082     0.00112556     0.0990914       100    
reco:pmtrack:PMAlgTrackMaker                          0.00250454      3.90333       61.5376      0.313355       10.3488        100    
reco:pmtrackcalo:Calorimetry                          0.000166123    0.0210938     0.706975     0.00196275     0.0792424       100    
reco:pmtrackpid:Chi2ParticleID                        7.0924e-05    0.000661149   0.00509098    0.000313858   0.000927226      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000412076     3.62693       41.6613      0.547253       8.56267        100    
reco:pmtrajfitcalo:Calorimetry                         0.0001769     0.0367152      0.65681      0.0030069     0.106153        100    
reco:pmtrajfitpid:Chi2ParticleID                      6.8585e-05    0.00041706    0.00170175    0.000333819   0.000330847      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00217632      4.6998        72.524       0.458646       12.4304        100    
reco:pmtracktccalo:Calorimetry                        0.000164961    0.015144      0.530763     0.00199232     0.0568173       100    
reco:pmtracktcpid:Chi2ParticleID                      7.0583e-05    0.000595735   0.00403627    0.000319581   0.000683691      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000428612     3.1677        49.5107      0.601547       7.61732        100    
reco:pmtrajfittccalo:Calorimetry                      0.000148096    0.0334622     0.790927     0.00273899     0.108873        100    
reco:pmtrajfittcpid:Chi2ParticleID                    6.8323e-05    0.000867831   0.00835301    0.000343425   0.00130971       100    
reco:blurredcluster:BlurredClustering                 0.000150496     1.62994       12.0511      0.829787       2.03287        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000240295     19.9094       226.08        5.26055       38.5204        100    
reco:emshower:EMShower                                0.000372568    0.059733      0.397203      0.0144746     0.0900848       100    
reco:cvnmap:CVNMapper                                 4.4712e-05     0.0265608     0.0817457     0.0294525     0.0170404       100    
reco:cvneva:CVNEvaluator                              3.5943e-05      1.73729       3.1904        2.09087       0.95172        100    
reco:energyrecnumu:EnergyReco                         0.000958433    0.0851883      2.91863      0.0101539     0.341891        100    
reco:energyrecnue:EnergyReco                          0.000300173    0.0040303     0.0581848     0.0012544    0.00817247       100    
reco:energyrecnc:EnergyReco                           0.00299409    0.00845416     0.0699765    0.00464211     0.0107093       100    
reco:energyrecnumurange:EnergyReco                    0.000615016    0.0848921      3.1634       0.0041188     0.366702        100    
reco:energyrecnumumcs:EnergyReco                      0.00287518     0.0900804      3.56277     0.00694388     0.401526        100    
reco:opdec:Deconvolution                               0.169498      0.444685       1.52217      0.436996       0.17151        100    
reco:ophitspe:OpHitFinderDeco                          0.209679       2.5523        4.52328       2.53168      0.958312        100    
reco:opflash:OpFlashFinder                            0.000202859    0.0026611    0.00903884    0.00252625    0.00118831       100    
reco:rns:RandomNumberSaver                            3.8332e-05    7.55431e-05   0.000643862   6.93785e-05   5.91226e-05      100    
reco:anglereconue:NuAngularReco                       0.000279885   0.00290652     0.0398189    0.000758828   0.00613148       100    
reco:anglereconumu:NuAngularReco                      0.000174999   0.00247822     0.0394424    0.000458409   0.00603448       100    
reco:anglereconuepfps:NuAngularReco                   0.000257982    0.0027819     0.0406223    0.000688925   0.00615047       100    
reco:anglereconumupfps:NuAngularReco                  0.000263748   0.00275114     0.0408023    0.000653255   0.00629203       100    
[art]:TriggerResults:TriggerResultInserter            1.6066e-05    2.7839e-05    0.000111249   2.59065e-05   9.82506e-06      100    
end_path:out1:RootOutput                               4.531e-06    1.13564e-05   3.9079e-05    1.1395e-05    4.04754e-06      100    
end_path:out1:RootOutput(write)                        0.0408665     0.426858       1.62179      0.414894      0.252717        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8375.5 MB
  Peak resident set size usage (VmHWM): 2274.6 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66050399_600_20231207T225949Z_gen_g4_detsim_hitreco__20240510T050119Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66050399_600_20231207T225949Z_gen_g4_detsim_hitreco__20240510T050119Z_reco2_reco_2024-11-23T_053451Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_66050399_600_20231207T225949Z_gen_g4_detsim_hitreco__20240510T050119Z_reco2_reco_data_2024-11-23T_053451Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work