Jobsub ID 108841.43@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108841.43@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:56:54
Site	UK_RAL-PPD
Entry	CMSHTPC_T2_UK_SGrid_RALPP_hep208
Last heartbeat	2024-11-23 07:22:47
From worker node	Hostname	heplnc076.pp.rl.ac.uk
	cpuinfo	Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:58:59
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74506302_878_20231202T182343Z_gen_g4_detsim_hitreco__20240508T065431Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (5001s)
	CPU time	1h (4452s = 89%)
Outputting started	2024-11-23 07:22:21
Output files
Finished	2024-11-23 07:22:47
List job events Wrapper job log
Jobscript log (last 10,000 characters)

rocessing
> Running Algorithm: Alg0002, LArDLMaster
> Running Algorithm: Alg0003, LArNeutrinoEventValidation
---RAW-MATCHING-OUTPUT--------------------------------------------------------------------------
Failure in algorithm Alg0003, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  23-Nov-2024 07:22:13 GMT run: 74506302 subRun: 1 event: 87900
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  23-Nov-2024 07:22:13 GMT run: 74506302 subRun: 1 event: 87900
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtrack@BeginModule  23-Nov-2024 07:22:13 GMT run: 74506302 subRun: 1 event: 87900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 07:22:13 GMT run: 74506302 subRun: 1 event: 87900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  23-Nov-2024 07:22:14 GMT run: 74506302 subRun: 1 event: 87900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:22:14 GMT run: 74506302 subRun: 1 event: 87900
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:22:15 GMT run: 74506302 subRun: 1 event: 87900
Comparing two wires in the same plane: return failure
%MSG
200
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 417!
23-Nov-2024 07:22:18 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74506302_878_20231202T182343Z_gen_g4_detsim_hitreco__20240508T065431Z_reco2_reco_data_2024-11-23T_055904Z.root"
23-Nov-2024 07:22:19 GMT  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/f6/21/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74506302_878_20231202T182343Z_gen_g4_detsim_hitreco__20240508T065431Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              6.53957       46.1637       444.058       22.5256       73.398         100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0232139     0.0519561     0.201878      0.0525914     0.0242839       100    
reco:linecluster:LineCluster                           0.213393      0.416658       8.66081      0.266735      0.865329        100    
reco:trajcluster:TrajCluster                           0.0660463      1.2877        41.0795      0.105952       5.68497        100    
reco:pandora:StandardPandora                            2.12326       4.41827       41.9501       3.48352       4.35821        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000289946     0.01118      0.130082     0.00408385     0.0210723       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000320396    0.0218176     0.193495     0.00856708     0.0362779       100    
reco:pandoracalo:Calorimetry                          0.000211437    0.0228946     0.381306     0.00353604     0.0609357       100    
reco:pandorapid:Chi2ParticleID                        8.1633e-05    0.000382813   0.00308569    0.00026723    0.000393246      100    
reco:trkshowersplit:TrackShowerHits                   0.000216928    0.0230659     0.512962     0.00210352     0.0748996       100    
reco:pmtrack:PMAlgTrackMaker                          0.00190799      3.37131       56.1109      0.819173       8.26779        100    
reco:pmtrackcalo:Calorimetry                          0.000132726    0.0162186     0.245545     0.00295101     0.0429168       100    
reco:pmtrackpid:Chi2ParticleID                        5.9948e-05    0.000607976   0.00457397    0.000364776   0.000750344      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000407255     2.95524       31.5156      0.920205       5.44247        100    
reco:pmtrajfitcalo:Calorimetry                        0.000157406    0.0244925     0.461308     0.00422815     0.0654005       100    
reco:pmtrajfitpid:Chi2ParticleID                      6.4152e-05    0.000393818   0.00198475    0.00032241    0.000292218      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00267662      4.75104       106.132       0.7341        14.3455        100    
reco:pmtracktccalo:Calorimetry                        0.00020598     0.0128884      0.21153     0.00279124     0.0322813       100    
reco:pmtracktcpid:Chi2ParticleID                      8.4072e-05    0.000563007   0.00391296    0.000367203   0.000634009      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000478505     2.4785        22.7167      0.837421       4.77422        100    
reco:pmtrajfittccalo:Calorimetry                      0.000166145    0.0253235     0.506033     0.00331329     0.0725893       100    
reco:pmtrajfittcpid:Chi2ParticleID                    7.1205e-05    0.000861041    0.0117227    0.000371507   0.00148848       100    
reco:blurredcluster:BlurredClustering                  0.0288492      1.44858       12.0219      0.657888       2.05171        100    
reco:emtrkmichelid:EmTrackMichelId                     0.373475       18.6253       168.101       7.87088       30.9482        100    
reco:emshower:EMShower                                0.00047564     0.0614673     0.395719      0.0150661     0.0905455       100    
reco:cvnmap:CVNMapper                                 4.5659e-05     0.0273028     0.0706202     0.0301947     0.0174366       100    
reco:cvneva:CVNEvaluator                               3.269e-05      1.81344       3.04971       2.20667       1.00198        100    
reco:energyrecnumu:EnergyReco                         0.000722816    0.0670881      1.15635      0.0111674     0.175906        100    
reco:energyrecnue:EnergyReco                          0.00023314    0.00362838     0.0395908    0.00123836    0.00688046       100    
reco:energyrecnc:EnergyReco                           0.00254413    0.00880645     0.0514083    0.00490719     0.0088692       100    
reco:energyrecnumurange:EnergyReco                    0.000486333    0.0634544      1.09611     0.00527408     0.179986        100    
reco:energyrecnumumcs:EnergyReco                      0.00218902     0.0666151      1.14264     0.00878295     0.182257        100    
reco:opdec:Deconvolution                                0.18546      0.812759       32.7099      0.482294       3.20943        100    
reco:ophitspe:OpHitFinderDeco                          0.773158       2.8183        4.90778       2.78916      0.844706        100    
reco:opflash:OpFlashFinder                            0.00082215    0.00286279     0.0140117    0.00229077    0.00211079       100    
reco:rns:RandomNumberSaver                            3.3006e-05    6.18264e-05   0.000574335   5.46535e-05   5.40591e-05      100    
reco:anglereconue:NuAngularReco                       0.000273013   0.00271499     0.0353158    0.000767328   0.00564197       100    
reco:anglereconumu:NuAngularReco                      0.000179406   0.00256574     0.033641     0.000583296   0.00567554       100    
reco:anglereconuepfps:NuAngularReco                   0.00024519    0.00282118     0.0493245    0.000830542   0.00660728       100    
reco:anglereconumupfps:NuAngularReco                  0.000193832   0.00274638     0.0520112    0.000728342   0.00682315       100    
[art]:TriggerResults:TriggerResultInserter             1.559e-05    2.90144e-05   0.000103683   2.73725e-05   1.01435e-05      100    
end_path:out1:RootOutput                               4.099e-06    9.98088e-06   3.6597e-05    9.1055e-06    4.93078e-06      100    
end_path:out1:RootOutput(write)                        0.0485688     0.438175       1.30111      0.444373      0.234283        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 8284.27 MB
  Peak resident set size usage (VmHWM): 2188.94 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74506302_878_20231202T182343Z_gen_g4_detsim_hitreco__20240508T065431Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74506302_878_20231202T182343Z_gen_g4_detsim_hitreco__20240508T065431Z_reco2_reco_2024-11-23T_055904Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74506302_878_20231202T182343Z_gen_g4_detsim_hitreco__20240508T065431Z_reco2_reco_data_2024-11-23T_055904Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work