Jobsub ID 301641.148@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID	301641.148@justin-prod-sched01.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 02:12:27
Site	UK_Lancaster
Entry	UBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat	2024-11-23 03:00:48
From worker node	Hostname	comp11-00
	cpuinfo	Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 02:21:49
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6404651_754_20231202T103538Z_gen_g4_detsim_hitreco__20240507T214447Z_reco2.root
Jobscript	Exit code	0
	Real time	38m (2324s)
	CPU time	33m (2014s = 86%)
Outputting started	2024-11-23 03:00:34
Output files
Finished	2024-11-23 03:00:48
List job events Wrapper job log
Jobscript log (last 10,000 characters)

in and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.976336, 0.00302708, 0.0206365, 
Output 1: 0.844533, 0.135315, 0.0171652, 0.00298713, 
Output 2: 0.983184, 0.0168159, 

297
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 893!
Begin processing the 100th record. run: 6404651 subRun: 1 event: 75500 at 23-Nov-2024 03:00:27 GMT
> Running Algorithm: Alg0001, LArPreProcessing
> Running Algorithm: Alg0002, LArDLMaster
> Running Algorithm: Alg0003, LArNeutrinoEventValidation
---RAW-MATCHING-OUTPUT--------------------------------------------------------------------------
Failure in algorithm Alg0003, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  23-Nov-2024 03:00:29 GMT run: 6404651 subRun: 1 event: 75500
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 03:00:29 GMT run: 6404651 subRun: 1 event: 75500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 03:00:29 GMT run: 6404651 subRun: 1 event: 75500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 03:00:29 GMT run: 6404651 subRun: 1 event: 75500
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 03:00:29 GMT run: 6404651 subRun: 1 event: 75500
Comparing two wires in the same plane: return failure
%MSG
247
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 571!
23-Nov-2024 03:00:31 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6404651_754_20231202T103538Z_gen_g4_detsim_hitreco__20240507T214447Z_reco2_reco_data_2024-11-23T_022153Z.root"
23-Nov-2024 03:00:32 GMT  Closed input file "root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/1b/5c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6404651_754_20231202T103538Z_gen_g4_detsim_hitreco__20240507T214447Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.20638       20.6196       360.559       10.9019       39.7684        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0322376     0.131496      0.588707      0.099949      0.0983911       100    
reco:linecluster:LineCluster                           0.144912      0.463647       9.91337      0.331001      0.966969        100    
reco:trajcluster:TrajCluster                           0.0369522     0.678529       38.5771       0.12642       3.87316        100    
reco:pandora:StandardPandora                            1.05583       1.65926       19.119        1.35101       1.83444        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000117007    0.0037033     0.0639585    0.00149122    0.00773178       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000147978   0.00796009     0.0786736    0.00290994     0.0138876       100    
reco:pandoracalo:Calorimetry                          8.9853e-05     0.0128972     0.823436     0.000985018    0.0822567       100    
reco:pandorapid:Chi2ParticleID                        3.4015e-05    0.000136469   0.000917708   0.000120697   0.000111313      100    
reco:trkshowersplit:TrackShowerHits                    8.34e-05     0.00882143     0.333919     0.000606809    0.0370344       100    
reco:pmtrack:PMAlgTrackMaker                          0.00102176      1.11935       39.4756      0.153599       4.20566        100    
reco:pmtrackcalo:Calorimetry                          6.8837e-05     0.005342      0.222898     0.000802106    0.0228202       100    
reco:pmtrackpid:Chi2ParticleID                        2.7944e-05    0.000189039   0.00227051    0.000127139   0.000261144      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000165176     1.09131       22.3701      0.263384       2.84976        100    
reco:pmtrajfitcalo:Calorimetry                        5.7674e-05     0.0137574     0.872983     0.00122949     0.0871162       100    
reco:pmtrajfitpid:Chi2ParticleID                      2.5092e-05    0.000144056   0.000613047   0.000126667   9.68468e-05      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000892879     1.46846       60.8943      0.194051       6.37928        100    
reco:pmtracktccalo:Calorimetry                        6.3822e-05    0.00420013     0.105566     0.000856867    0.0121004       100    
reco:pmtracktcpid:Chi2ParticleID                      2.5611e-05    0.000180584   0.00170896    0.00013287    0.000209716      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000156871     0.84605       25.8435      0.270501       2.67472        100    
reco:pmtrajfittccalo:Calorimetry                      5.5151e-05    0.00873046     0.434238     0.00107357     0.0440046       100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.4629e-05    0.000230394   0.00301619    0.000140522   0.000344984      100    
reco:blurredcluster:BlurredClustering                 0.00397092     0.973051       5.59317      0.551519       1.14796        100    
reco:emtrkmichelid:EmTrackMichelId                     0.112823       8.7918        132.217       3.35092       16.7365        100    
reco:emshower:EMShower                                0.000214521    0.0178503     0.141162     0.00284937     0.0297434       100    
reco:cvnmap:CVNMapper                                 1.7266e-05     0.0141171     0.0452169     0.0129826    0.00966131       100    
reco:cvneva:CVNEvaluator                              1.4569e-05     0.562223      0.811453      0.715131      0.308489        100    
reco:energyrecnumu:EnergyReco                         0.000268076    0.0201591     0.413419     0.00468393     0.0568882       100    
reco:energyrecnue:EnergyReco                          0.000121936   0.00118956     0.0138736    0.000477767   0.00198734       100    
reco:energyrecnc:EnergyReco                           0.00168485    0.00271533     0.015525     0.00208535    0.00202962       100    
reco:energyrecnumurange:EnergyReco                    0.000193306    0.0180644     0.410023     0.00185254     0.0564871       100    
reco:energyrecnumumcs:EnergyReco                      0.00164769     0.0191502     0.410639     0.00288099     0.0563866       100    
reco:opdec:Deconvolution                               0.0929912     0.374305      0.743954      0.371348      0.123411        100    
reco:ophitspe:OpHitFinderDeco                          0.201348       2.10047       3.64324       2.25593      0.783148        100    
reco:opflash:OpFlashFinder                            0.000181428   0.00117791    0.00329554    0.00120343    0.000478647      100    
reco:rns:RandomNumberSaver                            1.6697e-05    2.84926e-05   0.000207633   2.68605e-05   1.92673e-05      100    
reco:anglereconue:NuAngularReco                       0.000133367   0.000839174   0.00964984    0.000370491   0.00148886       100    
reco:anglereconumu:NuAngularReco                      6.5188e-05     0.0006638     0.008828     0.000196733   0.00141177       100    
reco:anglereconuepfps:NuAngularReco                   0.000106676   0.000763316   0.00896967    0.000286527   0.00143238       100    
reco:anglereconumupfps:NuAngularReco                   8.853e-05    0.000723762   0.00892474    0.000261338   0.00142146       100    
[art]:TriggerResults:TriggerResultInserter             7.908e-06    1.05255e-05   3.6173e-05     9.733e-06    3.4059e-06       100    
end_path:out1:RootOutput                               2.198e-06    4.47626e-06   1.4856e-05     4.275e-06    2.01475e-06      100    
end_path:out1:RootOutput(write)                       0.00679607     0.194382      0.765102      0.206076      0.113212        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9567.61 MB
  Peak resident set size usage (VmHWM): 2410.59 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6404651_754_20231202T103538Z_gen_g4_detsim_hitreco__20240507T214447Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6404651_754_20231202T103538Z_gen_g4_detsim_hitreco__20240507T214447Z_reco2_reco_2024-11-23T_022153Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6404651_754_20231202T103538Z_gen_g4_detsim_hitreco__20240507T214447Z_reco2_reco_data_2024-11-23T_022153Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work