Jobsub ID 108787.37@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID	108787.37@justin-prod-sched02.dune.hep.ac.uk
Workflow ID	4213
Stage ID	1
User name	ismerio@fnal.gov
HTCondor Group	group_dune
Requested	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	80000 (22 hours)
Submitted time	2024-11-23 05:19:04
Site	UK_Lancaster
Entry	UBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat	2024-11-23 06:30:30
From worker node	Hostname	comp12-03
	cpuinfo	Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
	OS release	Scientific Linux release 7.9 (Nitrogen)
	Processors	1
	RSS bytes	4194304000 (4000 MiB)
	Wall seconds limit	257400 (71 hours)
	Inner Apptainer?	True
Job state	outputting_failed
Allocator name	justin-allocator-pro.dune.hep.ac.uk
Started	2024-11-23 05:20:16
Input files	fardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6494524_427_20231208T070226Z_gen_g4_detsim_hitreco__20240510T054108Z_reco2.root
Jobscript	Exit code	0
	Real time	1h (4197s)
	CPU time	1h (3768s = 89%)
Outputting started	2024-11-23 06:30:14
Output files
Finished	2024-11-23 06:30:30
List job events Wrapper job log
Jobscript log (last 10,000 characters)

failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0454546, 0.0959034, 0.858642, 
Output 1: 0.0406132, 0.46839, 0.444008, 0.0469885, 
Output 2: 0.661828, 0.338172, 

596
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2178!
Begin processing the 100th record. run: 6494524 subRun: 1 event: 42800 at 23-Nov-2024 06:30:10 GMT
> Running Algorithm: Alg0001, LArPreProcessing
> Running Algorithm: Alg0002, LArDLMaster
PandoraContentApi::GetList(*this, listname, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: Infer
    in file:     /exp/dune/app/users/ismerio/env-testreco/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 181
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0002, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
PandoraContentApi::GetList(*this, m_inputVertexListName, pVertexList) throw STATUS_CODE_NOT_INITIALIZED
    in function: GetHitRegion
    in file:     /exp/dune/app/users/ismerio/env-testreco/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 624
Failure in algorithm Alg0003, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
> Running Algorithm: Alg0003, LArNeutrinoEventValidation
---RAW-MATCHING-OUTPUT--------------------------------------------------------------------------
Failure in algorithm Alg0003, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
112
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 132!
23-Nov-2024 06:30:12 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6494524_427_20231208T070226Z_gen_g4_detsim_hitreco__20240510T054108Z_reco2_reco_data_2024-11-23T_052020Z.root"
23-Nov-2024 06:30:12 GMT  Closed input file "root://xrootd1.esc.qmul.ac.uk:1094//dune/RSE/fardet-hd/f2/36/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6494524_427_20231208T070226Z_gen_g4_detsim_hitreco__20240510T054108Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.884681       37.8028       862.77        10.6154       112.425        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0105229     0.0792482     0.740673      0.0354462     0.114647        100    
reco:linecluster:LineCluster                           0.102861      0.767901       22.8763      0.137546       2.91055        100    
reco:trajcluster:TrajCluster                           0.0281947      1.99699       78.8311      0.0476687      9.5605         100    
reco:pandora:StandardPandora                           0.0490263      4.69151       227.571       1.14581       23.6913        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000136561   0.00875587     0.166091     0.00175709     0.025376        100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000166039    0.0105313     0.112122      0.0026964     0.0201851       100    
reco:pandoracalo:Calorimetry                          8.8848e-05     0.0501965      2.18747     0.00109927     0.287423        100    
reco:pandorapid:Chi2ParticleID                        3.4377e-05    0.000156715   0.00111996    0.000118168   0.000147769      100    
reco:trkshowersplit:TrackShowerHits                   9.8772e-05     0.0312025     0.820911     0.000700512    0.125253        100    
reco:pmtrack:PMAlgTrackMaker                          0.00114242      3.29726       93.2418       0.1809        13.3078        100    
reco:pmtrackcalo:Calorimetry                          7.6576e-05     0.0165747     0.583801     0.000887055    0.072899        100    
reco:pmtrackpid:Chi2ParticleID                        2.9971e-05    0.000266487   0.00307099    0.000124388   0.000479606      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.00019083      2.7769        88.7586      0.252752       11.2361        100    
reco:pmtrajfitcalo:Calorimetry                        6.5956e-05     0.0491441      2.15231     0.00132935     0.278966        100    
reco:pmtrajfitpid:Chi2ParticleID                      2.8252e-05    0.000175913   0.000798499   0.000134109   0.000161287      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00100824      3.40939       107.615      0.213294       13.7785        100    
reco:pmtracktccalo:Calorimetry                        7.3287e-05    0.00956366     0.204656     0.00105313     0.0323834       100    
reco:pmtracktcpid:Chi2ParticleID                      2.8122e-05    0.000242062   0.00212399    0.000128087   0.000373893      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000177687     1.70258       37.6323      0.308889       5.43427        100    
reco:pmtrajfittccalo:Calorimetry                      6.2451e-05     0.0281246     0.795578     0.00123329     0.120184        100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.7558e-05    0.000339733   0.00402174    0.000149003   0.000678388      100    
reco:blurredcluster:BlurredClustering                 0.000119432     1.08441       6.21604      0.571448       1.42493        100    
reco:emtrkmichelid:EmTrackMichelId                     0.031765       14.1871       197.459       3.90002       33.9607        100    
reco:emshower:EMShower                                0.000237302    0.0276774     0.333239      0.0032047     0.0613893       100    
reco:cvnmap:CVNMapper                                  2.042e-05     0.0129647     0.0500467     0.0126777    0.00944601       100    
reco:cvneva:CVNEvaluator                              1.5945e-05     0.598744       0.90067       0.78595      0.347342        100    
reco:energyrecnumu:EnergyReco                         0.000284255    0.0705907      2.48527      0.0055582     0.284768        100    
reco:energyrecnue:EnergyReco                          0.000134021   0.00280015     0.0477411    0.000503762   0.00799105       100    
reco:energyrecnc:EnergyReco                           0.00170562    0.00449366     0.0503779    0.00220027    0.00803481       100    
reco:energyrecnumurange:EnergyReco                    0.000204605    0.0677929      2.52908     0.00194404      0.28692        100    
reco:energyrecnumumcs:EnergyReco                      0.00123664     0.0693906      2.48268     0.00271409     0.286189        100    
reco:opdec:Deconvolution                               0.0387652     0.329252       2.06771      0.258785      0.324252        100    
reco:ophitspe:OpHitFinderDeco                          0.164419       2.19118       5.93064       2.1246       0.963046        100    
reco:opflash:OpFlashFinder                            0.000195825   0.00126979    0.00369725    0.00116541    0.000561786      100    
reco:rns:RandomNumberSaver                             1.743e-05    3.06491e-05   0.000265157   2.67255e-05   2.5001e-05       100    
reco:anglereconue:NuAngularReco                       0.00014923    0.00224064     0.0456798    0.000356055   0.00675808       100    
reco:anglereconumu:NuAngularReco                      6.8304e-05     0.0020437     0.0437569    0.000199043   0.00659498       100    
reco:anglereconuepfps:NuAngularReco                   0.000114483   0.00216408     0.0440078    0.000315235   0.00663409       100    
reco:anglereconumupfps:NuAngularReco                  9.8206e-05     0.0021085     0.0431118    0.000270594   0.00655318       100    
[art]:TriggerResults:TriggerResultInserter             8.712e-06    1.07172e-05   4.0788e-05    1.00055e-05   3.39082e-06      100    
end_path:out1:RootOutput                               2.224e-06    4.41751e-06   1.7115e-05     3.586e-06    2.15394e-06      100    
end_path:out1:RootOutput(write)                       0.00854924     0.217829       1.08104      0.203708      0.180307        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9552.02 MB
  Peak resident set size usage (VmHWM): 2394.19 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6494524_427_20231208T070226Z_gen_g4_detsim_hitreco__20240510T054108Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6494524_427_20231208T070226Z_gen_g4_detsim_hitreco__20240510T054108Z_reco2_reco_2024-11-23T_052020Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6494524_427_20231208T070226Z_gen_g4_detsim_hitreco__20240510T054108Z_reco2_reco_data_2024-11-23T_052020Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work