justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 301657.39@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID301657.39@justin-prod-sched01.dune.hep.ac.uk
Workflow ID4213
Stage ID1
User nameismerio@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-11-23 02:40:56
SiteES_PIC
EntryDUNE_T1_ES_PIC_ce15-multicore
Last heartbeat2024-11-23 04:15:25
From worker nodeHostnametds408.pic.es
cpuinfoAMD EPYC 7502 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit216000 (60 hours)
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-11-23 02:51:57
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406796_129_20231202T132231Z_gen_g4_detsim_hitreco__20240507T222338Z_reco2.root
JobscriptExit code0
Real time1h (4986s)
CPU time1h (4204s = 84%)
Outputting started2024-11-23 04:15:04
Output files
Finished2024-11-23 04:15:25
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

ed alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.874903, 0.11774, 0.00735719, 
Output 1: 0.362474, 0.627877, 0.0082466, 0.00140202, 
Output 2: 0.943985, 0.0560148, 

653
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2531!
Begin processing the 100th record. run: 6406796 subRun: 1 event: 13000 at 23-Nov-2024 05:14:59 CET
> Running Algorithm: Alg0001, LArPreProcessing
> Running Algorithm: Alg0002, LArDLMaster
PandoraContentApi::GetList(*this, listname, pCaloHitList) return STATUS_CODE_NOT_INITIALIZED
    in function: Infer
    in file:     /exp/dune/app/users/ismerio/env-testreco/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 181
iter->second->Run() throw STATUS_CODE_NOT_INITIALIZED
    in function: RunAlgorithm
    in file:     /scratch/workspace/build-larbase/BUILDTYPE/prof/QUAL/s131-e26/label1/swarm/label2/SLF7/build/pandora/v03_16_00l/src/pandora-v03-16-00/PandoraSDK-v03-04-01/src/Api/PandoraContentApiImpl.cc line#: 235
Failure in algorithm Alg0002, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
PandoraContentApi::GetList(*this, m_inputVertexListName, pVertexList) throw STATUS_CODE_NOT_INITIALIZED
    in function: GetHitRegion
    in file:     /exp/dune/app/users/ismerio/env-testreco/srcs/larpandoracontent/larpandoradlcontent/LArVertex/DlVertexingAlgorithm.cc line#: 624
Failure in algorithm Alg0003, LArDLVertexing, STATUS_CODE_NOT_INITIALIZED
> Running Algorithm: Alg0003, LArNeutrinoEventValidation
---RAW-MATCHING-OUTPUT--------------------------------------------------------------------------
Failure in algorithm Alg0003, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
68
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 152!
23-Nov-2024 05:15:02 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406796_129_20231202T132231Z_gen_g4_detsim_hitreco__20240507T222338Z_reco2_reco_data_2024-11-23T_025203Z.root"
23-Nov-2024 05:15:02 CET  Closed input file "root://ccxrootdegee.in2p3.fr:1094/pnfs/in2p3.fr/data/dune/disk/fardet-hd/dc/21/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406796_129_20231202T132231Z_gen_g4_detsim_hitreco__20240507T222338Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.03344       42.9186       1115.62       12.4956       128.64         100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0337308     0.0564705     0.0927218     0.0568331     0.0184154       100    
reco:linecluster:LineCluster                            0.19808       1.12813       58.7887      0.348453       5.89142        100    
reco:trajcluster:TrajCluster                           0.0993243      3.42356       262.852      0.110091       26.2721        100    
reco:pandora:StandardPandora                           0.114088       3.22428       62.7018       1.8072        7.38461        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000175342   0.00765853     0.139301      0.0017828     0.019643        100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000247193    0.0109108     0.119612     0.00354029     0.0192212       100    
reco:pandoracalo:Calorimetry                          0.000131471    0.0310364     0.969081     0.00114492     0.137656        100    
reco:pandorapid:Chi2ParticleID                         4.986e-05    0.000219067   0.00220751    0.000150872   0.000245487      100    
reco:trkshowersplit:TrackShowerHits                   9.9582e-05     0.0351506      1.31382     0.000766635    0.159583        100    
reco:pmtrack:PMAlgTrackMaker                          0.00169618      4.58506       148.35       0.186133       18.8514        100    
reco:pmtrackcalo:Calorimetry                          9.1321e-05     0.0111413     0.288251      0.0011661     0.0372788       100    
reco:pmtrackpid:Chi2ParticleID                         3.784e-05    0.000389405   0.00320612    0.000179453   0.000567512      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000252133     3.12511       102.476      0.267895       11.9147        100    
reco:pmtrajfitcalo:Calorimetry                        9.0941e-05     0.0302356     0.938632     0.00164681     0.129768        100    
reco:pmtrajfitpid:Chi2ParticleID                       3.77e-05     0.00023044    0.00111819    0.000177367   0.000181884      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00159501      5.57256       211.198      0.268471       24.0676        100    
reco:pmtracktccalo:Calorimetry                        9.2311e-05     0.0077834     0.174751     0.00117792     0.0239383       100    
reco:pmtracktcpid:Chi2ParticleID                       3.822e-05    0.00035925    0.00328423    0.000182247   0.000516744      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000238543     2.46326       49.9388      0.321859       7.36851        100    
reco:pmtrajfittccalo:Calorimetry                      8.4511e-05     0.0232323     0.739776     0.00122209     0.0986751       100    
reco:pmtrajfittcpid:Chi2ParticleID                    3.6361e-05    0.000500088   0.00566293    0.000191512   0.000885527      100    
reco:blurredcluster:BlurredClustering                 0.000113851     1.10003       8.60517      0.591285       1.51683        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000162132     13.7607       197.671       3.2936        30.3825        100    
reco:emshower:EMShower                                0.000283183    0.0471902     0.386017      0.0152725     0.0757094       100    
reco:cvnmap:CVNMapper                                  2.476e-05    0.00888721     0.0537029    0.00819807    0.00966465       100    
reco:cvneva:CVNEvaluator                              1.8241e-05     0.816168       1.89812       1.1405       0.546683        100    
reco:energyrecnumu:EnergyReco                         0.000400556    0.036175       1.04227      0.0057476     0.133229        100    
reco:energyrecnue:EnergyReco                          0.000165003    0.0025792     0.0395905    0.000662504   0.00589912       100    
reco:energyrecnc:EnergyReco                           0.00241974    0.00576501     0.0480574    0.00387807    0.00667998       100    
reco:energyrecnumurange:EnergyReco                    0.000232053    0.0337947      1.08506     0.00370272     0.133374        100    
reco:energyrecnumumcs:EnergyReco                      0.00254433     0.0357357      1.09049      0.0045633     0.132197        100    
reco:opdec:Deconvolution                               0.103254       0.49904       1.51321      0.477176      0.192346        100    
reco:ophitspe:OpHitFinderDeco                          0.323794       2.49798       4.69071       2.45234      0.953244        100    
reco:opflash:OpFlashFinder                            0.000221703   0.00174222    0.00656695    0.00163454    0.000876125      100    
reco:rns:RandomNumberSaver                             1.822e-05    3.87853e-05   0.000411505   3.09655e-05   3.93129e-05      100    
reco:anglereconue:NuAngularReco                       0.000189292   0.00177209     0.031265     0.000438491    0.0046163       100    
reco:anglereconumu:NuAngularReco                      9.9132e-05    0.00157808     0.0308107    0.000251548   0.00456936       100    
reco:anglereconuepfps:NuAngularReco                   0.000168982   0.00176019     0.0318482    0.000390405   0.00465502       100    
reco:anglereconumupfps:NuAngularReco                  0.000145462   0.00168016     0.0309905    0.000337655   0.00462281       100    
[art]:TriggerResults:TriggerResultInserter             1.064e-05    1.65903e-05   7.3801e-05    1.48905e-05   7.9658e-06       100    
end_path:out1:RootOutput                               3.15e-06     6.37169e-06    2.95e-05      4.505e-06    4.19619e-06      100    
end_path:out1:RootOutput(write)                        0.0194486     0.326125       1.5777       0.317874      0.251927        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21354.3 MB
  Peak resident set size usage (VmHWM): 2217.94 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406796_129_20231202T132231Z_gen_g4_detsim_hitreco__20240507T222338Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406796_129_20231202T132231Z_gen_g4_detsim_hitreco__20240507T222338Z_reco2_reco_2024-11-23T_025203Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_6406796_129_20231202T132231Z_gen_g4_detsim_hitreco__20240507T222338Z_reco2_reco_data_2024-11-23T_025203Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work
justIN time: 2024-11-23 11:47:31 UTC       justIN version: 01.01.09