justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 108777.46@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID108777.46@justin-prod-sched02.dune.hep.ac.uk
Workflow ID4213
Stage ID1
User nameismerio@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-11-23 05:01:56
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce511
Last heartbeat2024-11-23 05:54:43
From worker nodeHostnameb9p11p5970.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-11-23 05:03:15
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552530_639_20231207T120613Z_gen_g4_detsim_hitreco__20240510T055325Z_reco2.root
JobscriptExit code0
Real time51m (3068s)
CPU time51m (3097s = 100%)
Outputting started2024-11-23 05:54:23
Output files
Finished2024-11-23 05:54:43
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

with InputDigiType = 'recob'
Found hits: 478!
Begin processing the 100th record. run: 74552530 subRun: 1 event: 64000 at 23-Nov-2024 06:54:18 CET
> Running Algorithm: Alg0001, LArPreProcessing
> Running Algorithm: Alg0002, LArDLMaster
> Running Algorithm: Alg0003, LArNeutrinoEventValidation
---RAW-MATCHING-OUTPUT--------------------------------------------------------------------------
Failure in algorithm Alg0003, LArNeutrinoEventValidation, STATUS_CODE_NOT_INITIALIZED
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  23-Nov-2024 06:54:20 CET run: 74552530 subRun: 1 event: 64000
Trying to add data product: InitialTrack. This element does not exist in the element holder
%MSG
%MSG-e ShowerProducedPtrsHolder:   LArPandoraModularShowerCreation:pandoraShower@BeginModule  23-Nov-2024 06:54:20 CET run: 74552530 subRun: 1 event: 64000
Trying to add data product: ShowerPCA. This element does not exist in the element holder
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 06:54:20 CET run: 74552530 subRun: 1 event: 64000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfit@BeginModule  23-Nov-2024 06:54:20 CET run: 74552530 subRun: 1 event: 64000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:54:20 CET run: 74552530 subRun: 1 event: 64000
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:54:20 CET run: 74552530 subRun: 1 event: 64000
Comparing two wires in the same plane: return failure
%MSG
74
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 88!
23-Nov-2024 06:54:22 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552530_639_20231207T120613Z_gen_g4_detsim_hitreco__20240510T055325Z_reco2_reco_data_2024-11-23T_050321Z.root"
23-Nov-2024 06:54:22 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/bb/25/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552530_639_20231207T120613Z_gen_g4_detsim_hitreco__20240510T055325Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              2.14908       28.7436       471.606       8.91157       65.0865        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0168133     0.0432156     0.120778      0.041562      0.0176383       100    
reco:linecluster:LineCluster                           0.151217      0.357431       11.0286      0.168131       1.1393         100    
reco:trajcluster:TrajCluster                           0.047453      0.928804       25.4091      0.0619665      3.59113        100    
reco:pandora:StandardPandora                           0.911299       2.77691       55.6459       1.26408       6.94885        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000116922   0.00589019     0.0936575    0.00135001     0.0142528       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000154152    0.010336      0.182912     0.00275854     0.0226367       100    
reco:pandoracalo:Calorimetry                          9.8711e-05     0.0272745     0.954068     0.000965023    0.114372        100    
reco:pandorapid:Chi2ParticleID                         4.29e-05     0.000163421   0.000988692   0.000125126   0.000143798      100    
reco:trkshowersplit:TrackShowerHits                   9.2901e-05     0.0266712     0.581437     0.00078402     0.0919328       100    
reco:pmtrack:PMAlgTrackMaker                          0.000979552     2.59924       77.992       0.200429       9.02436        100    
reco:pmtrackcalo:Calorimetry                          6.1311e-05     0.0127461     0.355393     0.000888131    0.0502088       100    
reco:pmtrackpid:Chi2ParticleID                        2.4731e-05     0.0003215     0.0039574    0.000147992   0.000591513      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000177462     2.41728       44.3972      0.286186       7.23738        100    
reco:pmtrajfitcalo:Calorimetry                        5.5921e-05     0.0283952     0.914407     0.00134352     0.116162        100    
reco:pmtrajfitpid:Chi2ParticleID                       2.387e-05    0.00019726    0.00085812    0.000144537   0.000177036      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00080055       3.39         91.2921      0.216864       11.2914        100    
reco:pmtracktccalo:Calorimetry                         5.832e-05    0.00677265     0.153463     0.000952426    0.0197935       100    
reco:pmtracktcpid:Chi2ParticleID                       2.328e-05    0.000268321   0.00250922    0.000144397   0.000369558      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000145682     1.75375       37.1486      0.323062       4.82245        100    
reco:pmtrajfittccalo:Calorimetry                      5.6161e-05     0.0162522      0.52655     0.00119333     0.0625056       100    
reco:pmtrajfittcpid:Chi2ParticleID                     2.231e-05    0.000376596   0.00484487    0.000164202   0.000651172      100    
reco:blurredcluster:BlurredClustering                 0.00384985     0.968411        7.404        0.50231       1.27777        100    
reco:emtrkmichelid:EmTrackMichelId                     0.199773       9.86557       118.936       2.48128       20.6798        100    
reco:emshower:EMShower                                0.000240273    0.0270125     0.271184     0.00492064     0.0495009       100    
reco:cvnmap:CVNMapper                                  2.422e-05     0.0114955     0.0326873     0.0140333    0.00790255       100    
reco:cvneva:CVNEvaluator                               1.857e-05     0.723745       4.77003      0.870037      0.591908        100    
reco:energyrecnumu:EnergyReco                         0.000365164    0.0505546      1.85546     0.00381151     0.206566        100    
reco:energyrecnue:EnergyReco                          0.000165172   0.00223967     0.0293745    0.000611597   0.00479772       100    
reco:energyrecnc:EnergyReco                           0.00146007    0.00490696     0.033704     0.00329349    0.00556237       100    
reco:energyrecnumurange:EnergyReco                    0.000179952    0.0477907      1.93013     0.00299965     0.210862        100    
reco:energyrecnumumcs:EnergyReco                      0.00149557     0.0491014      1.91035     0.00406168     0.209068        100    
reco:opdec:Deconvolution                               0.0872554     0.272949      0.510948      0.277679      0.0910951       100    
reco:ophitspe:OpHitFinderDeco                          0.377165       2.05154       4.21281       1.94578      0.954659        100    
reco:opflash:OpFlashFinder                            0.000331884   0.00159394    0.00345197    0.00148978    0.000720065      100    
reco:rns:RandomNumberSaver                             2.168e-05    3.85788e-05   0.000252853   3.70305e-05   2.24851e-05      100    
reco:anglereconue:NuAngularReco                       0.000142992   0.00155832     0.0229521    0.000457456   0.00363124       100    
reco:anglereconumu:NuAngularReco                      6.8091e-05    0.00126397     0.0224409    0.000222867   0.00347211       100    
reco:anglereconuepfps:NuAngularReco                   0.000105242   0.00142594     0.0217422    0.00040505    0.00341653       100    
reco:anglereconumupfps:NuAngularReco                  9.0791e-05    0.00133339     0.021972     0.000308424   0.00341706       100    
[art]:TriggerResults:TriggerResultInserter             8.02e-06     1.46909e-05   4.5071e-05     1.388e-05    4.80196e-06      100    
end_path:out1:RootOutput                               2.88e-06     6.03917e-06    1.763e-05     6.19e-06     1.5621e-06       100    
end_path:out1:RootOutput(write)                        0.0198483     0.256662       1.16981      0.258013      0.197134        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21740.7 MB
  Peak resident set size usage (VmHWM): 2311.1 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552530_639_20231207T120613Z_gen_g4_detsim_hitreco__20240510T055325Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552530_639_20231207T120613Z_gen_g4_detsim_hitreco__20240510T055325Z_reco2_reco_2024-11-23T_050321Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74552530_639_20231207T120613Z_gen_g4_detsim_hitreco__20240510T055325Z_reco2_reco_data_2024-11-23T_050321Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work
justIN time: 2024-11-23 14:08:51 UTC       justIN version: 01.01.09