justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 301709.26@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID301709.26@justin-prod-sched01.dune.hep.ac.uk
Workflow ID4213
Stage ID1
User nameismerio@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-11-23 04:37:41
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce512
Last heartbeat2024-11-23 05:04:01
From worker nodeHostnameb9p08p4962.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-11-23 04:39:42
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74553259_251_20231207T132206Z_gen_g4_detsim_hitreco__20240510T063156Z_reco2.root
JobscriptExit code0
Real time24m (1443s)
CPU time26m (1589s = 110%)
Outputting started2024-11-23 05:03:46
Output files
Finished2024-11-23 05:04:01
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

rajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:03:25 CET run: 74553259 subRun: 1 event: 25200
TuneFullTree failed.
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 06:03:25 CET run: 74553259 subRun: 1 event: 25200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:03:40 CET run: 74553259 subRun: 1 event: 25200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:03:40 CET run: 74553259 subRun: 1 event: 25200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:03:40 CET run: 74553259 subRun: 1 event: 25200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:03:40 CET run: 74553259 subRun: 1 event: 25200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 06:03:40 CET run: 74553259 subRun: 1 event: 25200
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 777, 1194, 624
minwire 0: 2108
minwire 1: 91
minwire 2: 2476
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0116174, 0.964645, 0.0237373, 
Output 1: 0.268909, 0.486225, 0.182395, 0.0624706, 
Output 2: 0.0416264, 0.958374, 

550
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1817!
23-Nov-2024 06:03:44 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74553259_251_20231207T132206Z_gen_g4_detsim_hitreco__20240510T063156Z_reco2_reco_data_2024-11-23T_043946Z.root"
23-Nov-2024 06:03:44 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/a9/0c/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74553259_251_20231207T132206Z_gen_g4_detsim_hitreco__20240510T063156Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.545982       12.5773       86.8475       7.84676       13.0988        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0169093     0.0400954     0.074077      0.040707     0.00999878       100    
reco:linecluster:LineCluster                           0.093859      0.179393       0.54297      0.168276      0.0461641       100    
reco:trajcluster:TrajCluster                           0.0474929     0.153089       3.66288      0.0548302     0.392344        100    
reco:pandora:StandardPandora                           0.0565608      1.25718       3.48926       1.21845      0.400727        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000129831   0.00263224     0.0196596    0.00108268    0.00366077       100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000227912   0.00850436     0.195436     0.00174922     0.0231695       100    
reco:pandoracalo:Calorimetry                           9.726e-05    0.00447136     0.111095     0.000680085    0.0124643       100    
reco:pandorapid:Chi2ParticleID                         3.42e-05     0.000144162   0.00174187    0.00010925    0.000176588      100    
reco:trkshowersplit:TrackShowerHits                    8.228e-05    0.00501279     0.152712     0.000527034    0.0164899       100    
reco:pmtrack:PMAlgTrackMaker                          0.000973677    0.667705       10.9523      0.138832       1.42236        100    
reco:pmtrackcalo:Calorimetry                          6.0941e-05    0.00250912     0.0205858      0.00064     0.00420424       100    
reco:pmtrackpid:Chi2ParticleID                         2.342e-05    0.000163699   0.000663556   0.000114766   0.000143712      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000233941    0.737443       10.4981      0.235532       1.39761        100    
reco:pmtrajfitcalo:Calorimetry                         9.023e-05    0.00420283     0.0603289    0.000969002   0.00876161       100    
reco:pmtrajfitpid:Chi2ParticleID                       3.941e-05    0.000145097   0.000629584   0.000111741   9.82282e-05      100    
reco:pmtracktc:PMAlgTrackMaker                        0.00103341      0.76194       13.0699      0.174975       1.62861        100    
reco:pmtracktccalo:Calorimetry                         6.177e-05    0.00220057     0.0226303    0.000592284   0.00364787       100    
reco:pmtracktcpid:Chi2ParticleID                       2.403e-05    0.000163484   0.000617894   0.000110071   0.000132452      100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000222522     0.54784       3.72794      0.177833      0.775535        100    
reco:pmtrajfittccalo:Calorimetry                      9.6811e-05    0.00337638     0.0549281    0.000758931   0.00715965       100    
reco:pmtrajfittcpid:Chi2ParticleID                     4.063e-05    0.000229627   0.00264669    0.000120616   0.000328339      100    
reco:blurredcluster:BlurredClustering                 0.000129491    0.784227       3.54493      0.476393      0.850297        100    
reco:emtrkmichelid:EmTrackMichelId                    0.000139311     4.64864       33.8813       1.81049       6.26638        100    
reco:emshower:EMShower                                0.000202352    0.0217587     0.230913     0.00764838     0.0347772       100    
reco:cvnmap:CVNMapper                                  2.524e-05    0.00827407     0.0316743     0.0102033    0.00709761       100    
reco:cvneva:CVNEvaluator                               1.741e-05     0.540692       1.06846      0.731194      0.397702        100    
reco:energyrecnumu:EnergyReco                         0.000300992   0.00838961     0.104081     0.00333561     0.0158758       100    
reco:energyrecnue:EnergyReco                          0.00012156    0.000843611   0.00506582    0.000337917   0.00104903       100    
reco:energyrecnc:EnergyReco                           0.00142126    0.00301417    0.00846456    0.00293418    0.00125905       100    
reco:energyrecnumurange:EnergyReco                    0.000151941   0.00682396     0.105329     0.00199347     0.0152735       100    
reco:energyrecnumumcs:EnergyReco                      0.00136187    0.00849671     0.113487      0.0031939     0.0162381       100    
reco:opdec:Deconvolution                               0.0510364      0.26119      0.406165      0.273495      0.0773196       100    
reco:ophitspe:OpHitFinderDeco                          0.154905       1.71891       3.35386       1.70196       0.73029        100    
reco:opflash:OpFlashFinder                            0.000127721   0.00122888    0.00595757    0.00113666    0.000675625      100    
reco:rns:RandomNumberSaver                             1.565e-05    3.71106e-05   0.000322762   3.0415e-05    3.7836e-05       100    
reco:anglereconue:NuAngularReco                       0.000155451   0.000654214   0.00324209    0.000383203   0.000668466      100    
reco:anglereconumu:NuAngularReco                      7.3131e-05    0.000449598   0.00321444    0.000192466   0.000617646      100    
reco:anglereconuepfps:NuAngularReco                   0.000133221   0.000586729   0.00342832    0.000316243   0.000674275      100    
reco:anglereconumupfps:NuAngularReco                  0.000103781   0.000498591   0.00313898    0.000240656   0.000609097      100    
[art]:TriggerResults:TriggerResultInserter             8.07e-06     1.2516e-05    7.6101e-05    1.12455e-05   7.20662e-06      100    
end_path:out1:RootOutput                               2.85e-06     6.23145e-06    2.265e-05     5.865e-06    2.83563e-06      100    
end_path:out1:RootOutput(write)                        0.0122059     0.182384      0.530664      0.195911      0.110905        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21396.5 MB
  Peak resident set size usage (VmHWM): 1975.92 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74553259_251_20231207T132206Z_gen_g4_detsim_hitreco__20240510T063156Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74553259_251_20231207T132206Z_gen_g4_detsim_hitreco__20240510T063156Z_reco2_reco_2024-11-23T_043946Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_74553259_251_20231207T132206Z_gen_g4_detsim_hitreco__20240510T063156Z_reco2_reco_data_2024-11-23T_043946Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work
justIN time: 2024-11-24 03:05:24 UTC       justIN version: 01.01.09