justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 108633.186@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID108633.186@justin-prod-sched02.dune.hep.ac.uk
Workflow ID4213
Stage ID1
User nameismerio@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-11-23 00:37:10
SiteUS_UConn-HPC
EntryGLUEX_US_UConn-HPC_osgce
Last heartbeat2024-11-23 01:36:15
From worker nodeHostnamecn425
cpuinfoAMD EPYC 7452 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit172800 (48 hours)
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-11-23 00:44:33
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_585_20231205T103110Z_gen_g4_detsim_hitreco__20240510T040308Z_reco2.root
JobscriptExit code0
Real time51m (3088s)
CPU time48m (2904s = 94%)
Outputting started2024-11-23 01:36:02
Output files
Finished2024-11-23 01:36:15
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

50602933 subRun: 1 event: 58600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrackMaker:pmtracktc@BeginModule  22-Nov-2024 20:35:56 EST run: 50602933 subRun: 1 event: 58600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 20:35:56 EST run: 50602933 subRun: 1 event: 58600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 20:35:56 EST run: 50602933 subRun: 1 event: 58600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  22-Nov-2024 20:35:56 EST run: 50602933 subRun: 1 event: 58600
1st wire C:0 T:3 P:2 W:622 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  22-Nov-2024 20:35:58 EST run: 50602933 subRun: 1 event: 58600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  22-Nov-2024 20:35:58 EST run: 50602933 subRun: 1 event: 58600
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 49, 61, 46
minwire 0: 280
minwire 1: 2612
minwire 2: 5
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.0132397, 0.983908, 0.00285262, 
Output 1: 0.857337, 0.138199, 0.00377545, 0.000688526, 
Output 2: 0.998243, 0.00175724, 

212
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 463!
22-Nov-2024 20:36:00 EST  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_585_20231205T103110Z_gen_g4_detsim_hitreco__20240510T040308Z_reco2_reco_data_2024-11-23T_004439Z.root"
22-Nov-2024 20:36:00 EST  Closed input file "root://dcdndoor.sdcc.bnl.gov:1094//pnfs/sdcc.bnl.gov/data/dune/RSE/fardet-hd/fb/8e/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_585_20231205T103110Z_gen_g4_detsim_hitreco__20240510T040308Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                              1.99531       28.9107       417.595       12.076        57.6161        100    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                0.00762574     0.0543448      0.13219      0.0542417     0.0306341       100    
reco:linecluster:LineCluster                           0.0683751     0.171552        2.698       0.109042      0.278228        100    
reco:trajcluster:TrajCluster                           0.0207126     0.481786       18.8857      0.0346681      2.07294        100    
reco:pandora:StandardPandora                           0.974516       2.10227       21.6824       1.53874       2.48137        100    
reco:pandoraTrack:LArPandoraTrackCreation             0.000142068   0.00650444     0.108638     0.00162879     0.015882        100    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000213523    0.0184903      0.36783     0.00323922     0.0507865       100    
reco:pandoracalo:Calorimetry                          0.000122231    0.0140007      0.29886     0.00128074     0.0417597       100    
reco:pandorapid:Chi2ParticleID                        3.8472e-05    0.000209472   0.00247798    0.000146662   0.000268981      100    
reco:trkshowersplit:TrackShowerHits                   0.000109667    0.0147653      0.3506      0.000946924    0.0482727       100    
reco:pmtrack:PMAlgTrackMaker                           0.0011808      2.3139        46.3712      0.278778       7.0319         100    
reco:pmtrackcalo:Calorimetry                          6.3019e-05     0.0131774      0.32811     0.00118907     0.049354        100    
reco:pmtrackpid:Chi2ParticleID                        2.6941e-05    0.000337147   0.00339054    0.000181944   0.000498842      100    
reco:pmtrajfit:PMAlgTrajFitter                        0.000219014     1.86163       28.0922      0.345509       4.46583        100    
reco:pmtrajfitcalo:Calorimetry                        5.9652e-05     0.0143509     0.295842     0.00178727     0.0454098       100    
reco:pmtrajfitpid:Chi2ParticleID                       2.654e-05    0.000219639   0.00121619    0.000173132   0.000180403      100    
reco:pmtracktc:PMAlgTrackMaker                        0.000954403     2.60218       70.8256      0.326928       8.45113        100    
reco:pmtracktccalo:Calorimetry                        5.8681e-05    0.00909319     0.221311      0.0013738     0.0321327       100    
reco:pmtracktcpid:Chi2ParticleID                      2.7683e-05    0.000306212    0.0025252    0.000186988   0.00039092       100    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000168839     1.5432        27.0656      0.381828       4.12393        100    
reco:pmtrajfittccalo:Calorimetry                      5.6998e-05     0.0139219     0.425275     0.00174287     0.0500919       100    
reco:pmtrajfittcpid:Chi2ParticleID                    2.4196e-05    0.000391055   0.00381819    0.000191016   0.000557656      100    
reco:blurredcluster:BlurredClustering                 0.00280105      1.14915       9.11743      0.524951       1.61044        100    
reco:emtrkmichelid:EmTrackMichelId                     0.237059       13.3636       171.243       4.85947       26.7151        100    
reco:emshower:EMShower                                0.000297292    0.0274507     0.464001     0.00511829     0.0578257       100    
reco:cvnmap:CVNMapper                                 2.4666e-05     0.0125637     0.0555608     0.0117595     0.010663        100    
reco:cvneva:CVNEvaluator                              1.7754e-05      0.90937       13.3878       1.0522        1.34635        100    
reco:energyrecnumu:EnergyReco                         0.00037581     0.0327517     0.739449     0.00495628     0.0936923       100    
reco:energyrecnue:EnergyReco                          0.000121389   0.00199175     0.0349668    0.00055037    0.00481655       100    
reco:energyrecnc:EnergyReco                           0.00196123    0.00436678     0.0372622    0.00310441    0.00517125       100    
reco:energyrecnumurange:EnergyReco                    0.000191121    0.0317269     0.808802     0.00284386     0.0996646       100    
reco:energyrecnumumcs:EnergyReco                       0.0010962     0.0325678     0.740979     0.00381541     0.0944632       100    
reco:opdec:Deconvolution                               0.0537527     0.278576      0.576369       0.25991       0.13309        100    
reco:ophitspe:OpHitFinderDeco                          0.229164       1.57842       3.23939       1.6513       0.710714        100    
reco:opflash:OpFlashFinder                             0.0003438    0.00164732     0.0112907    0.00145028    0.00119918       100    
reco:rns:RandomNumberSaver                              2.1e-05     5.10632e-05   0.00107945    3.73405e-05   0.000104476      100    
reco:anglereconue:NuAngularReco                       0.000166224   0.00146988     0.029042     0.000434576   0.00387974       100    
reco:anglereconumu:NuAngularReco                      7.4461e-05    0.00123133     0.0287895    0.000235249   0.00382309       100    
reco:anglereconuepfps:NuAngularReco                   0.000122011   0.00143983     0.0292246    0.000394285   0.00389943       100    
reco:anglereconumupfps:NuAngularReco                  9.6012e-05    0.00128173     0.0286582    0.000297491   0.00381884       100    
[art]:TriggerResults:TriggerResultInserter             8.987e-06    1.48064e-05   9.5771e-05    1.34905e-05   8.79994e-06      100    
end_path:out1:RootOutput                               4.097e-06    8.20629e-06   4.0116e-05    6.5775e-06    4.76117e-06      100    
end_path:out1:RootOutput(write)                        0.018397      0.242067       1.15078      0.255601      0.171385        100    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 21233.9 MB
  Peak resident set size usage (VmHWM): 2350.85 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_585_20231205T103110Z_gen_g4_detsim_hitreco__20240510T040308Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_585_20231205T103110Z_gen_g4_detsim_hitreco__20240510T040308Z_reco2_reco_2024-11-23T_004439Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_50602933_585_20231205T103110Z_gen_g4_detsim_hitreco__20240510T040308Z_reco2_reco_data_2024-11-23T_004439Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work
justIN time: 2024-11-23 12:09:35 UTC       justIN version: 01.01.09