justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 301680.5@justin-prod-sched01.dune.hep.ac.uk

Jobsub ID301680.5@justin-prod-sched01.dune.hep.ac.uk
Workflow ID4213
Stage ID1
User nameismerio@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-11-23 03:42:17
SiteCERN
EntryCMSHTPC_T2_CH_CERN_ce509
Last heartbeat2024-11-23 06:25:10
From worker nodeHostnameb9p05p3580.cern.ch
cpuinfoAMD EPYC 7543 32-Core Processor
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit171000 (47 hours)
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-11-23 03:43:13
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1794_207_20231028T042641Z_gen_g4_detsim_hitreco__20240503T044625Z_reco2.root
JobscriptExit code0
Real time2h (9682s)
CPU time2h (9699s = 100%)
Outputting started2024-11-23 06:24:36
Output files
Finished2024-11-23 06:25:10
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

ginModule  23-Nov-2024 07:24:18 CET run: 1794 subRun: 1 event: 41600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:24:18 CET run: 1794 subRun: 1 event: 41600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:24:18 CET run: 1794 subRun: 1 event: 41600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:24:18 CET run: 1794 subRun: 1 event: 41600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 07:24:18 CET run: 1794 subRun: 1 event: 41600
1st wire C:0 T:22 P:2 W:486 does not exist (max wire number: 480)
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:24:30 CET run: 1794 subRun: 1 event: 41600
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 07:24:30 CET run: 1794 subRun: 1 event: 41600
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 541, 511, 542
minwire 0: 2143
minwire 1: 134
minwire 2: 2530
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.397794, 0.332922, 0.269284, 
Output 1: 0.295065, 0.382992, 0.198221, 0.123723, 
Output 2: 0.0515064, 0.948494, 

430
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 1299!
23-Nov-2024 07:24:34 CET  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1794_207_20231028T042641Z_gen_g4_detsim_hitreco__20240503T044625Z_reco2_reco_data_2024-11-23T_034319Z.root"
23-Nov-2024 07:24:35 CET  Closed input file "root://golias100.farm.particle.cz:1094/dpm/farm.particle.cz/home/dune/RSE/fardet-hd/98/82/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1794_207_20231028T042641Z_gen_g4_detsim_hitreco__20240503T044625Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.523585       46.3068       1354.26       9.75852       147.661        200    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0175305     0.0424732     0.0965129     0.0433842     0.0137785       200    
reco:linecluster:LineCluster                           0.0949204     0.892133       37.4499      0.184029       3.63522        200    
reco:trajcluster:TrajCluster                           0.0470928      2.4226        108.314      0.0633074      11.3471        200    
reco:pandora:StandardPandora                           0.0542527      4.55569       209.717       1.20529       19.3002        200    
reco:pandoraTrack:LArPandoraTrackCreation             0.000114852    0.0103772     0.417579     0.00162534     0.0374891       200    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000151862    0.0160498     0.232614     0.00338761     0.0378617       200    
reco:pandoracalo:Calorimetry                          8.6311e-05     0.0547701      2.70638     0.00134354     0.255203        200    
reco:pandorapid:Chi2ParticleID                         3.158e-05    0.000171121   0.00148931    0.000125892   0.000184248      200    
reco:trkshowersplit:TrackShowerHits                   6.1671e-05     0.048202       1.4889      0.00109984     0.193777        200    
reco:pmtrack:PMAlgTrackMaker                          0.000917006     5.39604       163.008      0.239314       21.6899        200    
reco:pmtrackcalo:Calorimetry                          5.7471e-05     0.0327901      1.88653     0.00100651     0.166063        200    
reco:pmtrackpid:Chi2ParticleID                         2.283e-05    0.000347572   0.00508559    0.000151248   0.000687597      200    
reco:pmtrajfit:PMAlgTrajFitter                        0.000146932     5.23333       240.345      0.324017       23.0785        200    
reco:pmtrajfitcalo:Calorimetry                        5.3961e-05     0.0586738      2.34079     0.00155666     0.259947        200    
reco:pmtrajfitpid:Chi2ParticleID                       2.249e-05    0.000192552   0.00140634    0.000143987   0.000191794      200    
reco:pmtracktc:PMAlgTrackMaker                        0.000766983     6.19243       207.072      0.298659       24.4883        200    
reco:pmtracktccalo:Calorimetry                        5.5201e-05     0.015363      0.956475     0.00104013     0.0758238       200    
reco:pmtracktcpid:Chi2ParticleID                       2.223e-05    0.000298767   0.00374434    0.000151233   0.000497758      200    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000122673     3.07121       131.257       0.3599        12.238         200    
reco:pmtrajfittccalo:Calorimetry                      5.2271e-05     0.0330407      1.68662     0.00128381     0.153425        200    
reco:pmtrajfittcpid:Chi2ParticleID                    2.1531e-05    0.000434534   0.00564468    0.000168837   0.000846075      200    
reco:blurredcluster:BlurredClustering                 0.000108541     1.0113        9.4374       0.460676       1.48788        200    
reco:emtrkmichelid:EmTrackMichelId                    0.000112042     13.7505       282.359       2.95214       33.6687        200    
reco:emshower:EMShower                                0.000171733    0.0351242     0.551011     0.00594401     0.0801475       200    
reco:cvnmap:CVNMapper                                 2.0781e-05    0.00979309     0.0459998    0.00729105    0.00898469       200    
reco:cvneva:CVNEvaluator                               1.521e-05     0.695638       2.0091       0.791259      0.411577        200    
reco:energyrecnumu:EnergyReco                         0.000305565    0.132068       12.2374     0.00460706     0.920303        200    
reco:energyrecnue:EnergyReco                          0.000114512   0.00384043     0.155454     0.000625211    0.0151137       200    
reco:energyrecnc:EnergyReco                           0.00163182    0.00617335      0.20208     0.00301694     0.0177782       200    
reco:energyrecnumurange:EnergyReco                    0.000125552    0.125863       11.7121     0.00260219     0.884444        200    
reco:energyrecnumumcs:EnergyReco                      0.000570799    0.126257       11.5075     0.00345539     0.871814        200    
reco:opdec:Deconvolution                               0.0541286     0.275926      0.498157      0.277585      0.0915134       200    
reco:ophitspe:OpHitFinderDeco                          0.140385       1.79978       4.13538       1.75617      0.837229        200    
reco:opflash:OpFlashFinder                            0.000124802   0.00132324    0.00427099    0.00123564    0.000640572      200    
reco:rns:RandomNumberSaver                             1.579e-05    3.16217e-05   0.000301735   3.00755e-05   2.14261e-05      200    
reco:anglereconue:NuAngularReco                       0.000149353   0.00293132     0.130635     0.000384331    0.0119472       200    
reco:anglereconumu:NuAngularReco                      6.0481e-05    0.00267808     0.127602     0.000239764    0.0116417       200    
reco:anglereconuepfps:NuAngularReco                   0.000104172   0.00278526     0.125555     0.000354926    0.0115228       200    
reco:anglereconumupfps:NuAngularReco                  8.1291e-05    0.00272222     0.125461     0.00030042     0.0114913       200    
[art]:TriggerResults:TriggerResultInserter             8.29e-06     1.23699e-05    4.346e-05    1.16455e-05   4.03831e-06      200    
end_path:out1:RootOutput                               2.77e-06     6.01076e-06    1.939e-05    6.0405e-06    2.55625e-06      200    
end_path:out1:RootOutput(write)                        0.0114189     0.243644       1.64977      0.211939      0.224272        200    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 22290.2 MB
  Peak resident set size usage (VmHWM): 2848.92 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1794_207_20231028T042641Z_gen_g4_detsim_hitreco__20240503T044625Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1794_207_20231028T042641Z_gen_g4_detsim_hitreco__20240503T044625Z_reco2_reco_2024-11-23T_034319Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1794_207_20231028T042641Z_gen_g4_detsim_hitreco__20240503T044625Z_reco2_reco_data_2024-11-23T_034319Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work
justIN time: 2024-11-23 15:02:06 UTC       justIN version: 01.01.09