justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 108690.31@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID108690.31@justin-prod-sched02.dune.hep.ac.uk
Workflow ID4213
Stage ID1
User nameismerio@fnal.gov
HTCondor Groupgroup_dune
RequestedProcessors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit80000 (22 hours)
Submitted time2024-11-23 02:34:28
SiteUK_Lancaster
EntryUBoone_UK_Lancaster_HEC_grendel_ce02
Last heartbeat2024-11-23 04:57:30
From worker nodeHostnamecomp23-29
cpuinfoIntel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4194304000 (4000 MiB)
Wall seconds limit257400 (71 hours)
Inner Apptainer?True
Job stateoutputting_failed
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-11-23 02:41:39
Input filesfardet-hd:atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1499_330_20231028T023200Z_gen_g4_detsim_hitreco__20240503T050905Z_reco2.root
JobscriptExit code0
Real time2h (8134s)
CPU time2h (7375s = 90%)
Outputting started2024-11-23 04:57:13
Output files
Finished2024-11-23 04:57:30
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

re
%MSG
%MSG-e WireIDIntersectionCheck:  PMAlgTrajFitter:pmtrajfittc@BeginModule  23-Nov-2024 04:56:25 GMT run: 1499 subRun: 1 event: 66200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 04:57:06 GMT run: 1499 subRun: 1 event: 66200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 04:57:06 GMT run: 1499 subRun: 1 event: 66200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 04:57:06 GMT run: 1499 subRun: 1 event: 66200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 04:57:06 GMT run: 1499 subRun: 1 event: 66200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 04:57:06 GMT run: 1499 subRun: 1 event: 66200
Comparing two wires in the same plane: return failure
%MSG
%MSG-e WireIDIntersectionCheck:  EMShower:emshower@BeginModule  23-Nov-2024 04:57:06 GMT run: 1499 subRun: 1 event: 66200
Comparing two wires in the same plane: return failure
%MSG
Boundary wire vector sizes: 1691, 1688, 1382
minwire 0: 404
minwire 1: 1603
minwire 2: 285
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Used alternate method to get min and max tdcs due to vertex determination failure: 0, 199
Classifier summary: 
Output 0: 0.00485053, 0.984738, 0.0104114, 
Output 1: 0.18212, 0.279627, 0.228276, 0.309978, 
Output 2: 0.055826, 0.944174, 

717
***Wiener Filter*******ApplyPostfilter***
Running Ophitfinder with InputDigiType = 'recob'
Found hits: 2569!
23-Nov-2024 04:57:12 GMT  Closed output file "atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1499_330_20231028T023200Z_gen_g4_detsim_hitreco__20240503T050905Z_reco2_reco_data_2024-11-23T_024143Z.root"
23-Nov-2024 04:57:12 GMT  Closed input file "root://xrootd.echo.stfc.ac.uk:1094/dune:/protodune/RSE/fardet-hd/18/66/atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1499_330_20231028T023200Z_gen_g4_detsim_hitreco__20240503T050905Z_reco2.root"

========================================================================================================================================
TimeTracker printout (sec)                                Min           Avg           Max         Median          RMS         nEvts   
========================================================================================================================================
Full event                                             0.769691       37.4267       838.384        10.9         94.2083        200    
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read)                                 0.0198438     0.122308      0.844909      0.0844773     0.107695        200    
reco:linecluster:LineCluster                           0.137073       0.96926       56.4741      0.326563       5.15204        200    
reco:trajcluster:TrajCluster                           0.0318602      1.91387       111.546      0.143858       9.94024        200    
reco:pandora:StandardPandora                           0.104396       3.06634       118.051       1.27643       10.3314        200    
reco:pandoraTrack:LArPandoraTrackCreation             0.000141084   0.00882633     0.209347     0.00223699     0.022163        200    
reco:pandoraShower:LArPandoraModularShowerCreation    0.000174898    0.0123993     0.174634     0.00298982     0.0249115       200    
reco:pandoracalo:Calorimetry                          0.000103551    0.0388113      2.40405     0.00100999     0.225248        200    
reco:pandorapid:Chi2ParticleID                        3.9309e-05    0.000168714   0.00126481    0.000127791   0.000150975      200    
reco:trkshowersplit:TrackShowerHits                   0.000110476    0.0313952      1.29911     0.000688766    0.137133        200    
reco:pmtrack:PMAlgTrackMaker                          0.00124222      2.84889       142.569      0.170692       12.0776        200    
reco:pmtrackcalo:Calorimetry                          8.4185e-05     0.0152014     0.497626     0.000967784    0.0550384       200    
reco:pmtrackpid:Chi2ParticleID                        3.4149e-05    0.000313181   0.00325465    0.000138872   0.000482239      200    
reco:pmtrajfit:PMAlgTrajFitter                        0.000191288     2.70251       115.399      0.311565       11.5282        200    
reco:pmtrajfitcalo:Calorimetry                        6.5941e-05     0.0400017      2.38385      0.0013306     0.226514        200    
reco:pmtrajfitpid:Chi2ParticleID                      2.8612e-05    0.000200176   0.00152248    0.000144984   0.000185492      200    
reco:pmtracktc:PMAlgTrackMaker                         0.0011098      3.08432       93.4945      0.202287       10.8325        200    
reco:pmtracktccalo:Calorimetry                        7.8513e-05    0.00974115     0.236202     0.000981462    0.0285785       200    
reco:pmtracktcpid:Chi2ParticleID                      3.0896e-05    0.000280354   0.00234812    0.000140153   0.000360997      200    
reco:pmtrajfittc:PMAlgTrajFitter                      0.000186256     1.71475       39.9216      0.280823       4.52539        200    
reco:pmtrajfittccalo:Calorimetry                      6.7095e-05     0.0240125     0.933193     0.00119746     0.0941371       200    
reco:pmtrajfittcpid:Chi2ParticleID                    2.8059e-05    0.000391308   0.00467457    0.000158676   0.000620443      200    
reco:blurredcluster:BlurredClustering                 8.2114e-05      1.20963       10.867       0.547998       1.72082        200    
reco:emtrkmichelid:EmTrackMichelId                    0.000142886     15.7814       232.996       3.58278       31.3392        200    
reco:emshower:EMShower                                0.000208159    0.0256471     0.311977     0.00502641     0.0458562       200    
reco:cvnmap:CVNMapper                                 2.1246e-05     0.016074      0.0635796     0.015121      0.0119103       200    
reco:cvneva:CVNEvaluator                               1.556e-05     0.649559       1.64625      0.811443       0.37222        200    
reco:energyrecnumu:EnergyReco                         0.00031335     0.0647203      3.60788     0.00886434     0.307799        200    
reco:energyrecnue:EnergyReco                          0.000143749   0.00286446     0.0684852    0.000551426   0.00751807       200    
reco:energyrecnc:EnergyReco                           0.00196036    0.00466093     0.0717169    0.00231614    0.00765724       200    
reco:energyrecnumurange:EnergyReco                    0.000226535    0.0603069      3.57549     0.00226456     0.305965        200    
reco:energyrecnumumcs:EnergyReco                      0.00118264     0.0616255      3.60224     0.00321911     0.307134        200    
reco:opdec:Deconvolution                               0.0735575     0.388375       1.21866      0.375139       0.14763        200    
reco:ophitspe:OpHitFinderDeco                           0.22421       2.31492       4.35454       2.25484      0.894267        200    
reco:opflash:OpFlashFinder                            0.000275647   0.00147116    0.00585822    0.00139873    0.000710956      200    
reco:rns:RandomNumberSaver                             1.983e-05    3.4051e-05    0.000267952   3.19075e-05   2.07335e-05      200    
reco:anglereconue:NuAngularReco                       0.000150296   0.00210106     0.0512189    0.000392639   0.00561672       200    
reco:anglereconumu:NuAngularReco                      7.1357e-05    0.00189203     0.0500252    0.000205639   0.00576007       200    
reco:anglereconuepfps:NuAngularReco                   0.000120057   0.00197074     0.051018     0.000318567   0.00555698       200    
reco:anglereconumupfps:NuAngularReco                  0.000101265    0.0019239     0.0508916    0.000260846    0.0055617       200    
[art]:TriggerResults:TriggerResultInserter             8.65e-06     1.13945e-05   3.8404e-05    1.0492e-05    3.34959e-06      200    
end_path:out1:RootOutput                               2.569e-06    5.45434e-06   1.9492e-05    4.9755e-06    2.72929e-06      200    
end_path:out1:RootOutput(write)                        0.0105068     0.231674       1.19069      0.212562      0.169276        200    
========================================================================================================================================

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 9796.71 MB
  Peak resident set size usage (VmHWM): 2628 MB
====================================================================================================
Art has completed and will exit with status 0.
=== End last 100 lines of lar log file ===
Validation_ccnc_atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1499_330_20231028T023200Z_gen_g4_detsim_hitreco__20240503T050905Z_reco2.root
all-input-dids.txt
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1499_330_20231028T023200Z_gen_g4_detsim_hitreco__20240503T050905Z_reco2_reco_2024-11-23T_024143Z.log
atmnu_max_weighted_randompolicy_dune10kt_1x2x6_1499_330_20231028T023200Z_gen_g4_detsim_hitreco__20240503T050905Z_reco2_reco_data_2024-11-23T_024143Z.root
debugprod.log
jobscript.log
justin-processed-pfns.txt
reco2_hist.root
MyPandoraSettings_Master_Atmos_DUNEFD.xml
MyPandoraSettings_Master_DUNEFD.xml
build_slf7.x86_64
localProducts_larsoft_v09_91_02_e26_prof
setup_env-testreco.sh
srcs
temp.txt
temp2.txt
test
work
justIN time: 2024-11-23 11:31:59 UTC       justIN version: 01.01.09