Jobsub ID 263419.12@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 263419.12@justin-prod-sched01.dune.hep.ac.uk |
| Workflow ID | 3447 |
| Stage ID | 1 |
| User name | calcuttj@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2024-09-26 10:39:09 |
| Site | UK_Sheffield |
| Entry | DUNE_UK_Sheffield_lcgce1 |
| Last heartbeat | 2024-09-26 12:34:14 |
| From worker node | Hostname | wn033.hep |
| cpuinfo | Intel(R) Xeon(R) Gold 6226 CPU @ 2.70GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| Inner Apptainer? | True |
| Job state | outputting_failed |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-09-26 10:59:13 |
| Input files | hd-protodune:np04hd_raw_run029081_0484_dataflow4_datawriter_0_20240905T003233.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Outputting started | |
| Output files | |
| Finished | 2024-09-26 12:34:14 |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
ed "wiener"
FrameSaver: q=769287 n=538012 tag=wiener
1763 hits in TPC 0
1392 hits in TPC 1
5048 hits in TPC 2
2616 hits in TPC 3
2589 hits in TPC 4
4856 hits in TPC 5
4301 hits in TPC 6
1918 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 804 XUs and 352 XVs -> 32 XUVs
C:0 T:2 28647 XUs and 86836 XVs -> 1561 XUVs
C:0 T:3 268 XUs and 442 XVs -> 18 XUVs
C:0 T:4 47 XUs and 66 XVs -> 4 XUVs
C:0 T:5 20281 XUs and 13066 XVs -> 537 XUVs
C:0 T:6 15696 XUs and 28121 XVs -> 608 XUVs
C:0 T:7 124 XUs and 200 XVs -> 5 XUVs
2765 XUVs total
993 collection wire objects
2765 potential space points
Neighbour search...
169297 tests to find 92024 neighbours
Iterating with no regularization...
Begin: 7.55273e+06
0 7.14381e+06
1 7.11817e+06
2 7.11373e+06
Now with regularization...
Begin: 6.80943e+06
0 6.80766e+06
BdtBeamParticleIdTool::SliceFeatures::GetLArTPCIntercepts - inconsistent number of intercepts between a line and the LArTPC
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 29th record. run: 29081 subRun: 1 event: 116388 at 26-Sep-2024 13:12:44 BST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 64345 traces, 64345 summary
Retagger: tagging trace set: gauss with 48805 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 48805 traces tagged "gauss"
FrameSaver: q=1.09705e+06 n=922504 tag=gauss
wclsFrameSaver: saving 64345 traces tagged "wiener"
FrameSaver: q=1.20424e+06 n=848566 tag=wiener
3318 hits in TPC 0
4127 hits in TPC 1
7365 hits in TPC 2
4871 hits in TPC 3
2973 hits in TPC 4
4572 hits in TPC 5
4687 hits in TPC 6
2650 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 18336 XUs and 12406 XVs -> 143 XUVs
C:0 T:2 32885 XUs and 43691 XVs -> 3585 XUVs
C:0 T:3 1059 XUs and 624 XVs -> 37 XUVs
C:0 T:4 15 XUs and 36 XVs -> 4 XUVs
C:0 T:5 9687 XUs and 11568 XVs -> 242 XUVs
C:0 T:6 10226 XUs and 10587 XVs -> 286 XUVs
C:0 T:7 152 XUs and 287 XVs -> 4 XUVs
4301 XUVs total
1370 collection wire objects
4301 potential space points
Neighbour search...
235309 tests to find 143578 neighbours
Iterating with no regularization...
Begin: 9.89505e+06
0 9.07051e+06
1 8.97758e+06
2 8.95884e+06
3 8.95256e+06
Now with regularization...
Begin: 8.28469e+06
0 8.28068e+06
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 30th record. run: 29081 subRun: 1 event: 116396 at 26-Sep-2024 13:14:09 BST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 45364 traces, 45364 summary
Retagger: tagging trace set: gauss with 33148 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 33148 traces tagged "gauss"
FrameSaver: q=687844 n=589774 tag=gauss
wclsFrameSaver: saving 45364 traces tagged "wiener"
FrameSaver: q=748585 n=531218 tag=wiener
4043 hits in TPC 0
2964 hits in TPC 1
2365 hits in TPC 2
1471 hits in TPC 3
1220 hits in TPC 4
4080 hits in TPC 5
3399 hits in TPC 6
3168 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 3623 XUs and 2486 XVs -> 131 XUVs
C:0 T:2 7935 XUs and 2498 XVs -> 107 XUVs
C:0 T:3 18 XUs and 27 XVs -> 2 XUVs
C:0 T:4 258 XUs and 166 XVs -> 23 XUVs
C:0 T:5 41296 XUs and 44761 XVs -> 1782 XUVs
C:0 T:6 7069 XUs and 6601 XVs -> 339 XUVs
C:0 T:7 49 XUs and 76 XVs -> 2 XUVs
2386 XUVs total
672 collection wire objects
2386 potential space points
Neighbour search...
148768 tests to find 88442 neighbours
Iterating with no regularization...
Begin: 7.24623e+06
0 6.70119e+06
1 6.6776e+06
2 6.67273e+06
Now with regularization...
Begin: 6.31513e+06
0 6.31357e+06
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
26-Sep-2024 13:15:16 BST Closed output file "np04hd_raw_run029081_0484_dataflow4_datawriter_0_20240905T003233_reco_stage1_reco_stage2_20240926T121516_keepup.root"
26-Sep-2024 13:15:16 BST Closed input file "np04hd_raw_run029081_0484_dataflow4_datawriter_0_20240905T003233_reco_stage1.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 67.2904 90.1799 135.317 87.1216 15.1868 30
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000384875 0.00190024 0.00751258 0.00163183 0.00161092 30
produce:fembfilter:PDHDFEMBFilter 0.599659 0.630524 0.811793 0.619604 0.0430333 30
produce:wclsdatahd:WireCellToolkit 45.0222 47.8175 49.5197 47.8495 1.21473 30
produce:gaushit:GausHitFinder 0.382066 0.680387 0.886907 0.673771 0.12693 30
produce:nhitsfilter:NumberOfHitsFilter 0.000270849 0.000448028 0.00114729 0.0004127 0.000149917 30
produce:reco3d:SpacePointSolver 0.360424 1.19325 2.6572 0.991638 0.623561 30
produce:hitpdune:DisambigFromSpacePoints 0.171494 0.527907 1.02723 0.512198 0.200837 30
produce:pandora:StandardPandora 5.29163 11.4033 23.4468 10.2963 4.05975 30
produce:pandoraWriter:StandardPandora 0.0893941 0.158818 0.233117 0.162992 0.033035 30
produce:pandoraTrack:LArPandoraTrackCreation 0.660904 2.7749 6.09235 2.56419 1.40357 30
produce:pandoraShower:LArPandoraShowerCreation 0.772415 3.20529 8.11385 2.61312 1.86767 30
produce:pandoracalo:Calorimetry 0.314068 0.991937 2.41795 0.897938 0.454949 30
produce:pandoracalonosce:Calorimetry 0.272057 0.971297 2.40353 0.87766 0.472683 30
produce:pandorapid:Chi2ParticleID 0.0010682 0.00178264 0.00408406 0.0017178 0.000524197 30
produce:pandoraShowercalo:ShowerCalorimetry 0.576732 2.55452 6.40668 2.28261 1.35197 30
produce:pandoraShowercalonosce:ShowerCalorimetry 0.591417 2.61827 7.92532 2.21701 1.52951 30
produce:pandoraGnocchiCalo:GnocchiCalorimetry 0.0600945 0.117384 0.174622 0.115361 0.0288013 30
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0277813 0.0484966 0.0730282 0.0470963 0.011758 30
produce:emtrkmichelid:EmTrackMichelId 7.18668 12.615 23.1177 12.3539 3.01384 30
[art]:TriggerResults:TriggerResultInserter 3.505e-05 5.1827e-05 0.000119217 5.0654e-05 1.4551e-05 30
end_path:out1:RootOutput 7.587e-06 1.28886e-05 3.2261e-05 1.21185e-05 4.80757e-06 30
end_path:michelremoving:michelremoving 0.0234415 0.0912624 0.432536 0.0620942 0.0790838 30
end_path:michelremovinggnocchi:michelremoving 0.0219975 0.0868221 0.419902 0.0527512 0.0789235 30
end_path:calibanagnocchi:CalibAnaTree 0.21545 0.745745 1.67919 0.687281 0.355797 30
end_path:out1:RootOutput(write) 0.40633 0.693139 1.20141 0.668582 0.175823 30
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 5851.28 MB
Peak resident set size usage (VmHWM): 2685.87 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np04hd_raw_run029081_0484_dataflow4_datawriter_0_20240905T003233_reco_stage1_reco_stage2_20240926T121516_keepup.root
\tHists: np04hd_raw_run029081_0484_dataflow4_datawriter_0_20240905T003233_reco_stage1_reco_stage2_20240926T121516_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
