Jobsub ID 263290.58@justin-prod-sched01.dune.hep.ac.uk
| Jobsub ID | 263290.58@justin-prod-sched01.dune.hep.ac.uk |
| Workflow ID | 3461 |
| Stage ID | 1 |
| User name | calcuttj@fnal.gov |
| HTCondor Group | group_dune.prod_mcsim |
| Requested | Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 18000 (5 hours) |
| Submitted time | 2024-09-26 09:57:08 |
| Site | UK_Sheffield |
| Entry | DUNE_UK_Sheffield_lcgce1 |
| Last heartbeat | 2024-09-26 12:06:13 |
| From worker node | Hostname | wn023.hep |
| cpuinfo | Intel(R) Xeon(R) Gold 5115 CPU @ 2.40GHz |
| OS release | Scientific Linux release 7.9 (Nitrogen) |
| Processors | 1 |
| RSS bytes | 4193255424 (3999 MiB) |
| Wall seconds limit | 257400 (71 hours) |
| Inner Apptainer? | True |
| Job state | jobscript_error |
| Allocator name | justin-allocator-pro.dune.hep.ac.uk |
| Started | 2024-09-26 09:59:31 |
| Input files | hd-protodune:np04hd_raw_run027298_0119_dataflow3_datawriter_0_20240619T144048.hdf5
|
| Jobscript | Exit code | 1 |
| Real time | 0m (0s) |
| CPU time | 0m (0s = 0%) |
| Outputting started | |
| Output files | |
| Finished | 2024-09-26 12:06:13 |
| Saved logs | justin-logs:263290.58-justin-prod-sched01.dune.hep.ac.uk.logs.tgz |
| List job events Wrapper job log |
Jobscript log (last 10,000 characters)
FrameSaver: q=1.61843e+06 n=829802 tag=gauss
wclsFrameSaver: saving 55548 traces tagged "wiener"
FrameSaver: q=1.72309e+06 n=759041 tag=wiener
3376 hits in TPC 0
3640 hits in TPC 1
4794 hits in TPC 2
2637 hits in TPC 3
3810 hits in TPC 4
5086 hits in TPC 5
5279 hits in TPC 6
4603 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 4169 XUs and 4052 XVs -> 128 XUVs
C:0 T:2 10430 XUs and 10599 XVs -> 955 XUVs
C:0 T:3 151 XUs and 260 XVs -> 9 XUVs
C:0 T:4 27 XUs and 32 XVs -> 0 XUVs
C:0 T:5 10899 XUs and 12571 XVs -> 374 XUVs
C:0 T:6 15649 XUs and 14431 XVs -> 396 XUVs
C:0 T:7 522 XUs and 441 XVs -> 8 XUVs
1870 XUVs total
971 collection wire objects
1870 potential space points
Neighbour search...
26362 tests to find 14834 neighbours
Iterating with no regularization...
Begin: 2.49129e+07
0 2.40859e+07
1 2.40408e+07
2 2.40374e+07
Now with regularization...
Begin: 2.2626e+07
0 2.26215e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 34th record. run: 27298 subRun: 1 event: 16795 at 26-Sep-2024 12:42:45 BST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 32882 traces, 32882 summary
Retagger: tagging trace set: gauss with 25282 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 25282 traces tagged "gauss"
FrameSaver: q=856860 n=482791 tag=gauss
wclsFrameSaver: saving 32882 traces tagged "wiener"
FrameSaver: q=918769 n=440042 tag=wiener
2286 hits in TPC 0
2558 hits in TPC 1
1482 hits in TPC 2
1247 hits in TPC 3
3533 hits in TPC 4
6768 hits in TPC 5
1828 hits in TPC 6
1124 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 543 XUs and 560 XVs -> 12 XUVs
C:0 T:2 2967 XUs and 1342 XVs -> 31 XUVs
C:0 T:3 541 XUs and 222 XVs -> 39 XUVs
C:0 T:4 495 XUs and 465 XVs -> 15 XUVs
C:0 T:5 18663 XUs and 15097 XVs -> 736 XUVs
C:0 T:6 1375 XUs and 1481 XVs -> 140 XUVs
C:0 T:7 102 XUs and 114 XVs -> 10 XUVs
983 XUVs total
566 collection wire objects
983 potential space points
Neighbour search...
17731 tests to find 10670 neighbours
Iterating with no regularization...
Begin: 7.9327e+06
0 7.30528e+06
1 7.26086e+06
2 7.25458e+06
Now with regularization...
Begin: 6.48831e+06
0 6.48735e+06
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 35th record. run: 27298 subRun: 1 event: 16799 at 26-Sep-2024 12:44:24 BST
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 27731 traces, 27731 summary
Retagger: tagging trace set: gauss with 21554 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 21554 traces tagged "gauss"
FrameSaver: q=684954 n=392452 tag=gauss
wclsFrameSaver: saving 27731 traces tagged "wiener"
FrameSaver: q=749081 n=359371 tag=wiener
1406 hits in TPC 0
2754 hits in TPC 1
1145 hits in TPC 2
721 hits in TPC 3
2109 hits in TPC 4
3271 hits in TPC 5
2860 hits in TPC 6
2989 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 9155 XUs and 2158 XVs -> 119 XUVs
C:0 T:2 1468 XUs and 993 XVs -> 47 XUVs
C:0 T:3 26 XUs and 14 XVs -> 0 XUVs
C:0 T:4 151 XUs and 149 XVs -> 9 XUVs
C:0 T:5 6946 XUs and 5021 XVs -> 329 XUVs
C:0 T:6 38386 XUs and 9527 XVs -> 471 XUVs
C:0 T:7 508 XUs and 338 XVs -> 20 XUVs
995 XUVs total
589 collection wire objects
995 potential space points
Neighbour search...
24805 tests to find 13284 neighbours
Iterating with no regularization...
Begin: 1.62368e+07
0 1.54992e+07
1 1.54525e+07
2 1.54453e+07
Now with regularization...
Begin: 1.51102e+07
0 1.51072e+07
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
26-Sep-2024 12:45:54 BST Closed output file "np04hd_raw_run027298_0119_dataflow3_datawriter_0_20240619T144048_reco_stage1_reco_stage2_20240926T114554_keepup.root"
26-Sep-2024 12:45:54 BST Closed input file "np04hd_raw_run027298_0119_dataflow3_datawriter_0_20240619T144048_reco_stage1.root"
========================================================================================================================================
TimeTracker printout (sec) Min Avg Max Median RMS nEvts
========================================================================================================================================
Full event 90.0127 139.766 212.648 130.385 29.5747 35
----------------------------------------------------------------------------------------------------------------------------------------
source:RootInput(read) 0.000612736 0.00276219 0.00576618 0.00265056 0.00138092 35
produce:fembfilter:PDHDFEMBFilter 0.804433 0.82123 0.954401 0.814329 0.0269293 35
produce:wclsdatahd:WireCellToolkit 45.3985 55.2726 61.2263 55.9902 3.45164 35
produce:gaushit:GausHitFinder 0.401951 0.874577 1.29058 0.831667 0.216961 35
produce:nhitsfilter:NumberOfHitsFilter 0.00027269 0.000464511 0.00080206 0.000447248 0.000102816 35
produce:reco3d:SpacePointSolver 0.137259 1.02891 6.3864 0.736335 1.08319 35
produce:hitpdune:DisambigFromSpacePoints 0.143629 0.665022 1.45273 0.651468 0.308809 35
produce:pandora:StandardPandora 7.25033 17.1719 41.0232 15.5689 7.76297 35
produce:pandoraWriter:StandardPandora 0.0865605 0.175257 0.26534 0.174686 0.0402391 35
produce:pandoraTrack:LArPandoraTrackCreation 0.511377 2.68586 6.25422 2.12667 1.50326 35
produce:pandoraShower:LArPandoraShowerCreation 0.379844 3.0876 8.65929 2.1918 2.17906 35
produce:pandoracalo:Calorimetry 0.231233 1.0644 2.13922 0.925681 0.495857 35
produce:pandoracalonosce:Calorimetry 0.204141 0.997475 1.9874 0.868613 0.4871 35
produce:pandorapid:Chi2ParticleID 0.00119405 0.00236084 0.00353098 0.00223699 0.000512811 35
produce:pandoraShowercalo:ShowerCalorimetry 0.310772 2.50303 5.75646 2.14584 1.40719 35
produce:pandoraShowercalonosce:ShowerCalorimetry 0.301567 2.32779 5.46947 1.92232 1.2869 35
produce:pandoraGnocchiCalo:GnocchiCalorimetry 0.0452218 0.123175 0.187385 0.122166 0.0336437 35
produce:pandoraGnocchiCalonosce:GnocchiCalorimetry 0.0176904 0.0490554 0.0948044 0.0477004 0.0149765 35
produce:emtrkmichelid:EmTrackMichelId 22.4657 48.9656 77.194 47.3492 12.3233 35
[art]:TriggerResults:TriggerResultInserter 3.7126e-05 5.14572e-05 9.8376e-05 4.8711e-05 1.14496e-05 35
end_path:out1:RootOutput 1.0556e-05 1.40585e-05 3.2322e-05 1.2718e-05 4.34556e-06 35
end_path:michelremoving:michelremoving 0.0173428 0.0891102 0.491276 0.0466622 0.101335 35
end_path:michelremovinggnocchi:michelremoving 0.0159903 0.0802026 0.480835 0.0395499 0.0966682 35
end_path:calibanagnocchi:CalibAnaTree 0.164873 0.78127 1.5955 0.678847 0.369169 35
end_path:out1:RootOutput(write) 0.361863 0.832479 1.2639 0.837792 0.204764 35
========================================================================================================================================
====================================================================================================
MemoryTracker summary (base-10 MB units used)
Peak virtual memory usage (VmPeak) : 4996.87 MB
Peak resident set size usage (VmHWM): 2413.3 MB
Details saved in: 'mem.db'
====================================================================================================
Art has completed and will exit with status 0.
Output files:
\tReco: np04hd_raw_run027298_0119_dataflow3_datawriter_0_20240619T144048_reco_stage1_reco_stage2_20240926T114554_keepup.root
\tHists: np04hd_raw_run027298_0119_dataflow3_datawriter_0_20240619T144048_reco_stage1_reco_stage2_20240926T114554_keepup_hists.root
Forming reco metadata
/cvmfs/larsoft.opensciencegrid.org/products/python/v3_9_15/Linux64bit+3.10-2.17/bin/python: No module named meta_maker
Error in reco metadata
