Jobsub ID 79737.168@justin-prod-sched02.dune.hep.ac.uk
Jobsub ID | 79737.168@justin-prod-sched02.dune.hep.ac.uk |
Workflow ID | 3517 |
Stage ID | 1 |
User name | calcuttj@fnal.gov |
HTCondor Group | group_dune.prod_mcsim |
Requested | Processors | 1 |
RSS bytes | 4193255424 (3999 MiB) |
Wall seconds limit | 18000 (5 hours) |
Submitted time | 2024-09-27 02:58:33 |
Site | CA_Victoria |
Entry | DUNE_CA_Victoria_dune-condor_whole |
Last heartbeat | 2024-09-27 13:16:57 |
From worker node | Hostname | dune--heprc--3456932580--41021190986552-2 |
cpuinfo | Intel Core Processor (Broadwell, IBRS) |
OS release | Scientific Linux release 7.9 (Nitrogen) |
Processors | 1 |
RSS bytes | 4193255424 (3999 MiB) |
Wall seconds limit | 171000 (47 hours) |
Inner Apptainer? | True |
Job state | jobscript_error |
Allocator name | justin-allocator-pro.dune.hep.ac.uk |
Started | 2024-09-27 07:17:41 |
Input files | hd-protodune:np04hd_raw_run027398_0393_dataflow2_datawriter_0_20240624T024347.hdf5
|
Jobscript | Exit code | 137 |
Real time | 0m (0s) |
CPU time | 0m (0s = 0%) |
Outputting started | |
Output files | |
Finished | 2024-09-27 13:16:57 |
Saved logs | justin-logs:79737.168-justin-prod-sched02.dune.hep.ac.uk.logs.tgz |
List job events Wrapper job log |
Jobscript log (last 10,000 characters)
FrameSaver: q=1.83638e+06 n=804850 tag=wiener
4632 hits in TPC 0
3920 hits in TPC 1
7355 hits in TPC 2
3514 hits in TPC 3
3899 hits in TPC 4
5085 hits in TPC 5
3816 hits in TPC 6
2888 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 3723 XUs and 2800 XVs -> 102 XUVs
C:0 T:2 24968 XUs and 23980 XVs -> 815 XUVs
C:0 T:3 145 XUs and 149 XVs -> 6 XUVs
C:0 T:4 136 XUs and 178 XVs -> 4 XUVs
C:0 T:5 12662 XUs and 16625 XVs -> 466 XUVs
C:0 T:6 10521 XUs and 14922 XVs -> 477 XUVs
C:0 T:7 29 XUs and 37 XVs -> 0 XUVs
1870 XUVs total
1130 collection wire objects
1870 potential space points
Neighbour search...
56944 tests to find 28558 neighbours
Iterating with no regularization...
Begin: 1.78099e+07
0 1.67738e+07
1 1.67033e+07
2 1.66952e+07
Now with regularization...
Begin: 1.58146e+07
0 1.58121e+07
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 0.00303096 6.6936e-05 6.69956e-05 6.6936e-05 6.6936e-05 6.6936e-05 0.000441313 0.00011003 0.000109971 0.00011003 0.00011003 0.000109971 0.00011003
----view 1: 4.66108e-05 4.66108e-05 4.66704e-05 4.66108e-05 4.66108e-05 4.66108e-05 4.66108e-05 4.66108e-05 4.66704e-05 4.66108e-05 4.66108e-05 4.66108e-05 4.66108e-05
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 6.6936e-05 6.69956e-05 6.6936e-05 6.6936e-05 6.6936e-05 0.000441313 0.00011003 0.000109971 0.00011003 0.00011003 0.000109971 0.00011003 0.00011003
----view 1: 4.66108e-05 4.66704e-05 4.66108e-05 4.66108e-05 4.66108e-05 4.66108e-05 4.66108e-05 4.66704e-05 4.66108e-05 4.66108e-05 4.66108e-05 4.66108e-05 0.00274926
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 0.00303096 6.6936e-05 6.69956e-05 6.6936e-05 6.6936e-05 6.6936e-05 0.000441313 0.00011003 0.000109971 0.00011003 0.00011003 0.000109971 0.00011003
----view 1: 4.66108e-05 4.66108e-05 4.66704e-05 4.66108e-05 4.66108e-05 4.66108e-05 4.66108e-05 4.66108e-05 4.66704e-05 4.66108e-05 4.66108e-05 4.66108e-05 4.66108e-05
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 6.6936e-05 6.69956e-05 6.6936e-05 6.6936e-05 6.6936e-05 0.000441313 0.00011003 0.000109971 0.00011003 0.00011003 0.000109971 0.00011003 0.00011003
----view 1: 4.66108e-05 4.66704e-05 4.66108e-05 4.66108e-05 4.66108e-05 4.66108e-05 4.66108e-05 4.66704e-05 4.66108e-05 4.66108e-05 4.66108e-05 4.66108e-05 0.00274926
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
27-Sep-2024 12:10:37 UTC Opened output file with pattern "%ifb_reco_stage2_%tc_keepup.root"
Begin processing the 2nd record. run: 27398 subRun: 1 event: 55026 at 27-Sep-2024 12:13:21 UTC
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 70349 traces, 70349 summary
Retagger: tagging trace set: gauss with 53655 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 53655 traces tagged "gauss"
FrameSaver: q=2.25868e+06 n=1040583 tag=gauss
wclsFrameSaver: saving 70349 traces tagged "wiener"
FrameSaver: q=2.45575e+06 n=950428 tag=wiener
2174 hits in TPC 0
2850 hits in TPC 1
8544 hits in TPC 2
3978 hits in TPC 3
7045 hits in TPC 4
10879 hits in TPC 5
6694 hits in TPC 6
4407 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 7968 XUs and 6955 XVs -> 134 XUVs
C:0 T:2 29995 XUs and 28304 XVs -> 619 XUVs
C:0 T:3 180 XUs and 183 XVs -> 10 XUVs
C:0 T:4 3433 XUs and 1575 XVs -> 87 XUVs
C:0 T:5 218084 XUs and 86515 XVs -> 11097 XUVs
C:0 T:6 37836 XUs and 20825 XVs -> 573 XUVs
C:0 T:7 1272 XUs and 793 XVs -> 18 XUVs
12538 XUVs total
2455 collection wire objects
12538 potential space points
Neighbour search...
2454276 tests to find 780662 neighbours
Iterating with no regularization...
Begin: 1.94706e+08
0 1.21767e+08
1 1.14687e+08
2 1.13608e+08
3 1.13238e+08
4 1.13086e+08
5 1.13001e+08
Now with regularization...
Begin: 1.07004e+08
0 1.06806e+08
1 1.06727e+08
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 0.0556183 0.0597687 0.0159206 0.0504627 0.0078249 0.00053978 0.00896263 0.000568748 0.00145423 0.000359237 0.000990927
----view 1: 0.00023371 0.00023365 0.000227511 0.000172794 0.000172734 8.67844e-05 0.000209451 0.000305533 0.00027895 5.8651e-05 5.87106e-05
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 0.0597687 0.0159206 0.0504627 0.0078249 0.00053978 0.00896263 0.000568748 0.00145423 0.000359237 0.000990927 0.000746131
----view 1: 0.00023365 0.000227511 0.000172794 0.000172734 8.67844e-05 0.000209451 0.000305533 0.00027895 5.8651e-05 5.87106e-05 5.8651e-05
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 0.0159206 0.0504627 0.0078249 0.00053978 0.00896263 0.000568748 0.00145423 0.000359237 0.000990927 0.000746131 0.00271475
----view 1: 0.000227511 0.000172794 0.000172734 8.67844e-05 0.000209451 0.000305533 0.00027895 5.8651e-05 5.87106e-05 5.8651e-05 5.8651e-05
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 0.0504627 0.0078249 0.00053978 0.00896263 0.000568748 0.00145423 0.000359237 0.000990927 0.000746131 0.00271475 0.00274396
----view 1: 0.000172794 0.000172734 8.67844e-05 0.000209451 0.000305533 0.00027895 5.8651e-05 5.87106e-05 5.8651e-05 5.8651e-05 4.40478e-05
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 0.0078249 0.00053978 0.00896263 0.000568748 0.00145423 0.000359237 0.000990927 0.000746131 0.00271475 0.00274396 0.000703096
----view 1: 0.000172734 8.67844e-05 0.000209451 0.000305533 0.00027895 5.8651e-05 5.87106e-05 5.8651e-05 5.8651e-05 4.40478e-05 0
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 0.00053978 0.00896263 0.000568748 0.00145423 0.000359237 0.000990927 0.000746131 0.00271475 0.00274396 0.000703096 0.000702977
----view 1: 8.67844e-05 0.000209451 0.000305533 0.00027895 5.8651e-05 5.87106e-05 5.8651e-05 5.8651e-05 4.40478e-05 0 0
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 0.000511885 0.000693798 0.000837326 0.000836849 0.000837088 0.000837088 0.000837326 0.000837088 0.000836849 0.000837088 0.000837326
----view 1: 0.0948029 0.0447426 0.0125008 0.0261612 0.0186386 0.0356216 0.055975 0.00653648 0.00654221 0.0042038 0.00517654
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 0.000693798 0.000837326 0.000836849 0.000837088 0.000837088 0.000837326 0.000837088 0.000836849 0.000837088 0.000837326 0.000837088
----view 1: 0.0447426 0.0125008 0.0261612 0.0186386 0.0356216 0.055975 0.00653648 0.00654221 0.0042038 0.00517654 0.00577927
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 3rd record. run: 27398 subRun: 1 event: 55030 at 27-Sep-2024 12:50:01 UTC
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 64599 traces, 64599 summary
Retagger: tagging trace set: gauss with 48711 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 48711 traces tagged "gauss"
FrameSaver: q=1.85855e+06 n=895941 tag=gauss
wclsFrameSaver: saving 64599 traces tagged "wiener"
FrameSaver: q=2.01414e+06 n=819987 tag=wiener
3253 hits in TPC 0
2893 hits in TPC 1
4311 hits in TPC 2
2753 hits in TPC 3
5722 hits in TPC 4
9719 hits in TPC 5
5227 hits in TPC 6
2817 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 4937 XUs and 2792 XVs -> 144 XUVs
C:0 T:2 12896 XUs and 9675 XVs -> 372 XUVs
C:0 T:3 78 XUs and 91 XVs -> 4 XUVs
C:0 T:4 327 XUs and 333 XVs -> 26 XUVs
C:0 T:5 32970 XUs and 38587 XVs -> 2204 XUVs
C:0 T:6 15117 XUs and 16100 XVs -> 579 XUVs
C:0 T:7 390 XUs and 322 XVs -> 14 XUVs
3343 XUVs total
1325 collection wire objects
3343 potential space points
Neighbour search...
107035 tests to find 62098 neighbours
Iterating with no regularization...
Begin: 2.81585e+07
0 2.53778e+07
1 2.51231e+07
2 2.50894e+07
3 2.50784e+07
Now with regularization...
Begin: 2.32737e+07
0 2.32598e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLArTPCIntercepts - inconsistent number of intercepts between a line and the LArTPC
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------
Begin processing the 4th record. run: 27398 subRun: 1 event: 55034 at 27-Sep-2024 13:15:07 UTC
RawFrameSource: got 10240 raw::RawDigit objects
input nticks=5859 keeping as is
../justin-jobscript: line 164: 1361 Killed lar -c ${FCL2} $output_stage1_file
Error in reco2