justIN           Dashboard       Workflows       Jobs       AWT       Sites       Storages       Docs       Login

Jobsub ID 79737.168@justin-prod-sched02.dune.hep.ac.uk

Jobsub ID79737.168@justin-prod-sched02.dune.hep.ac.uk
Workflow ID3517
Stage ID1
User namecalcuttj@fnal.gov
HTCondor Groupgroup_dune.prod_mcsim
RequestedProcessors1
RSS bytes4193255424 (3999 MiB)
Wall seconds limit18000 (5 hours)
Submitted time2024-09-27 02:58:33
SiteCA_Victoria
EntryDUNE_CA_Victoria_dune-condor_whole
Last heartbeat2024-09-27 13:16:57
From worker nodeHostnamedune--heprc--3456932580--41021190986552-2
cpuinfoIntel Core Processor (Broadwell, IBRS)
OS releaseScientific Linux release 7.9 (Nitrogen)
Processors1
RSS bytes4193255424 (3999 MiB)
Wall seconds limit171000 (47 hours)
Inner Apptainer?True
Job statejobscript_error
Allocator namejustin-allocator-pro.dune.hep.ac.uk
Started2024-09-27 07:17:41
Input fileshd-protodune:np04hd_raw_run027398_0393_dataflow2_datawriter_0_20240624T024347.hdf5
JobscriptExit code137
Real time0m (0s)
CPU time0m (0s = 0%)
Outputting started 
Output files
Finished2024-09-27 13:16:57
Saved logsjustin-logs:79737.168-justin-prod-sched02.dune.hep.ac.uk.logs.tgz
List job events     Wrapper job log

Jobscript log (last 10,000 characters)

FrameSaver: q=1.83638e+06 n=804850 tag=wiener
4632 hits in TPC 0
3920 hits in TPC 1
7355 hits in TPC 2
3514 hits in TPC 3
3899 hits in TPC 4
5085 hits in TPC 5
3816 hits in TPC 6
2888 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 3723 XUs and 2800 XVs -> 102 XUVs
C:0 T:2 24968 XUs and 23980 XVs -> 815 XUVs
C:0 T:3 145 XUs and 149 XVs -> 6 XUVs
C:0 T:4 136 XUs and 178 XVs -> 4 XUVs
C:0 T:5 12662 XUs and 16625 XVs -> 466 XUVs
C:0 T:6 10521 XUs and 14922 XVs -> 477 XUVs
C:0 T:7 29 XUs and 37 XVs -> 0 XUVs
1870 XUVs total
1130 collection wire objects
1870 potential space points
Neighbour search...
56944 tests to find 28558 neighbours
Iterating with no regularization...
Begin: 1.78099e+07
0 1.67738e+07
1 1.67033e+07
2 1.66952e+07
Now with regularization...
Begin: 1.58146e+07
0 1.58121e+07
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 0.00303096 6.6936e-05 6.69956e-05 6.6936e-05 6.6936e-05 6.6936e-05 0.000441313 0.00011003 0.000109971 0.00011003 0.00011003 0.000109971 0.00011003 
----view 1: 4.66108e-05 4.66108e-05 4.66704e-05 4.66108e-05 4.66108e-05 4.66108e-05 4.66108e-05 4.66108e-05 4.66704e-05 4.66108e-05 4.66108e-05 4.66108e-05 4.66108e-05 
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 6.6936e-05 6.69956e-05 6.6936e-05 6.6936e-05 6.6936e-05 0.000441313 0.00011003 0.000109971 0.00011003 0.00011003 0.000109971 0.00011003 0.00011003 
----view 1: 4.66108e-05 4.66704e-05 4.66108e-05 4.66108e-05 4.66108e-05 4.66108e-05 4.66108e-05 4.66704e-05 4.66108e-05 4.66108e-05 4.66108e-05 4.66108e-05 0.00274926 
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 0.00303096 6.6936e-05 6.69956e-05 6.6936e-05 6.6936e-05 6.6936e-05 0.000441313 0.00011003 0.000109971 0.00011003 0.00011003 0.000109971 0.00011003 
----view 1: 4.66108e-05 4.66108e-05 4.66704e-05 4.66108e-05 4.66108e-05 4.66108e-05 4.66108e-05 4.66108e-05 4.66704e-05 4.66108e-05 4.66108e-05 4.66108e-05 4.66108e-05 
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 6.6936e-05 6.69956e-05 6.6936e-05 6.6936e-05 6.6936e-05 0.000441313 0.00011003 0.000109971 0.00011003 0.00011003 0.000109971 0.00011003 0.00011003 
----view 1: 4.66108e-05 4.66704e-05 4.66108e-05 4.66108e-05 4.66108e-05 4.66108e-05 4.66108e-05 4.66704e-05 4.66108e-05 4.66108e-05 4.66108e-05 4.66108e-05 0.00274926 
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------

27-Sep-2024 12:10:37 UTC  Opened output file with pattern "%ifb_reco_stage2_%tc_keepup.root"
Begin processing the 2nd record. run: 27398 subRun: 1 event: 55026 at 27-Sep-2024 12:13:21 UTC
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 70349 traces, 70349 summary
Retagger: tagging trace set: gauss with 53655 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 53655 traces tagged "gauss"
FrameSaver: q=2.25868e+06 n=1040583 tag=gauss
wclsFrameSaver: saving 70349 traces tagged "wiener"
FrameSaver: q=2.45575e+06 n=950428 tag=wiener
2174 hits in TPC 0
2850 hits in TPC 1
8544 hits in TPC 2
3978 hits in TPC 3
7045 hits in TPC 4
10879 hits in TPC 5
6694 hits in TPC 6
4407 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 7968 XUs and 6955 XVs -> 134 XUVs
C:0 T:2 29995 XUs and 28304 XVs -> 619 XUVs
C:0 T:3 180 XUs and 183 XVs -> 10 XUVs
C:0 T:4 3433 XUs and 1575 XVs -> 87 XUVs
C:0 T:5 218084 XUs and 86515 XVs -> 11097 XUVs
C:0 T:6 37836 XUs and 20825 XVs -> 573 XUVs
C:0 T:7 1272 XUs and 793 XVs -> 18 XUVs
12538 XUVs total
2455 collection wire objects
12538 potential space points
Neighbour search...
2454276 tests to find 780662 neighbours
Iterating with no regularization...
Begin: 1.94706e+08
0 1.21767e+08
1 1.14687e+08
2 1.13608e+08
3 1.13238e+08
4 1.13086e+08
5 1.13001e+08
Now with regularization...
Begin: 1.07004e+08
0 1.06806e+08
1 1.06727e+08
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 0.0556183 0.0597687 0.0159206 0.0504627 0.0078249 0.00053978 0.00896263 0.000568748 0.00145423 0.000359237 0.000990927 
----view 1: 0.00023371 0.00023365 0.000227511 0.000172794 0.000172734 8.67844e-05 0.000209451 0.000305533 0.00027895 5.8651e-05 5.87106e-05 
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 0.0597687 0.0159206 0.0504627 0.0078249 0.00053978 0.00896263 0.000568748 0.00145423 0.000359237 0.000990927 0.000746131 
----view 1: 0.00023365 0.000227511 0.000172794 0.000172734 8.67844e-05 0.000209451 0.000305533 0.00027895 5.8651e-05 5.87106e-05 5.8651e-05 
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 0.0159206 0.0504627 0.0078249 0.00053978 0.00896263 0.000568748 0.00145423 0.000359237 0.000990927 0.000746131 0.00271475 
----view 1: 0.000227511 0.000172794 0.000172734 8.67844e-05 0.000209451 0.000305533 0.00027895 5.8651e-05 5.87106e-05 5.8651e-05 5.8651e-05 
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 0.0504627 0.0078249 0.00053978 0.00896263 0.000568748 0.00145423 0.000359237 0.000990927 0.000746131 0.00271475 0.00274396 
----view 1: 0.000172794 0.000172734 8.67844e-05 0.000209451 0.000305533 0.00027895 5.8651e-05 5.87106e-05 5.8651e-05 5.8651e-05 4.40478e-05 
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 0.0078249 0.00053978 0.00896263 0.000568748 0.00145423 0.000359237 0.000990927 0.000746131 0.00271475 0.00274396 0.000703096 
----view 1: 0.000172734 8.67844e-05 0.000209451 0.000305533 0.00027895 5.8651e-05 5.87106e-05 5.8651e-05 5.8651e-05 4.40478e-05 0 
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 0.00053978 0.00896263 0.000568748 0.00145423 0.000359237 0.000990927 0.000746131 0.00271475 0.00274396 0.000703096 0.000702977 
----view 1: 8.67844e-05 0.000209451 0.000305533 0.00027895 5.8651e-05 5.87106e-05 5.8651e-05 5.8651e-05 4.40478e-05 0 0 
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 0.000511885 0.000693798 0.000837326 0.000836849 0.000837088 0.000837088 0.000837326 0.000837088 0.000836849 0.000837088 0.000837326 
----view 1: 0.0948029 0.0447426 0.0125008 0.0261612 0.0186386 0.0356216 0.055975 0.00653648 0.00654221 0.0042038 0.00517654 
TwoViewTransverseTracksAlgorithm: failed to calculate correlation coefficient p-value for these numbers
----view 0: 0.000693798 0.000837326 0.000836849 0.000837088 0.000837088 0.000837326 0.000837088 0.000836849 0.000837088 0.000837326 0.000837088 
----view 1: 0.0447426 0.0125008 0.0261612 0.0186386 0.0356216 0.055975 0.00653648 0.00654221 0.0042038 0.00517654 0.00577927 
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------

Begin processing the 3rd record. run: 27398 subRun: 1 event: 55030 at 27-Sep-2024 12:50:01 UTC
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
Retagger: tagging trace set: wiener with 64599 traces, 64599 summary
Retagger: tagging trace set: gauss with 48711 traces, 0 summary
wclsFrameSaver saving cooked to 5859 ticks
wclsFrameSaver: saving 48711 traces tagged "gauss"
FrameSaver: q=1.85855e+06 n=895941 tag=gauss
wclsFrameSaver: saving 64599 traces tagged "wiener"
FrameSaver: q=2.01414e+06 n=819987 tag=wiener
3253 hits in TPC 0
2893 hits in TPC 1
4311 hits in TPC 2
2753 hits in TPC 3
5722 hits in TPC 4
9719 hits in TPC 5
5227 hits in TPC 6
2817 hits in TPC 7
0 X, 0 U, 0 V bad channels
Finding XUV coincidences...
C:0 T:1 4937 XUs and 2792 XVs -> 144 XUVs
C:0 T:2 12896 XUs and 9675 XVs -> 372 XUVs
C:0 T:3 78 XUs and 91 XVs -> 4 XUVs
C:0 T:4 327 XUs and 333 XVs -> 26 XUVs
C:0 T:5 32970 XUs and 38587 XVs -> 2204 XUVs
C:0 T:6 15117 XUs and 16100 XVs -> 579 XUVs
C:0 T:7 390 XUs and 322 XVs -> 14 XUVs
3343 XUVs total
1325 collection wire objects
3343 potential space points
Neighbour search...
107035 tests to find 62098 neighbours
Iterating with no regularization...
Begin: 2.81585e+07
0 2.53778e+07
1 2.51231e+07
2 2.50894e+07
3 2.50784e+07
Now with regularization...
Begin: 2.32737e+07
0 2.32598e+07
BdtBeamParticleIdTool::SliceFeatures::GetLeadingCaloHits - empty calo hit list
BdtBeamParticleIdTool::SliceFeatures::GetLArTPCIntercepts - inconsistent number of intercepts between a line and the LArTPC
---INTERPRETED-MATCHING-OUTPUT------------------------------------------------------------------
---SUMMARY--------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------

Begin processing the 4th record. run: 27398 subRun: 1 event: 55034 at 27-Sep-2024 13:15:07 UTC
RawFrameSource: got 10240 raw::RawDigit objects
	input nticks=5859 keeping as is
../justin-jobscript: line 164:  1361 Killed                  lar -c ${FCL2} $output_stage1_file
Error in reco2
justIN time: 2024-09-29 09:25:52 UTC       justIN version: 01.01.08