An empirical approach for probing the definiteness of kernels

Abstract

Models like support vector machines or Gaussian process regression often require positive semi-definite kernels. These kernels may be based on distance functions. While definiteness is proven for common distances and kernels, a proof for a new kernel may require too much time and effort for users who simply aim at practical usage. Furthermore, designing definite distances or kernels may be equally intricate. Finally, models can be enabled to use indefinite kernels. This may deteriorate the accuracy or computational cost of the model. Hence, an efficient method to determine definiteness is required. We propose an empirical approach. We show that sampling as well as optimization with an evolutionary algorithm may be employed to determine definiteness. We provide a proof of concept with 16 different distance measures for permutations. Our approach allows to disprove definiteness if a respective counterexample is found. It can also provide an estimate of how likely it is to obtain indefinite kernel matrices. This provides a simple, efficient tool to decide whether additional effort should be spent on designing/selecting a more suitable kernel or algorithm.

Appendices

Appendix A: Distance measures for permutations

In the following, we describe the distance measures employed in the experiments.

• The Levenshtein distance is an edit distance measure:

  ${d} _{Lev}(\pi ,\pi ') = edits_{\pi \rightarrow \pi '}$

  Here, $edits_{\pi \rightarrow \pi '}$ is the minimal number of deletions, insertions, or substitutions required to transform one string (or here: permutation) $\pi$ into another string $\pi '$. The implementation is based on Wagner and Fischer (1974).

• Swaps are transpositions of two adjacent elements. The Swap distance [also: Kendall’s Tau (Kendall and Gibbons 1990; Sevaux and Sörensen 2005) or Precedence distance (Schiavinotto and Stützle 2007)] counts the minimum number of swaps required to transform one permutation into another. For permutations, it is (Sevaux and Sörensen 2005):

  $$\begin{aligned} {d} _{Swa}(\pi ,\pi ')&= \sum _{i=1}^{m} \sum _{j=1}^{m} z_{ij} ~~ \text {with}\\ z_{ij}&= \left\{ \begin{array}{l l} 1 &{} \quad \text {if } \pi _i < \pi _j ~\text {and}~ \pi '_i > \pi '_j ,\\ 0 &{} \quad \text {otherwise.} \end{array} \right. \end{aligned}$$

• An interchange operation is the transposition of two arbitrary elements. Respectively, the Interchange (also: Cayley) distance counts the minimum number of interchanges ($interchanges_{\pi \rightarrow \pi '}$) required to transform one permutation into another (Schiavinotto and Stützle 2007):

  ${d} _{Int}(\pi ,\pi ') = interchanges_{\pi \rightarrow \pi '}$

• The Insert distance is based on the longest common subsequence $LCSeq(\pi ,\pi ')$. The longest common subsequence is the largest number of elements that follow each other in both permutations, with interruptions. The corresponding distance is

  ${d} _{Ins}(\pi ,\pi ') = m-LCSeq(\pi ,\pi ').$

  We use the algorithm described by Hirschberg (1975). The name is due to its interpretation as an edit distance measure. The corresponding edit operation is a combination of insertion and deletion. A single element is moved from one position (delete) to a new position (insert). It is also called Ulam’s distance (Schiavinotto and Stützle 2007).

• The Longest Common Substring distance is based on the largest number of elements that follow each other in both permutations, without interruption. Unlike the longest common subsequence all elements have to be adjacent. If $LCStr(\pi ,\pi ')$ is the length of the longest common string, the distance is

  $$\begin{aligned} {d} _{LCStr}(\pi ,\pi ')= m-LCStr(\pi ,\pi '). \end{aligned}$$

• The R-distance (Campos et al. 2005; Sevaux and Sörensen 2005) counts the number of times that one element follows another in one permutation, but not in the other. It is identical with the uni-directional adjacency distance (Reeves 1999). It is computed by

  $$\begin{aligned} {d} _{R}(\pi ,\pi ')&= \sum _{i=1}^{m-1} y_i ~~ \text {with}\\ y_i&= \left\{ \begin{array}{ll} 0 &{} \quad \text {if }\exists j : \pi _i=\pi '_j ~\text {and}~ \pi _{i+1}=\pi '_{j+1} ,\\ 1 &{} \quad \text {otherwise.} \end{array} \right. \end{aligned}$$

• The (bi-directional) Adjacency distance (Reeves 1999; Schiavinotto and Stützle 2007) counts the number of times two elements are neighbors in one, but not in the other permutation. Unlike R-distance (uni-directional), the order of the two elements does not matter. It is computed by

  $$\begin{aligned} {d} _{Adj}(\pi ,\pi ')&= \sum _{i=1}^{m-1} y_i ~~ \text {with}\\ y_i&= \left\{ \begin{array}{l l} 0 &{} \quad \text {if }\exists j : \pi _i=\pi '_j ~\text {and}~ \pi _{i+1} \in \{\pi '_{j+1}, \pi '_{j-1} \},\\ 1 &{} \quad \text {otherwise.} \end{array} \right. \end{aligned}$$

• The Position distance (Schiavinotto and Stützle 2007) is identical with the Deviation distance or Spearman’s footrule (Sevaux and Sörensen 2005), ${d} _{\text {Pos}}(\pi ,\pi ') = \sum _{k=1}^{m} |i-j | ~~\text {where}~~\pi _i = \pi '_j = k$ .

• The non-metric Squared Position distance is Spearman’s rank correlation coefficient (Sevaux and Sörensen 2005). In contrast to the Position distance, the term $|i-j|$ is replaced by $(i-j)^2$.

• The Hamming distance or Exact Match distance simply counts the number of unequal elements in two permutations, i.e., ${d} _{Ham}(\pi ,\pi ') = \sum _{i=1}^{m} a_i, ~~\text {where}~~ a_i = \left\{ \begin{array}{l l} 0 &{} \quad \text {if } \pi _i = \pi '_i,\\ 1 &{} \quad \text {otherwise.} \end{array} \right.$

• The Euclidean distance is ${d} _{Euc}(\pi ,\pi ') = \sqrt{\sum _{i=1}^{m} (\pi _i-\pi '_i)^2}$ .

• The Manhattan distance (A-Distance, cf. (Sevaux and Sörensen 2005; Campos et al. 2005)) is ${d} _{Man}(\pi ,\pi ') = \sum _{i=1}^{m} |\pi _i-\pi '_i|$ .

• The Chebyshev distance is ${d} _{Che}(\pi ,\pi ') = \underset{1 \le i \le m}{\max }(|\pi _i-\pi '_i|)$ .

• For permutations, the Lee distance (Lee 1958; Deza and Huang 1998) is ${d} _{Lee}(\pi ,\pi ') = \sum _{i=1}^{m} \min (|\pi _i-\pi '_i|,m-|\pi _i-\pi '_i|)$ .

• The non-metric Cosine distance is based on the dot product of two permutations. It is derived from the cosine similarity (Singhal 2001) of two vectors:

  $$\begin{aligned} {d} _{Cos}(\pi ,\pi ') = 1 - \frac{\pi \cdot \pi '}{||\pi ||~||\pi '||}. \end{aligned}$$

• The Lexicographic distance regards the lexicographic ordering of permutations. If the position of a permutation $\pi$ in the lexicographic ordering of all permutations with fixed m is $L(\pi )$, then the Lexicographic distance metric is

  $$\begin{aligned} {d} _{Lex}(\pi ,\pi ') =| L(\pi ) - L(\pi ')|. \end{aligned}$$

Table 3 Minimal examples for indefinite distance matrices. The matrix in the table is the actual distance matrix, while the eigenvalue refers to the transformed matrix $\hat{D}$ derived from Eq. (3). The lower triangular matrix is omitted due to symmetry

Appendix B: Minimal examples for indefinite sets

To showcase the usefulness of the proposed methods, this section lists small example datasets and the respective indefinite distance matrices. Besides the standard permutation distances we also tested:

• Signed permutations, reversal distance Permutations where each element has a sign are referred to as signed permutations. An application example for signed permutations is, e.g., weld path optimization (Voutchkov et al. 2005). The reversal distance counts the number of reversals required to transform one permutation into another. We used the non-cyclic reversal distance provided in the GRAPPA library version 2.0 (Bader et al. 2004).

• Labeled trees, tree edit distance Trees in general are widely applied as solution representation, e.g., in Genetic Programming. In this study, we considered labeled trees. The tree edit distance counts the number node insertions, deletions or relabels. We used the efficient implementation in the APTED 0.1.1 library (Pawlik and Augsten 2015, 2016). The labeled trees will be denoted with the bracket notation: curly brackets indicate the tree structure, letters indicate labels (internal and terminal nodes).

• Strings, optimal String Alignment distance (OSA) The OSA is an non-metric edit distance that counts insertions, deletions, substitutions and transpositions of characters. Each substring can be edited no more than once. It is also called the restricted Damerau-Levenshtein distance (Boytsov 2011). We used the implementation in the stringdist R-package (van der Loo 2014).

• Strings, Jaro–Winkler distance The Jaro Winkler distance is based on the number of matching characters in two strings as well as the number of transpositions required to bring all matches in the same order. We used the implementation in the stringdist R-package (van der Loo 2014).

The respective results are listed in Table 3. All of the listed distance measures are shown to be non-CNSD.