Conformal Predictive Distributions with Kernels

Abstract

This paper reviews the checkered history of predictive distributions in statistics and discusses two developments, one from recent literature and the other new. The first development is bringing predictive distributions into machine learning, whose early development was so deeply influenced by two remarkable groups at the Institute of Automation and Remote Control. As result, they become more robust and their validity ceases to depend on Bayesian or narrow parametric assumptions. The second development is combining predictive distributions with kernel methods, which were originated by one of those groups, including Emmanuel Braverman. As result, they become more flexible and, therefore, their predictive efficiency improves significantly for realistic non-linear data sets.

A Properties of the Hat Matrix

In the kernelized setting of this paper the hat matrix is defined as $H=(K+aI)^{-1}K$, where K is a symmetric positive semidefinite matrix whose size is denoted $n\times n$ in this appendix (cf. (10); in our current abstract setting we drop the bars over H and K and write n in place of $n+1$). We will prove, or give references for, various properties of the hat matrix used in the main part of the paper.

Numerous useful properties of the hat matrix can be found in literature (see, e.g., [3]). However, the usual definition of the hat matrix is different from ours, since it is not kernelized; therefore, we start from reducing our kernelized definition to the standard one. Since K is symmetric positive semidefinite, it can be represented in the form $K=XX'$ for some matrix X, whose size will be denoted $n\times p$ (in fact, a matrix is symmetric positive semidefinite if and only if it can be represented as the Gram matrix of n vectors; this easily follows from the fact that a symmetric positive semidefinite K can be diagonalized: $K=Q'\varLambda Q$, where Q and $\varLambda$ are $n\times n$ matrices, $\varLambda$ is diagonal with nonnegative entries, and $Q'Q=I$). Now we can transform the hat matrix as

$$ H = (K+aI)^{-1}K = (XX'+aI)^{-1}XX' = X(X'X+aI)^{-1}X' $$

(the last equality can be checked by multiplying both sides by $(XX'+aI)$ on the left). If we now extend X by adding $\sqrt{a}I_p$ on top of it (where $I_p=I$ is the $p\times p$ unit matrix),

$$\begin{aligned} \tilde{X} := \begin{pmatrix} \sqrt{a}I_p \\ X \end{pmatrix}, \end{aligned}$$

(30)

and set

$$\begin{aligned} \tilde{H} := \tilde{X}(\tilde{X}'\tilde{X})^{-1}\tilde{X}' = \tilde{X}(X'X+aI)^{-1}\tilde{X}', \end{aligned}$$

(31)

we will obtain a $(p+n)\times (p+n)$ matrix containing H in its lower right $n\times n$ corner. To find HY for a vector $Y\in \mathbb {R}^n$, we can extend Y to $\tilde{Y}\in \mathbb {R}^{p+n}$ by adding p zeros at the beginning of Y and then discard the first p elements in $\tilde{H} \tilde{Y}$. Notice that $\tilde{H}$ is the usual definition of the hat matrix associated with the data matrix $\tilde{X}$ (cf. [3, (1.4a)]).

When discussing (11), we used the fact that the diagonal elements of H are in [0, 1). It is well-known that the diagonal elements of the usual hat matrix, such as $\tilde{H}$, are in [0, 1] (see, e.g., [3, Property 2.5(a)]). Therefore, the diagonal elements of H are also in [0, 1]. Let us check that $h_i$ are in fact in the semi-open interval [0, 1) directly, without using the representation in terms of $\tilde{H}$. Representing $K=Q'\varLambda Q$ as above, where $\varLambda$ is diagonal with nonnegative entries and $Q'Q=I$, we have

$$\begin{aligned} H = (K+aI)^{-1} K&= (Q'\varLambda Q+aI)^{-1} Q'\varLambda Q = (Q'(\varLambda +aI)Q)^{-1} Q'\varLambda Q \nonumber \\&= Q^{-1}(\varLambda +aI)^{-1}(Q')^{-1} Q'\varLambda Q = Q'(\varLambda +aI)^{-1}\varLambda Q. \end{aligned}$$

(32)

The matrix $(\varLambda +aI)^{-1}\varLambda$ is diagonal with the diagonal entries in the semi-open interval [0, 1). Since $Q'Q=I$, the columns of Q are vectors of length 1. By (32), each diagonal element of H is then of the form $\sum _{i=1}^n \lambda _i q_i^2$, where all $\lambda _i\in [0,1)$ and $\sum _{i=1}^n q_i^2 = 1$. We can see that each diagonal element of H is in [0, 1).

The equality (11) itself was used only for motivation, so we do not prove it; for a proof in the non-kernelized case, see, e.g., [14, (4.11) and Appendix C.7].

Proof of Lemma  2

In our proof of $B_i>0$ we will assume $a>0$, as usual. We will apply the results discussed so far in this appendix to the matrix $\bar{H}$ in place of H and to $n+1$ in place of n.

Our goal is to check the strict inequality

$$\begin{aligned} \sqrt{1-\bar{h}_{n+1}} + \frac{\bar{h}_{i,n+1}}{\sqrt{1-\bar{h}_{i}}} > 0; \end{aligned}$$

(33)

remember that both $\bar{h}_{n+1}$ and $\bar{h}_i$ are numbers in the semi-open interval [0, 1). The inequality (33) can be rewritten as

$$\begin{aligned} \bar{h}_{i,n+1} > -\sqrt{(1-\bar{h}_{n+1})(1-\bar{h}_{i})} \end{aligned}$$

(34)

and in the weakened form

$$\begin{aligned} \bar{h}_{i,n+1} \ge -\sqrt{(1-\bar{h}_{n+1})(1-\bar{h}_{i})} \end{aligned}$$

(35)

follows from [3, Property 2.6(b)] (which can be applied to $\tilde{H}$).

Instead of the original hat matrix $\bar{H}$ we will consider the extended matrix (31), where $\tilde{X}$ is defined by (30) with $\bar{X}$ in place of X. The elements of $\tilde{H}$ will be denoted $\tilde{h}$ with suitable indices, which will run from $-p+1$ to $n+1$, in order to have the familiar indices for the submatrix $\bar{H}$. We will assume that we have an equality in (34) and arrive at a contradiction. There will still be an equality in (34) if we replace $\bar{h}$ by $\tilde{h}$, since $\tilde{H}$ contains $\bar{H}$. Consider auxiliary “random residuals” $E:=(I-\tilde{H})\epsilon$, where $\epsilon$ is a standard Gaussian random vector in $\mathbb {R}^{p+n+1}$; there are $p+n+1$ random residuals $E_{-p+1},\ldots ,E_{n+1}$. Since the correlation between the random residuals $E_i$ and $E_{n+1}$ is

$$ {{\mathrm{corr}}}(E_i,E_{n+1}) = \frac{-\tilde{h}_{i,n+1}}{\sqrt{(1-\tilde{h}_{n+1})(1-\tilde{h}_{i})}} $$

(this easily follows from $I-\tilde{H}$ being a projection matrix and is given in, e.g., [3, p. 11]), (35) is indeed true. Since we have an equality in (34) (with $\tilde{h}$ in place of $\bar{h}$), $E_i$ and $E_{n+1}$ are perfectly correlated. Remember that neither row number i nor row number $n+1$ of the matrix $I-\bar{H}$ are zero (since the diagonal elements of $\bar{H}$ are in the semi-open interval [0, 1)), and so neither $E_i$ nor $E_{n+1}$ are zero vectors. Since $E_i$ and $E_{n+1}$ are perfectly correlated, the row number i of the matrix $I-\tilde{H}$ is equal to a positive scalar c times its row number $n+1$. The projection matrix $I-\tilde{H}$ then projects $\mathbb {R}^{p+n+1}$ onto a subspace of the hyperplane in $\mathbb {R}^{p+n+1}$ consisting of the points with coordinate number i being c times the coordinate number $n+1$. The orthogonal complement of this subspace, i.e., the range of $\tilde{H}$, will contain the vector $(0,\ldots ,0,-1,0,\ldots ,0,c)$ ($-1$ being its coordinate number i). Therefore, this vector will be in the range of $\tilde{X}$ (cf. (31)). Therefore, this vector will be a linear combination of the columns of the extended matrix (30) (with $\bar{X}$ in place of X), which is impossible because of the first p rows $\sqrt{a}I_p$ of the extended matrix.