Non-linear Causal Inference Using Gaussianity Measures

Abstract

We provide theoretical and empirical evidence for a type of asymmetry between causes and effects that is present when these are related via linear models contaminated with additive non-Gaussian noise. Assuming that the causes and the effects have the same distribution, we show that the distribution of the residuals of a linear fit in the anti-causal direction is closer to a Gaussian than the distribution of the residuals in the causal direction. This Gaussianization effect is characterized by reduction of the magnitude of the high-order cumulants and by an increment of the differential entropy of the residuals. The problem of non-linear causal inference is addressed by performing an embedding in an expanded feature space, in which the relation between causes and effects can be assumed to be linear. The effectiveness of a method to discriminate between causes and effects based on this type of asymmetry is illustrated in a variety of experiments using different measures of Gaussianity. The proposed method is shown to be competitive with state-of-the-art techniques for causal inference.

The vast majority of the work was done while being at the Max Planck Institute for Intelligent Systems and at Cambridge University

Appendices

Appendix 1

In this appendix we show that if $\mathscr {X}$ and $\mathscr {Y}$ follow the same distribution and they have been centered, then the determinant of the covariance matrix of the random variable corresponding to 𝜖 _i, denoted with Cov(𝜖 _i), coincides with the determinant of the covariance matrix corresponding to the random variable $\tilde {\boldsymbol {\epsilon }}_i$, denoted with $\text{Cov}(\tilde {\boldsymbol {\epsilon }}_i)$.

From the causal model, i.e., y _i = Ax _i + 𝜖 _i, we have that:

$$\displaystyle \begin{aligned} \text{Cov}(\mathscr{Y}) &= \mathbf{A} \text{Cov}(\mathscr{X}) {\mathbf{A}}^{\text{T}} + \text{Cov}(\boldsymbol{\epsilon}_i)\,. \end{aligned} $$

(8.43)

Since $\mathscr {X}$ and $\mathscr {Y}$ follow the same distribution we have that $\text{Cov}(\mathscr {Y})=\text{Cov}(\mathscr {X})$. Furthermore, we know from the causal model that $\mathbf {A}=\text{Cov}(\mathscr {Y},\mathscr {X})\text{Cov}(\mathscr {X})^{-1}$. Then,

$$\displaystyle \begin{aligned} \text{Cov}(\boldsymbol{\epsilon}_i) & = \text{Cov}(\mathscr{X}) - \text{Cov}(\mathscr{Y},\mathscr{X}) \text{Cov}(\mathscr{X})^{-1} \text{Cov}(\mathscr{X},\mathscr{Y}) \,. \end{aligned} $$

(8.44)

In the case of $\tilde {\boldsymbol {\epsilon }}_i$ we know that the relation $\tilde {\boldsymbol {\epsilon }}_i = (\mathbf {I}-\tilde {\mathbf {A}} \mathbf {A}){\mathbf {x}}_i - \tilde {\mathbf {A}} \boldsymbol {\epsilon }_i$ must be satisfied, where $\tilde {\mathbf {A}}=\text{Cov}(\mathscr {X},\mathscr {Y})\text{Cov}(\mathscr {Y})^{-1}= \text{Cov}(\mathscr {X},\mathscr {Y})\text{Cov}(\mathscr {X})^{-1}$. Thus, we have that:

$$\displaystyle \begin{aligned} \text{Cov}(\tilde{\boldsymbol{\epsilon}}_i) &= (\mathbf{I}-\tilde{\mathbf{A}} \mathbf{A}) \text{Cov}(\mathscr{X}) (\mathbf{I}- {\mathbf{A}}^{\text{T}}\tilde{\mathbf{A}}^{\text{T}}) + \tilde{\mathbf{A}} \text{Cov}(\boldsymbol{\epsilon}_i) \tilde{\mathbf{A}}^{\text{T}} \end{aligned} $$

(8.45)

$$\displaystyle \begin{aligned} & = \text{Cov}(\mathscr{X}) - \text{Cov}(\mathscr{X}) {\mathbf{A}}^{\text{T}}\tilde{\mathbf{A}}^{\text{T}} - \tilde{\mathbf{A}}\mathbf{A} \text{Cov}(\mathscr{X}) + \tilde{\mathbf{A}}\mathbf{A}\text{Cov}(\mathscr{X}) {\mathbf{A}}^{\text{T}}\tilde{\mathbf{A}}^{\text{T}} \end{aligned} $$

(8.46)

$$\displaystyle \begin{aligned} & \quad + \tilde{\mathbf{A}} \text{Cov}(\mathscr{X}) \tilde{\mathbf{A}}^{\text{T}} - \tilde{\mathbf{A}} \text{Cov}(\mathscr{Y},\mathscr{X}) \text{Cov}(\mathscr{X})^{-1} \text{Cov}(\mathscr{X},\mathscr{Y}) \tilde{\mathbf{A}}^{\text{T}} \end{aligned} $$

(8.47)

$$\displaystyle \begin{aligned} & = \text{Cov}(\mathscr{X}) - \text{Cov}(\mathscr{X}) {\mathbf{A}}^{\text{T}}\tilde{\mathbf{A}}^{\text{T}} - \tilde{\mathbf{A}}\mathbf{A} \text{Cov}(\mathscr{X}) + \tilde{\mathbf{A}} \text{Cov}(\mathscr{X}) \tilde{\mathbf{A}}^{\text{T}} \end{aligned} $$

(8.48)

$$\displaystyle \begin{aligned} & = \text{Cov}(\mathscr{X}) - \text{Cov}(\mathscr{X},\mathscr{Y}) \text{Cov}{\mathscr{X}}^{-1} \text{Cov}(\mathscr{Y},\mathscr{X}) \end{aligned} $$

(8.49)

$$\displaystyle \begin{aligned} & \quad - \text{Cov}(\mathscr{X},\mathscr{Y}) \text{Cov}{\mathscr{X}}^{-1} \text{Cov}(\mathscr{Y},\mathscr{X}) \end{aligned} $$

(8.50)

$$\displaystyle \begin{aligned} & \quad + \text{Cov}(\mathscr{X},\mathscr{Y}) \text{Cov}{\mathscr{X}}^{-1} \text{Cov}(\mathscr{Y},\mathscr{X}) \end{aligned} $$

(8.51)

$$\displaystyle \begin{aligned} & = \text{Cov}(\mathscr{X}) - \text{Cov}(\mathscr{X},\mathscr{Y}) \text{Cov}(\mathscr{X})^{-1} \text{Cov}(\mathscr{Y},\mathscr{X}) \,. \end{aligned} $$

(8.52)

By the matrix determinant theorem we have that $\text{det} \text{Cov}(\tilde {\boldsymbol {\epsilon }}_i)= \text{det} \text{Cov}(\boldsymbol {\epsilon }_i)$. See [23, p. 117] for further details.

Appendix 2

In this Appendix we motivate that, if the distribution of the residuals is not Gaussian, but is close to Gaussian, one should also expected more Gaussian residuals in the anti-causal direction in terms of the energy distance described in Sect. 8.3.4. For simplicity we will consider the univariate case. We use the fact that the energy distance in the one-dimensional case is the squared distance between the cumulative distribution functions of the residuals and a Gaussian distribution [32]. Thus,

$$\displaystyle \begin{aligned} \tilde{D}^2 &= \int_{-\infty}^\infty \left[\tilde{F}(x) - \varPhi(x)\right]^2 d x\,, & D^2 &= \int_{-\infty}^\infty \left[F(x) - \varPhi(x)\right]^2 d x\,, \end{aligned} $$

(8.53)

where $\tilde {D}^2$ and D ² are the energy distances to the Gaussian distribution in the anti-causal and the causal direction respectively; $\tilde {F}(x)$ and F(x) are the c.d.f. of the residuals in the anti-causal and the causal direction, respectively; and finally, Φ(x) is the c.d.f. of a standard Gaussian.

One should expect that $\tilde {D}^2 \leq D^2$. To motivate this, we use the Gram-Charlier series and compute an expansion of $\tilde {F}(x)$ and F(x) around the standard Gaussian distribution [24]. Such an expansion only converges in the case of distributions that are close to be Gaussian (see Sect. 17.6.6a of [5] for further details). Namely,

$$\displaystyle \begin{aligned} \tilde{F}(x) & = \varPhi(x) - \phi(x) \left( \frac{\tilde{a}_3}{3!} H_2(x) + \frac{\tilde{a}_4}{4!} H_3(x) + \cdots \right)\,, \end{aligned} $$

(8.54)

$$\displaystyle \begin{aligned} F(x) & = \varPhi(x) - \phi(x) \left( \frac{a_3}{3!} H_2(x) + \frac{a_4}{4!} H_3(x) + \cdots \right)\,, \end{aligned} $$

(8.55)

where ϕ(x) is the p.d.f. of a standard Gaussian, H _n(x) are Hermite polynomials and $\tilde {a}_n$ and a _n are coefficients that depend on the cumulants, e.g., a ₃ = κ ₃, a ₄ = κ ₄, $\tilde {a}_3=\tilde {\kappa }_3$, $\tilde {a}_4=\tilde {\kappa }_4$. Note, however, that coefficients a _n and $\tilde {a}_n$ for n > 5 depend on combinations of the cumulants. Using such an expansion we find:

(8.56)

$$\displaystyle \begin{aligned} & \approx \frac{\tilde{\kappa}_3^2}{36} \mathbb{E}[H_2(x)^2\phi(x)] + \frac{\tilde{\kappa}_4^2}{576} \mathbb{E}[H_3(x)^2\phi(x)] \,, \end{aligned} $$

(8.57)

where $\mathbb {E}[\cdot ]$ denotes expectation with respect to a standard Gaussian and we have truncated the Gram-Charlier expansion after n = 4. Truncation of the Gram-Charlier expansion after n = 4 is a standard procedure that is often done in the ICA literature for entropy approximation. See for example Sect. 5.5.1 of [13]. We have also used the fact that $\mathbb {E}[H_3(x)H_2(x)\phi (x)] = 0$. The same approach can be followed in the case of D ², the energy distance in the causal direction. The consequence is that $D^2\approx \kappa _3^2/36 \cdot \mathbb {E}[H_2(x)^2\phi (x)] + \kappa _4^2/576 \cdot \mathbb {E}[H_3(x)^2\phi (x)]$. Finally, the fact that one should expect $\tilde {D}^2\leq D^2$ is obtained by noting that $\tilde {\kappa }_n = c_n \kappa _n$, where c _n is some constant that lies in the interval (−1, 1), as indicated in Sect. 8.2.1. We expect that this result extends to the multivariate case.

Appendix 3

In this Appendix we motivate that one should expect also more Gaussian residuals in the anti-causal direction, based on a reduction of the cumulants, when the residuals in feature space are projected onto the first principal component. That is, when they are multiplied by the first eigenvector of the covariance matrix of the residuals, and scaled by the corresponding eigenvalue. Recall from Sect. 8.2.2 that these covariance matrices are C = I −AA ^T and $\tilde {\mathbf {C}}=\mathbf {I} - {\mathbf {A}}^{\text{T}} \mathbf {A}$, in the causal and anti-causal direction respectively. Note that both matrices have the same eigenvalues.

If A is symmetric we have that both C and $\tilde {\mathbf {C}}$ have the same matrix of eigenvectors P. Let ${\mathbf {p}}_1^n$ be the first eigenvector multiplied n times using the Kronecker product. The cumulants in the anti-causal and the causal direction, after projecting the data onto the first eigenvector are $\tilde {\kappa }_n^{\text{proj}} = ({\mathbf {p}}_1^{\text{n}})^{\text{T}} {\mathbf {M}}_n \text{vect}(\tilde {\kappa }_n) = c({\mathbf {p}}_1^{\text{n}})^{\text{T}} \text{vect}(\tilde {\kappa }_n)$ and $\kappa _n^{\text{proj}} = ({\mathbf {p}}_1^{\text{n}})^{\text{T}} \text{vect}(\kappa _n)$, respectively, where M _n is the matrix that relates the cumulants in the causal and the anti-causal direction (see Sect. 8.2.2) and c is one of the eigenvalues of M _n. In particular, if A is symmetric, it is not difficult to show that ${\mathbf {p}}_1^{\text{n}}$ is one of the eigenvectors of M _n. Furthermore, we also showed in that case that ||M _n||_op < 1 for n ≥ 3 (see Sect. 8.2.2). The consequence is that c ∈ (−1, 1), which combined with the fact that $||{\mathbf {p}}_1^n||=1$ leads to smaller cumulants in magnitude in the case of the projected residuals in the anti-causal direction.

If A is not symmetric we motivate that one should also expect more Gaussian residuals in the anti-causal direction due to a reduction in the magnitude of the cumulants. For this, we derive a smaller upper bound on their magnitude. This smaller upper bound is based on an argument that uses the operator norm of vectors.

Definition 8.2

The operator norm of a vector w induced by the ℓ _p norm is ||w||_op = min{c ≥ 0 : ||w ^T v||_p ≤ c||v||_p, ∀v}.

The consequence is that ||w||_op ≥||w ^T v||_p∕||v||_p, ∀v. Thus, the smallest the operator norm of w, the smallest the expected value obtained when multiplying any vector by the vector w. Furthermore, it is clear that ||w||_op = ||w||₂, in the case of the ℓ ₂-norm. From the previous paragraph, in the anti-causal direction we have $||\tilde {\kappa }_n^{\text{proj}}||{ }_2= ||(\tilde {\mathbf {p}}_1^n)^{\text{T}} {\mathbf {M}}_n \text{vect}(\kappa _n)||{ }_2$, where $\tilde {\mathbf {p}}_1$ is the first eigenvector of $\tilde {\mathbf {C}}$, while in the causal direction we have $||\kappa _n^{\text{proj}}||{ }_2= ||({\mathbf {p}}_1^n)^{\text{T}} \text{vect}(\kappa _n)||{ }_2$, where p ₁ is the first eigenvector of C. Thus, because the norm of each vector $\tilde {\mathbf {p}}_1^n$ and ${\mathbf {p}}_1^n$ is one, we have that $||{\mathbf {p}}_1^n||{ }_{\text{op}}=1$. However, because we expect M _n, to reduce the norm of $(\tilde {\mathbf {p}}_1^n)^{\text{T}}$, as motivated in Sect. 8.2.2, $||(\tilde {\mathbf {p}}_1^n)^{\text{T}}{\mathbf {M}}_n||{ }_{\text{op}} < 1$ should follow. This is expected to lead to smaller cumulants in magnitude in the anti-causal direction.