Efficient Bayesian shape-restricted function estimation with constrained Gaussian process priors

Abstract

This article revisits the problem of Bayesian shape-restricted inference in the light of a recently developed approximate Gaussian process that admits an equivalent formulation of the shape constraints in terms of the basis coefficients. We propose a strategy to efficiently sample from the resulting constrained posterior by absorbing a smooth relaxation of the constraint in the likelihood and using circulant embedding techniques to sample from the unconstrained modified prior. We additionally pay careful attention to mitigate the computational complexity arising from updating hyperparameters within the covariance kernel of the Gaussian process. The developed algorithm is shown to be accurate and highly efficient in simulated and real data examples.

Appendices

Appendices

1.1 Full conditionals

Consider model (9) and the prior specified in Sect. 3.1. The joint distribution is given by:

$$\begin{aligned}&\pi (Y, \xi _0, \xi , \sigma ^2, \tau ^2) \propto \big (\sigma ^2 \big )^{-\frac{n}{2} - 1} \\&\quad \exp \bigg \{ - \frac{1}{2 \sigma ^2} \Vert Y - \xi _0 {\mathrm {1}}_n - \Psi \xi \Vert ^2 \bigg \} \\&\quad \big (\tau ^2 \big )^{-\frac{N+1}{2} - 1} \, \exp \bigg \{ - \frac{1}{2 \tau ^2} \xi ^{\mathrm {\scriptscriptstyle T}}K^{-1} \xi \bigg \} \, \mathbb {1}_{{\mathcal {C}}_{\xi }}(\xi ) \end{aligned}$$

Then,

$\xi \mid Y, \xi _0, \sigma ^2, \tau ^2$ is truncated multivariate Gaussian truncated on $\mathbb {1}_{{\mathcal {C}}_{\xi }}(\xi )$.

$\xi _0 \mid Y, \xi , \sigma ^2, \tau ^2 \sim {\mathcal {N}}({\bar{Y}}^*, \sigma ^2/n)$, where, ${\bar{Y}}^*$ is average of components of $Y^* = Y - \Psi \xi$.

$\sigma ^2 \mid Y, \xi _0, \xi , \tau ^2 \sim {\mathcal {I}} {\mathcal {G}} \big ( n/2, \Vert Y - \xi _0 {\mathrm {1}}_n - \Psi \xi \Vert ^2 /2 \big )$

$\tau ^2 \mid Y, \xi _0, \xi , \sigma ^2 \sim {\mathcal {I}} {\mathcal {G}} \big ((N+1)/2, \xi ^{\mathrm {\scriptscriptstyle T}}K^{-1} \xi /2 \big )$

Again, consider model (10) and the prior specified in Sect. 3.2. The joint distribution is given by:

$$\begin{aligned}&\pi (Y, \xi _0, \xi _*, \xi , \sigma ^2, \tau ^2) \propto \big (\sigma ^2 \big )^{-\frac{n}{2} - 1} \exp \bigg \{ \\&\quad - \frac{1}{2 \sigma ^2} \Vert Y - \xi _0 {\mathrm {1}}_n - \xi _* X - \Phi \xi \Vert ^2 \bigg \} \big (\tau ^2 \big )^{-\frac{N+1}{2} - 1} \\&\quad \exp \bigg \{ - \frac{1}{2 \tau ^2} \xi ^{\mathrm {\scriptscriptstyle T}}K^{-1} \xi \bigg \} \, \mathbb {1}_{{\mathcal {C}}_{\xi }}(\xi ) \end{aligned}$$

Then,

$\xi \mid Y, \xi _0, \xi _*, \sigma ^2, \tau ^2$ is truncated multivariate Gaussian truncated on $\mathbb {1}_{{\mathcal {C}}_{\xi }}(\xi )$.

$\xi _0 \mid Y, \xi _*, \xi , \sigma ^2, \tau ^2 \sim {\mathcal {N}}({\bar{Y}}^*, \sigma ^2/n)$, ${\bar{Y}}^*$ is average of components of $Y^* = Y - \xi _* X - \Phi \xi$.

$\xi _* \mid Y, \xi _0, \xi , \sigma ^2, \tau ^2 \sim {\mathcal {N}}( \sum _{i=1}^{n} x_i y_i^{**}/ \sum _{i=1}^{n} x_i^2, \sigma ^2/ \sum _{i=1}^{n} x_i^2)$, where $Y^{**} = Y - \xi _0 {\mathrm {1}}_n - \Phi \xi$.

$\sigma ^2 \mid Y, \xi _0, \xi _*, \xi , \tau ^2 \sim {\mathcal {I}} {\mathcal {G}} \big ( n/2, \Vert Y - \xi _0 {\mathrm {1}}_n - \xi _0 X - \Phi \xi \Vert ^2 /2 \big )$

$\tau ^2 \mid Y, \xi _0, \xi _*, \xi , \sigma ^2 \sim {\mathcal {I}} {\mathcal {G}} \big ( (N+1)/2, \xi ^{\mathrm {\scriptscriptstyle T}}K^{-1} \xi /2 \big )$

Fig. 4

Boxplots of effective sample sizes of the estimated function value at 75 different points for the monotone function estimation example. The effective sample sizes are calculated based on 10,000 MCMC runs and averaged over 5 random starting points

Algorithm 1 was used to draw samples from the full conditional distribution of $\xi$ while sampling from the full conditionals of $\xi _0$, $\xi _*$, $\sigma ^2$ and $\tau ^2$ are routine.

1.2 Effective sample sizes for the monotone example in Sect. 3.3

We provide some evidence towards the mixing behavior of our Gibbs sampler by computing the effective sample size of the estimated function value at 75 different test points. The effective sample size is a measure of the amount of the autocorrelation in a Markov chain, and essentially amounts to the number of independent samples in the MCMC path. From an algorithmic robustness perspective, it is desirable that the effective sample sizes remain stable across increasing sample size and/or dimension, and this is the aspect we wish to investigate here. We only report results for the monotonicity constraint; similar behavior is seen for the convexity constraint as well.

We consider 20 different values for the sample size n with equal spacing between 50 and 1000. Note that the dimension of $\xi$ itself grows between 25 and 500 as a result. For each value of n, we run the Gibbs sampler for 12,000 iterations with 5 randomly chosen initializations. For each starting point, we record the effective sample size at each of the 75 test points after discarding the first 2,000 iterations as burn-in, and average them over the different initializations. Figure 4 shows boxplots of these averaged effective sample sizes across n which are seen to be quite stable across growing n.

1.3 R code

We used R for the implementation of Algorithm 1 and Durbin’s recursion to find the inverse of the Cholesky factor, with the computation of the inverse Cholesky factor optimized with Rcpp. We provide our code for implementing the monotone and convex function estimation procedures in Sects. 3.1 and 3.2 in the Github page mentioned in Sect. 1. There are six different functions to perform the MCMC sampling for monotone increasing, monotone decreasing, and convex increasing functions with and without hyperparameter updates. Each of these main functions take x and y as inputs along with other available options, and return posterior samples on $\xi _0$, $\xi ^*$, $\xi$, $\sigma$, $\tau$ and f along with posterior mean and symmetric 95% credible interval of f on a user-specified grid. A detailed description on the available input and output options for each function can be found within the function files.