Bayesian Fusion Estimation via t Shrinkage

Abstract

Shrinkage prior has gained great successes in many data analysis, however, its applications mostly focus on the Bayesian modeling of sparse parameters. In this work, we will apply Bayesian shrinkage to model high dimensional parameter that possesses an unknown blocking structure. We propose to impose heavy-tail shrinkage prior, e.g., t prior, on the differences of successive parameter entries, and such a fusion prior will shrink successive differences towards zero and hence induce posterior blocking. Comparing to conventional Bayesian fused LASSO which implements Laplace fusion prior, t fusion prior induces stronger shrinkage effect and enjoys a nice posterior consistency property. Simulation studies and real data analyses show that t fusion has superior performance to the frequentist fusion estimator and Bayesian Laplace fusion prior. This t fusion strategy is further developed to conduct a Bayesian clustering analysis, and our simulations show that the proposed algorithm compares favorably to classical Dirichlet process modeling.

Appendix

First, let us state some useful lemmas.

Lemma A.1 (Lemma 1 of Laurent and Massart 2000).

Let ${\chi ^{2}_{d}}(\kappa )$be a chi-square distribution with degree of freedom d, and noncentral parameterκ, then we have the following concentration inequality

$$ \begin{array}{@{}rcl@{}} &&Pr({\chi^{2}_{d}}(\kappa)>d+\kappa+2x+\sqrt{(4d+8\kappa)x})\leq \exp(-x), \text{ and}\\ &&Pr({\chi^{2}_{d}}(\kappa)<d+\kappa-\sqrt{(4d+8\kappa)x})\leq \exp(-x). \end{array} $$

Lemma A.2 (Theorem 1 of Zubkov and Serov 2013).

Let X be a Binomial random variableX ∼B(n,v). For any1 < k < n − 1

$$ Pr(X\geq k+1)\leq 1- {\Phi}(\text{sign}(k-nv)\{2nH(v, k/n)\}^{1/2}), $$

where Φ is the cumulative distribution function of standard Gaussian distribution and H(v,k/n) = (k/n)log(k/nv) + (1 − k/n)log[(1 − k/n)/(1 − v)].

The next lemma is a refined result of Lemma 6 in Barron (1998):

Lemma A.3.

Let f^∗ be the true probability density of data generation,f_𝜃 be the likelihood function with parameter 𝜃 ∈Θ, and E^∗,E_𝜃 denote the corresponding expectation respectively. LetB_n andC_n be two subsets in parameter space Θ, and ϕ_n be some test function satisfying ϕ_n(D_n) ∈ [0,1] for any data D_n. Ifπ(B_n) ≤ b_n, $E^{*}\phi (D_{n})\leq b_{n}^{\prime }$,$\sup _{\theta \in C_{n}}E_{\theta }(1-\phi (D_{n}))\leq c_{n}$, and furthermore,

$$P^{*}\left\{\frac{m(D_{n})}{f^{*}(D_{n})}\geq a_{n} \right\}\geq 1-a_{n}^{\prime},$$

where $m(D_{n})={\int }_{\Theta } \pi (\theta )f_{\theta }(D_{n})d\theta$ is the margin probability of D_n. Then,

$$ E^{*}\left( \pi(C_{n}\cup B_{n})|D_{n})\right)\leq \frac{b_{n}+c_{n}}{a_{n}}+a_{n}^{\prime}+b_{n}^{\prime}. $$

Proof 1.

Define Ω_n to be the event of (m(D_n))/(f^∗(D_n)) ≥ a_n, and $m(D_{n}, C_{n}\cup B_{n}) = {\int }_{C_{n}\cup B_{n}} \pi (\theta )f_{\theta }(D_{n})d\theta$. Then

$$ \begin{array}{@{}rcl@{}} &&E^{*}\pi(C_{n}\cup B_{n})|D_{n}) = E^{*}\pi(C_{n}\cup B_{n})|D_{n})(1-\phi(D_{n}))1_{{\Omega}_{n}}\\ &+&E^{*}\pi(C_{n}\cup B_{n})|D_{n})(1-\phi(D_{n}))(1-1_{{\Omega}_{n}})+ E^{*}\pi(C_{n}\cup B_{n})|D_{n})\phi(D_{n})\\ &\leq& E^{*}\pi(C_{n}\cup B_{n})|D_{n})(1-\phi(D_{n}))1_{{\Omega}_{n}}+E^{*}(1-1_{{\Omega}_{n}})+ E^{*}\phi(D_{n})\\ &\leq& E^{*}\pi(C_{n}\cup B_{n})|D_{n})(1-\phi(D_{n}))1_{{\Omega}_{n}}+b_{n}^{\prime}+a_{n}^{\prime}\\ &\leq& E^{*}\{m(D_{n}, C_{n}\cup B_{n})/a_{n}f^{*}(D_{n})\}(1-\phi(D_{n}))+b_{n}^{\prime}+a_{n}^{\prime}. \end{array} $$

By Fubini theorem,

$$ \begin{array}{@{}rcl@{}} &&E^{*}(1-\phi(D_{n}))m(D_{n}, C_{n}\cup B_{n})/f^{*}(D_{n}) = {\int}_{C_{n}\cup B_{n}} {\int}_{\mathcal{X}}[1-\phi(D_{n})]f_{\theta}(D_{n}) dD_{n}\pi(\theta)d\theta\\ &\leq&{\int}_{C_{n}} E_{\theta}(1-\phi(D_{n}))\pi(\theta)d\theta+{\int}_{B_{n}} {\int}_{\mathcal{X}}f_{\theta}(D_{n}) dD_{n}\pi(\theta)d\theta\leq b_{n}+c_{n}. \end{array} $$

Combining the above inequalities leads to the conclusion.□

Proof 2 (Proof of Theorem 2.1 and 2.2).

Let G = {g₁,g₂,…,g_d} be a generic subset of {2,…,n}, and it also represents potential a (d + 1)-group structure of 𝜃 as {{1,…,g₁},{g₁ + 1,…,g₂},…,{g_d + 1,…,n}}. Given G and its corresponding blocking structure, $\widehat \theta _{G}(y)$denotes the estimation of 𝜃 based on block mean, i.e. $\widehat \theta _{G,j}(y)$$= {\sum }_{i=g_{j}+1}^{g_{j+1}}y_{i}/(g_{j+1}-g_{j})$for all g_j + 1 ≤ j ≤ g_j+ 1, and $\widehat {\sigma }^{2}_{G}(y)=\|y-\widehat \theta _{G}\|^{2}/(n-|G|-1)$.

To prove the posterior contraction, we will apply Lemma A.3. We define the following testing function

$$ \begin{array}{ll} \phi(y) = 1\{&\|\widehat\theta_{G}-\theta^{*}\|\geq \sqrt{n}\sigma^{*}\epsilon_{n}\text{ and\ } |\widehat{\sigma}^{2}_{G}-\sigma^{*2}|>\sigma^{*2}\epsilon_{n}\\ &\text{ for all\ } G\supset G^{*}, |G|\leq (1+\delta)|G^{*}|\} \end{array} $$

(A.1)

for some δ > 0, and define C_n and B_n as:

$$ \begin{array}{@{}rcl@{}} &&C_{n}=\{\theta: \|\theta-\theta^{*}\|\leq M\sqrt n\sigma^{*}\epsilon_{n}, (1-\epsilon_{n})/(1+\epsilon_{n})<\sigma^{2}/\sigma^{*2}<(1+\epsilon_{n})/(1-\epsilon_{n})\}^{c}\backslash B_{n},\\ &&B_{n}=\{\theta: \text{Among all\ } \{\vartheta_{i}'s\}_{i\notin G^{*}} , \text{ there are at least} \ \delta|G^{*}| \text{ of them are greater than} \ \sigma\epsilon_{n}/n\}. \end{array} $$

Note that when G ⊃ G^∗, $\|\widehat \theta _{G}(y)-\theta ^{*}\|^{2}\sim \sigma ^{*2}\chi ^{2}_{|G|+1}$ and $\|y-\widehat \theta _{G}\|^{2}$$\sim \sigma ^{*2}\chi ^{2}_{n-|G|-1}$, thus by Lemma A.1, we have that

$$ P(\|\widehat\theta_{G}(y)-\theta^{*}\|\geq \sqrt{n}\sigma^{*}\epsilon_{n}\text{ and\ } |\widehat{\sigma}^{2}_{G}-\sigma^{*2}|>\sigma^{*2}\epsilon_{n}) \leq \exp\{-c_{1}n{\epsilon_{n}^{2}}\}, $$

for some constant c₁, since $|G|=O(|G^{*}|)\prec n{\epsilon _{n}^{2}}$ and 𝜖_n ≺ 1. Therefore,

$$ E_{(\theta^{*},\sigma^{*2})}\phi(y)\leq {n-1 \choose (1+\delta)|G^{*}|}\exp\{-c_{1}n{\epsilon_{n}^{2}}\}=\exp\{-c_{1}^{\prime}n{\epsilon_{n}^{2}}\}, $$

(A.2)

as long as n𝜖²/[|G^∗|log n] is sufficiently large.

For any (𝜃,σ²) ∈ C_n satisfying $\|\theta -\theta ^{*}\|\leq M\sqrt n\sigma ^{*}\epsilon _{n}$ and σ²/σ^∗2 ≥ (1 − 𝜖_n)/(1 + 𝜖_n), we define $\widehat G=\{i:\theta _{i}-\theta _{i-1}\geq \sigma \epsilon _{n}/n^{2}\}\cup G^{*}$ (hence |G|≤ (1 + δ)|G^∗|), thus

$$ \begin{array}{@{}rcl@{}} && P_{(\theta,\sigma^{2})}(\|\widehat\theta_{\widehat G}(y)-\theta^{*}\|\leq \sqrt{n}\sigma^{*}\epsilon_{n}) = P_{(\theta,\sigma^{2})}(\|\widehat\theta_{\widehat G}(y)-\widehat\theta_{\widehat G}(\theta) + \widehat\theta_{\widehat G}(\theta)-\theta^{*}\|\leq \sqrt{n}\sigma^{*}\epsilon_{n})\\ &\leq&P_{(\theta,\sigma^{2})}(\|\widehat\theta_{\widehat G}(y)-\widehat\theta_{\widehat G}(\theta)\| \geq \| \widehat\theta_{\widehat G}(\theta)-\theta^{*}\|-\sqrt{n}\sigma^{*}\epsilon_{n}) \\ &\leq& P_{(\theta,\sigma^{2})}(\|\widehat\theta_{\widehat G}(y)-\widehat\theta_{\widehat G}(\theta)\| \geq M\sqrt{n}\sigma^{*}\epsilon_{n}-\sqrt{n}\sigma\epsilon_{n}-\sqrt{n}\sigma^{*}\epsilon_{n}) \\ &\leq&P\left( \chi_{|G+1|}^{2}\geq \left[\sqrt{\frac{1-\epsilon_{n}}{1+\epsilon_{n}}} (M-1)-1\right] n{\epsilon_{n}^{2}}\right)\leq \exp\{-c_{2}n{\epsilon_{n}^{2}}\} \end{array} $$

for some c₂ given a large M, where the second inequality is due to the fact that $\|\widehat \theta _{\widehat G}(\theta )-\theta \|\leq \sqrt {n}\sigma \epsilon _{n}$ when 𝜃 ∈ B_n.

For any (𝜃,σ²) ∈ C_n satisfying σ²/σ^∗2 < (1 − 𝜖_n)/(1 + 𝜖_n) or σ²/σ^∗2 > (1 + 𝜖_n)/(1 − 𝜖_n),

$$ \begin{array}{@{}rcl@{}} && P_{(\theta,\sigma^{2})}(|\widehat{\sigma}^{2}_{G}-\sigma^{*2}|<\sigma^{*2}\epsilon_{n}) = P_{(\theta,\sigma^{2})}(|\|y-\widehat\theta_{G}\|^{2}/\sigma^{*2}(n-|G|-1)-1|<\epsilon_{n})\\ &\leq & P_{(\theta,\sigma^{2})}(1-\epsilon_{n}< \|y-\widehat\theta_{G}\|^{2}/\sigma^{*2}(n-|G|-1)<1+\epsilon_{n})\\ &\leq &P_{(\theta,\sigma^{2})}\left( \left|\frac{\|y-\widehat\theta_{G}(y)\|^{2}}{\sigma^{2}}\right.-(n-|G|-1)\left|>(n-|G|-1)\epsilon_{n}\right.\right)\\ &=&P_{(\theta,\sigma^{2})}\left( |\chi_{n-|G|-1}^{2}(\lambda)-(n-|G|-1)|>(n-|G|-1)\epsilon_{n}\right)\leq \exp\{-c_{2}^{\prime}n{\epsilon_{n}^{2}}\} \end{array} $$

for some $c_{2}^{\prime }$, where the noncentral parameter $\lambda <n{\epsilon _{n}^{2}}\prec (n-|G|-1)\epsilon _{n}$.

Combining the results from the previous two paragraph, it is easy to obtain that

$$ \sup_{(\theta,\sigma^{2})\in C_{n}}E_{(\theta,\sigma^{2})}[1-\phi(y)]\leq \max\{\exp(-c_{2}n{\epsilon_{n}^{2}}),\exp(-c_{2}^{\prime}n{\epsilon_{n}^{2}})\}. $$

(A.3)

Now we consider the marginal posterior density of data y. With probability $P(\|\varepsilon \|\leq 2\sqrt n\sigma ^{*})$ (which converges to 1),

$$ \begin{array}{@{}rcl@{}} &&\frac{m(y)}{f^{*}(y)}=\frac{{\int}_{\sigma^{2}}{\int}_{\theta} \sigma^{*n}\exp\{-\|\theta^{*}-\theta+\varepsilon\|^{2}/\sigma^{2}\} d\theta d\sigma^{2}} {\sigma^{n}\exp\{-\|\varepsilon\|^{2}/\sigma^{*2}\}}\\ &\geq&{\int}_{\sigma^{2}}{\int}_{\theta}\exp\left\{-\frac{\|\theta^{*}-\theta\|^{2}}{\sigma^{2}} -2\frac{(\theta^{*}-\theta)^{T}\varepsilon}{\sigma^{2}}+\frac{\|\varepsilon\|^{2}}{\sigma^{*2}}-\frac{\|\varepsilon\|^{2}}{\sigma^{2}}-n\log(\sigma/\sigma^{*})\right\} \\ &&\pi(\theta,\sigma^{2})d\theta d\sigma^{2}\\ &\geq& \pi(\max\{|\theta_{1}-\theta_{1}^{*}|, |\vartheta_{i}-\vartheta_{i}^{*}|\}/\sigma\leq |G^{*}|\log n/n^{2}, 0\leq\sigma^{2}-\sigma^{*2}\\ &\leq& \sigma^{*2}|G^{*}|\log n/n)\exp\{-c_{3}^{\prime}|G^{*}|\log n \} \end{array} $$

for some constant $c_{3}^{\prime }$. Besides,

$$ \begin{array}{@{}rcl@{}} &&\pi(\max\{|\theta_{1}-\theta_{1}^{*}|, |\vartheta_{i}-\vartheta_{i}^{*}|\}/\sigma\leq |G^{*}|\log n/n^{2}, 0\leq\sigma^{2}-\sigma^{*2}\leq\sigma^{*2} |G^{*}|\log n/n)\\ &\geq&\pi_{\sigma}(\sigma^{*2})*O(\sigma^{*2}|G^{*}|\log n/n)* \underline\pi_{\theta}*O(|G^{*}|\log n/n^{2}) \\ &&*\underline\pi_{\vartheta}^{|G^{*}|} [ |G^{*}|\log n/n^{2})]^{|G^{*}|} * [\pi_{\vartheta}(\{-|G^{*}|\log n/n^{2}, |G^{*}|\log n/n^{2}\})]^{n}\\ &=&\exp\{-c_{3}^{\prime\prime}|G^{*}|\log n\}. \quad\text{ (by the conditions imposed on the prior specifications)} \end{array} $$

for some constant $c_{3}^{\prime \prime }$. Thus

$$ m(y)/f^{*}(y)\geq \exp\{-(c_{3}^{\prime}+c_{3}^{\prime\prime})|G^{*}|\log n\} =\exp\{-c_{3}n{\epsilon_{n}^{2}}\}, \text{ with probability tending to 1} $$

(A.4)

for some c₃, where c₃ can be sufficiently small when $n{\epsilon _{n}^{2}}/[|G^{*}|\log n]$ is large enough.

At last, we study the prior probability of set B_n. Due to the prior independence of $\vartheta _{i}^{\prime }$s, π(B_n) = π[Bin(n − 1 −|G^∗|,p) > (δ)|G^∗|], where p ≤ (1/n)^1+u. By Lemma A.2,

$$ \pi(B_{n}) \leq \exp\{ -c_{4}\delta |G^{*}|\log n\} $$

(A.5)

for some c₄. Combine results (A.2), (A.3), (A.4) and (A.5), and we apply Lemma A.3 to get the posterior consistency result that

$$ \pi(B_{n}\cup C_{n}|y) \rightarrow^{p} 0, $$

given a sufficient large constants δ and n𝜖_n/|G^∗|log n. □

Proof 3 (Proof of Theorem 3.1).

Consider y = 𝜃 + d, where $\theta _{i}^{*}\equiv 0$for all i and error d is order statistics of standard normal variables, i.e. the density of d is $f(d)=n!\prod \phi (d_{i})1(d_{1}\leq d_{2},\dots ,d_{n-1}\leq d_{n})$andϕ denotes the standard normal density. The prior of 𝜃 follows $\pi (\theta )=\pi _{1}(\theta _{1}){\prod }_{i=2}^{n}\pi _{t,s}(\theta _{i}-\theta _{i-1})$, where $\pi _{\lambda _{1}}$is the density of N(0,λ₁), and π_t,s is the density of t distribution with tiny scale parameter satisfying − log s ≍ log n, i.e. conditions in Corollary 2.1 holds, and we consider the misspecified posterior of form π(𝜃|D_n) = exp{−(y − 𝜃)²/2}π(𝜃).

Define $\mu \in \mathbb {R}^{n}$asμ_i = 0 for all 1 ≤ i ≤ k = 3n/4, and μ_i = Z_0.25/2 for i > k where Z_0.25 is the right 25% quantile of standard normal distribution, thus ∥μ − 𝜃^∗∥² ≍ n.

Let Δ𝜃 be any vector such that ∥Δ𝜃∥² ≤ M log n. Then

$$ -\log\left( \frac{\pi(\mu+{\Delta}\theta)}{\pi(\theta^{*}+{\Delta}\theta)}\right) =-\log\left( \frac{\pi_{t,s}(Z_{0.25}/2+{\Delta}\theta_{k})}{\pi_{t,s}({\Delta}\theta_{k})}\right) = O(-\log s) = O(\log n). $$

And

$$ \begin{array}{@{}rcl@{}} &&\log\left( \frac{\exp\{-(y-\mu-{\Delta}\theta)^{2}/2\}}{\exp\{-(y-\theta^{*}-{\Delta}\theta)^{2}/2\}}\right) =[(y-\theta^{*}-{\Delta}\theta)^{2}-(y-\mu-{\Delta}\theta)^{2}]/2\\ &=&\frac{1}{2}\sum\limits_{i=k+1}^{n}[(y_{i}-{\Delta}\theta_{i})^{2}-(y_{i}-Z_{0.25}/2-{\Delta}\theta_{i})^{2}]=\frac{1}{2}\sum\limits_{i=k+1}^{n}[(y_{i}-{\Delta}\theta_{i})Z_{0.25}-\frac{Z_{0.25}^{2}}{4}]\\ &\geq &\frac{1}{2}\left[(\|y_{k+1:n}\|_{1}-\sqrt{nM\log n/4})Z_{0.25}-\frac{nZ_{0.25}^{2}}{16} \right]\geq cn, \end{array} $$

for some positive constant c given sufficiently large n, where the inequalities above hold since y_n ≥ y_n− 1⋯ ≥ y_k+ 1, and y_k+ 1 ≈ Z_0.25 with high probability, due to large sample empirical quantile theory.

Combining the above two results, we have that the posterior density satisfies π(μ + Δ𝜃|D_n) ≫ π(𝜃^∗ + Δ𝜃|D_n) for any ∥Δ𝜃∥² ≤ M log n with high probability. Therefore, more posterior mass is distributed within the $\sqrt {M\log n}$-radius ball centered at μ than at the true parameter 𝜃^∗. □

Proof 4 (Proof of Theorem 3.2).

The proof of this theorem is quite similar to the proof of Theorem 2.1 and 2.2. We define the same testing function as in the proof of Theorem 2.1 and 2.2, and define the following two sets:

$$ \begin{array}{@{}rcl@{}} &&C_{n}=\{\theta: \|\theta-\theta^{*}\|\leq M\sqrt n\sigma^{*}\epsilon_{n}, (1-\epsilon_{n})/(1+\epsilon_{n})<\sigma^{2}/\sigma^{*2}<(1+\epsilon_{n})/(1-\epsilon_{n})\}^{c}\backslash B_{n},\\ &&B_{n}=\{\theta: \text{Among all} \ \{\theta_{i}-\theta_{i-1}\}_{i=2}^{n} , \text{ there are at least} \delta \text{ of them are greater than } \sigma\epsilon_{n}/n\}. \end{array} $$

Using the same arguments, one can still establish exponential separation results (A.2) and (A.3).

To establish (A.4), we notice that

$$ \begin{array}{@{}rcl@{}} &&\frac{m(y)}{f^{*}(y)}=\frac{{\int}_{\sigma^{2}}{\int}_{\theta} \sigma^{*n}\exp\{-\|\theta^{*}-\theta+\varepsilon\|^{2}/\sigma^{2}\} d\theta d\sigma^{2}} {\sigma^{n}\exp\{-\|\varepsilon\|^{2}/\sigma^{*2}\}}\\ &\geq&{\int}_{\sigma^{2}}{\int}_{\theta}\exp\left\{-\frac{\|\theta^{*}-\theta\|^{2}}{\sigma^{2}} -2\frac{(\theta^{*}-\theta)^{T}\varepsilon}{\sigma^{2}}+\frac{\|\varepsilon\|^{2}}{\sigma^{*2}}-\frac{\|\varepsilon\|^{2}}{\sigma^{2}}-n\log(\sigma/\sigma^{*})\right\} \pi(\theta,\sigma^{2})d\theta d\sigma^{2}\\ &\geq& \pi(\max\{|\theta_{i}-\theta_{i}^{*}|\}/\sigma\leq \log n/n, 0\leq\sigma^{2}-\sigma^{*2}\leq \sigma^{*2}\log n/n)\exp\{-c_{3}^{\prime}\log n \} \end{array} $$

for some constant $c_{3}^{\prime }$ and

$$ \begin{array}{@{}rcl@{}} &&\pi(\max\{|\theta_{i}-\theta_{i}^{*}|\}/\sigma\leq \log n/n, 0\leq\sigma^{2}-\sigma^{*2}\leq\sigma^{*2} \log n/n)\\ &\geq &\sum\limits_{r}\pi(\max\{|\theta_{r(1)}-\theta_{r(1)}^{*}|, |\theta_{r(i)}-\theta_{r(i-1)}|\}/\sigma\leq |G^{*}|\log n/n^{2}, 0\\ &\leq&\sigma^{2}-\sigma^{*2}\leq\sigma^{*2} \log n/n|r)\pi(r).\\ \end{array} $$

This ensures (A.4).

As for the prior probability of B_n, if the scale parameter for the t distribution is sufficiently small, i.e. s = n^−w for some large w and ${\int }_{\pm \epsilon _{n}/n^{2}}\pi _{t,s}(x)dx\geq 1-1/n^{1+u}$ for some sufficiently large u where π_t,s denotes the t density function with scale parameter s, then for any ranking r,

$$ \pi(B_{n}|r)\geq 1-\pi(\max\{\theta_{r(i)}-\theta_{r(i-1)}\}\leq \sigma\epsilon_{n}/n^{2} |r) \geq 1-(1-1/n^{1+u})^{n} \approx n^{-u}. $$

This hence implies that − log(π(B_n)) ≥ u log n. □