Logistic regression diagnostics in ridge regression

Abstract

The adverse effects of multicollinearity and unusual observations are seen in logistic regression and attention had been given in the literature to each of these problems separately. However, multicollinearity and unusual observations can arise simultaneously in logistic regression. The objective of this paper is to propose the statistics for detecting the unusual observations in an ill-conditioned data set under the ridge logistic estimator. A numerical example and two Monte Carlo simulation studies are used to illustrate the methodology. The present investigation shows that ridge logistic estimation copes with unusual observations by downweighting their influence.

Appendices

Appendix A

Theorem 1

The $\varvec{\hat{\beta }}^{(1)}(k)$ when the $i\hbox {th}$ observation is deleted is $\varvec{\hat{\beta }}_{(i)}^{(1)}(k)= \varvec{\hat{\beta }}^{(1)}(k)-\frac{(X^{\prime }V^{(0)}X+kI)^{-1}x_{i}}{ 1-h_{ii}(k)}v_{ii}^{(0)}[z_{i}^{(0)}-x_{i}^{\prime }\varvec{\hat{\beta }} ^{(1)}(k)]$ where $\sqrt{v_{ii}^{(0)}}[z_{i}^{(0)}-x_{i}^{\prime } \varvec{\hat{\beta }}^{(1)}(k)]$ is the $i\hbox {th}$ ridge logistic regression residual and $h_{ii}(k)$ is the $i\hbox {th}$ ridge logistic regression leverage value.

Proof

Let $Q=(V^{(0)})^{1/2}X$, $\mathbf {w}=(V^{(0)})^{1/2}\mathbf {z}^{(0)}$ and $\mathbf {s}=\mathbf {y}-\hat{\mathbf {\pi }}^{(0)}$. Then $\varvec{\hat{ \beta }}^{(1)}(k)$ and $\varvec{\hat{\beta }}^{(1)}$ given by (4) and (5) can be reexpressed as

$$\begin{aligned} \varvec{\hat{\beta }}^{(1)}(k)= \left( Q^{\prime }Q+kI\right) ^{-1}Q^{\prime }Q \varvec{\hat{\beta }}^{(1)} \end{aligned}$$

and

$$\begin{aligned} \varvec{\hat{\beta }}^{(1)}= & {} \left( Q^{\prime }Q\right) ^{-1}Q^{\prime }\mathbf {w} \\= & {} \varvec{\hat{\beta }}^{(0)}+ \left( Q^{\prime }Q\right) ^{-1}X^{\prime }\mathbf {s.} \end{aligned}$$

Let $\varvec{\hat{\beta }}_{(i)}^{(1)}(k)$ and $\varvec{\hat{\beta }} _{(i)}^{(1)}$ denote the ML and ridge logistic estimators of $\varvec{ \beta }$ based on the observations without the $i\hbox {th}$ observation, respectively. Then, we have

$$\begin{aligned} \varvec{\hat{\beta }}_{(i)}^{(1)}(k)= \left( Q_{(i)}^{\prime }Q_{(i)}+kI\right) ^{-1}Q_{(i)}^{\prime }Q_{(i)}\varvec{\hat{\beta }}_{(i)}^{(1)} \end{aligned}$$

(10)

and

$$\begin{aligned} \varvec{\hat{\beta }}_{(i)}^{(1)}=\varvec{\hat{\beta }} ^{(0)}+\left( Q_{(i)}^{\prime }Q_{(i)}\right) ^{-1}X_{(i)}^{\prime }\mathbf {s}_{(i)}. \end{aligned}$$

By applying the Sherman–Morrison–Woodbury theorem (see Schott 2005) to $(Q_{(i)}^{\prime }Q_{(i)})^{-1}$ and after algebraic simplifications, $\varvec{\hat{\beta }}_{(i)}^{(1)}$ can be written as

$$\begin{aligned} \varvec{\hat{\beta }}_{(i)}^{(1)}= & {} \varvec{\hat{\beta }} ^{(0)}+\left( Q_{(i)}^{\prime }Q_{(i)}\right) ^{-1}X_{(i)}^{\prime }\mathbf {s}_{(i)}= \varvec{\beta }^{(0)}+\left( Q^{\prime }Q-q_{i}q_{i}^{\prime }\right) ^{-1}\left( X^{\prime }\mathbf {s}-x_{i}s_{i}\right) \\= & {} \varvec{\hat{\beta }}^{(0)}+\left[ (Q^{\prime }Q)^{-1}+\frac{(Q^{\prime }Q)^{-1}q_{i}q_{i}^{\prime }\left( Q^{\prime }Q\right) ^{-1}}{1-h_{i}}\right] (X^{\prime } \mathbf {s}-x_{i}s_{i}) \\= & {} \varvec{\hat{\beta }}^{(0)}+(Q^{\prime }Q)^{-1}X^{\prime }\mathbf {s}- \frac{(Q^{\prime }Q)^{-1}q_{i}}{1-h_{ii}}\left( \frac{s_{i}}{\sqrt{v_{ii}}} -q_{i}^{\prime }(Q^{\prime }Q)^{-1}X^{\prime }\mathbf {s}\right) \end{aligned}$$

where $q_{i}^{\prime }=\sqrt{v_{ii}^{(0)}}x_{i}^{\prime }$ is the ith row vector of the Q matrix and $h_{ii}=q_{i}^{\prime }(Q^{\prime }Q)^{-1}q_{i}=v_{ii}^{(0)}x_{i}^{\prime }(X^{\prime }V^{(0)}X)^{-1}x_{i}$ is the $i\hbox {th}$ diagonal element of

$$\begin{aligned} H=(V^{(0)})^{1/2}X(X^{\prime }V^{(0)}X)^{-1}X^{\prime }(V^{(0)})^{1/2}. \end{aligned}$$

Noting that $(Q^{\prime }Q)^{-1}X^{\prime }\mathbf {s}=\varvec{\hat{\beta } }^{(1)}-\varvec{\hat{\beta }}^{(0)}$, $\varvec{\hat{\beta }} _{(i)}^{(1)}$ can be written as

$$\begin{aligned} \varvec{\hat{\beta }}_{(i)}^{(1)} =\varvec{\hat{\beta }}^{(1)}-\frac{ \left( X^{\prime }V^{(0)}X\right) ^{-1}x_{i}}{1-h_{ii}}v_{ii}^{(0)}\left( \frac{s_{i}}{ v_{ii}^{(0)}}+x_{i}^{\prime }\varvec{\hat{\beta }}^{(0)}\mathbf {-} x_{i}^{\prime }\varvec{\hat{\beta }}^{(1)}\right) \end{aligned}$$

(11)

where $\frac{s_{i}}{v_{ii}^{(0)}}+x_{i}^{\prime }\varvec{\hat{\beta }} ^{(0)}$ is the $i\hbox {th}$ element of working response $\mathbf {z}^{(0)}$. Then $\sqrt{v_{ii}^{(0)}}(\frac{s_{i}}{v_{ii}^{(0)}}+x_{i}^{\prime }\varvec{ \hat{\beta }}^{(0)}\mathbf {-}x_{i}^{\prime }\varvec{\hat{\beta }}^{(1)})$ presents the $i\hbox {th}$ standardized residual (Pearson residual) after one step iteration. $\square$

Using the expression

$$\begin{aligned} \left( Q_{(i)}^{\prime }Q_{(i)}+kI\right) ^{-1}= & {} \left[ (Q^{\prime }Q+kI)-q_{i}q_{i}^{\prime }\right] ^{-1} \\= & {} \left( Q^{\prime }Q+kI\right) ^{-1}+\frac{\left( Q^{\prime }Q+kI\right) ^{-1}q_{i}q_{i}^{\prime }(Q^{\prime }Q+kI)^{-1}}{1-h_{ii}(k)} \end{aligned}$$

where $h_{ii}(k)=q_{i}^{\prime }(Q^{\prime }Q+kI)^{-1}q_{i}=v_{ii}^{(0)}x_{i}^{\prime }(X^{\prime }V^{(0)}X+kI)^{-1}x_{i}$ is the $i\hbox {th}$ diagonal element of the matrix $H(k)=(V^{(0)})^{1/2}X(X^{\prime }V^{(0)}X+kI)^{-1}X^{\prime }(V^{(0)})^{1/2}$, we obtain

$$\begin{aligned} \left( Q_{(i)}^{\prime }Q_{(i)}+kI\right) ^{-1}Q_{(i)}^{\prime }Q_{(i)}= & {} \left[ \left( Q^{\prime }Q+kI\right) -q_{i}q_{i}^{\prime }\right] ^{-1} \left( Q^{\prime }Q-q_{i}q_{i}^{\prime }\right) \nonumber \\= & {} \left( Q^{\prime }Q+kI\right) ^{-1}Q^{\prime }Q \nonumber \\&+\frac{(Q^{\prime }Q+kI)^{-1}q_{i}q_{i}^{\prime }(Q^{\prime }Q+kI)^{-1}Q^{\prime }Q}{1-h_{ii}(k)} \nonumber \\&-(Q^{\prime }Q+kI)^{-1}q_{i}q_{i}^{\prime }-\frac{h_{ii}(k)(Q^{\prime }Q+kI)^{-1}q_{i}q_{i}^{\prime }}{1-h_{ii}(k)} \nonumber \\= & {} (Q^{\prime }Q+kI)^{-1}Q^{\prime }Q \nonumber \\&+\frac{(Q^{\prime }Q+kI)^{-1}q_{i}q_{i}^{\prime }(Q^{\prime }Q+kI)^{-1}Q^{\prime }Q}{1-h_{ii}(k)} \nonumber \\&-\frac{(Q^{\prime }Q+kI)^{-1}q_{i}q_{i}^{\prime }}{1-h_{ii}(k)}. \end{aligned}$$

(12)

Substituting results (11) and (12) into (10) and after algebraic simplifications, we obtain

$$\begin{aligned} \varvec{\hat{\beta }}_{(i)}^{(1)}(k)= & {} (Q^{\prime }Q+kI)^{-1}Q^{\prime }Q \varvec{\hat{\beta }}^{(1)}+\frac{(Q^{\prime }Q+kI)^{-1}q_{i}q_{i}^{\prime }(Q^{\prime }Q+kI)^{-1}Q^{\prime }Q\varvec{ \hat{\beta }}^{(1)}}{1-h_{ii}(k)} \\&-\frac{(Q^{\prime }Q+kI)^{-1}q_{i}q_{i}^{\prime }\varvec{\hat{\beta }} ^{(1)}}{1-h_{ii}(k)}-\left[ \frac{1}{1-h_{ii}}+\frac{h_{ii}(k)}{1-h_{ii}(k)}-\frac{ h_{ii}}{1-h_{ii}}\right] \\&\times (Q^{\prime }Q+kI)^{-1}q_{i}\left( \frac{s_{i}}{\sqrt{v_{ii}^{(0)}}} +q_{i}^{\prime }\varvec{\hat{\beta }}^{(0)}-q_{i}^{\prime }\varvec{ \hat{\beta }}^{(1)}\right) \\= & {} (Q^{\prime }Q+kI)^{-1}Q^{\prime }Q\varvec{\hat{\beta }}^{(1)}+\frac{ (Q^{\prime }Q+kI)^{-1}q_{i}q_{i}^{\prime }(Q^{\prime }Q+kI)^{-1}Q^{\prime }Q \varvec{\hat{\beta }}^{(1)}}{1-h_{ii}(k)} \\&+\frac{(Q^{\prime }Q+kI)^{-1}q_{i}}{1-h_{ii}(k)}\left( \frac{s_{i}}{\sqrt{ v_{ii}^{(0)}}}+q_{i}^{\prime }\varvec{\hat{\beta }}^{(0)}-q_{i}^{\prime } \varvec{\hat{\beta }}^{(1)}+q_{i}^{\prime }\varvec{\hat{\beta }}^{(1)}\right) \\= & {} (Q^{\prime }Q+kI)^{-1}Q^{\prime }Q\varvec{\hat{\beta }}^{(1)}+\frac{ (Q^{\prime }Q+kI)^{-1}q_{i}}{1-h_{ii}(k)} \\&\times \left( \frac{s_{i}}{\sqrt{v_{ii}^{(0)}}}+q_{i}^{\prime }\varvec{\hat{ \beta }}^{(0)}-q_{i}^{\prime }(Q^{\prime }Q+kI)^{-1}Q^{\prime }Q\varvec{ \hat{\beta }}^{(1)}\right) \\= & {} \varvec{\hat{\beta }}^{(1)}(k)-\frac{(Q^{\prime }Q+kI)^{-1}q_{i}}{ 1-h_{ii}(k)}\left[ \frac{s_{i}}{\sqrt{v_{ii}^{(0)}}}+q_{i}^{\prime }\varvec{ \hat{\beta }}^{(0)}-q_{i}^{\prime }\varvec{\hat{\beta }}^{(1)}(k)\right] . \end{aligned}$$

Since $\frac{s_{i}}{v_{ii}^{(0)}}+x_{i}^{\prime }\varvec{\hat{\beta }} ^{(0)}$ is the $i\hbox {th}$ element of working response $\mathbf {z}^{(0)}$, the proof is completed when Q and $q_{i}$ are written as defined earlier.

Appendix B

Theorem 2

The change in Pearson chi-square statistic for the ridge logistic estimator is

$$\begin{aligned} \Delta \chi _{i}^{2}(k)= & {} \frac{1}{(1-h_{ii}(k))^{2}}\left[ \sqrt{v_{ii}^{(0)}} (z_{i}^{(0)}-x_{i}^{\prime }\varvec{\hat{\beta }}^{(1)}(k))\right] ^{2}-\frac{2k }{1-h_{ii}(k)}\varvec{\hat{\beta }}^{(1)\prime }(k) \\&\times \left( X^{\prime }V^{(0)}X+kI\right) ^{-1}x_{i} v_{ii}^{(0)}\left( z_{i}^{(0)}-x_{i}^{\prime }\varvec{ \hat{\beta }}^{(1)}(k)\right) -\frac{1}{(1-h_{ii}(k))^{2}}x_{i}^{\prime } \\&\times \left( X^{\prime }V^{(0)}X+kI\right) ^{-1}X^{\prime }V^{(0)} X(X^{\prime }V^{(0)}X+kI)^{-1}x_{i}v_{ii}^{(0)} \\&\times \left[ \sqrt{v_{ii}^{(0)}}(z_{i}^{(0)}-x_{i}^{\prime }\varvec{ \hat{\beta }}^{(1)}(k))\right] ^{2}. \end{aligned}$$

Proof

Let $\chi ^{2}(k)$ and $\chi _{(i)}^{2}(k)$ denote the Pearson chi-square statistic based on the full data set and the Pearson chi-square statistic when observation i is deleted for the ridge logistic estimator. Then, by using the linear regression-like approximation of Pregibon (1981), we get the following expressions which are similar to residual sum of squares in linear regression:

$$\begin{aligned} \chi ^{2}(k)=\left( \mathbf {w}-Q\varvec{\hat{\beta }}^{(1)}(k)\right) ^{\prime }\left( \mathbf {w}-Q\varvec{\hat{\beta }}^{(1)}(k)\right) , \end{aligned}$$

and

$$\begin{aligned} \chi _{(i)}^{2}(k)= & {} \left( \mathbf {w}_{(i)}-Q_{(i)}\varvec{\hat{\beta }} _{(i)}^{(1)}(k)\right) ^{\prime }\left( \mathbf {w}_{(i)}-Q_{(i)}\varvec{\hat{\beta }} _{(i)}^{(1)}(k)\right) \\= & {} \mathbf {w}_{(i)}^{\prime }\mathbf {w}_{(i)}-2\varvec{\hat{\beta }} _{(i)}^{(1)\prime }(k)Q_{(i)}^{\prime }\mathbf {w}_{(i)}+\varvec{\hat{ \beta }}_{(i)}^{(1)\prime }(k)Q_{(i)}^{\prime }Q_{(i)}\varvec{\hat{\beta }} _{(i)}^{(1)}(k). \end{aligned}$$

The expression $\varvec{\hat{\beta }}_{(i)}^{(1)}(k)$ given by Theorem 1 can also be written as

$$\begin{aligned} \varvec{\hat{\beta }}_{(i)}^{(1)}(k)=\varvec{\hat{\beta }}^{(1)}(k)- \frac{(Q^{\prime }Q+kI)^{-1}q_{i}}{1-h_{ii}(k)}\left[ w_{i}-q_{i}^{\prime } \varvec{\hat{\beta }}^{(1)}(k)\right] . \end{aligned}$$

By using this expression, we get

$$\begin{aligned} \varvec{\hat{\beta }}_{(i)}^{(1)\prime }(k)Q_{(i)}^{\prime }w_{(i)}= & {} \varvec{\hat{\beta }}^{(1)\prime }(k)Q^{\prime }\mathbf {w}-\varvec{ \hat{\beta }}^{(1)\prime }(k)q_{i}w_{i}-\frac{1}{1-h_{ii}(k)} (w_{i}-q_{i}^{\prime }\varvec{\hat{\beta }}^{(1)}(k)) \nonumber \\&\times q_{i}^{\prime }\varvec{\hat{\beta }}^{(1)}(k)+\frac{h_{ii}(k)}{ 1-h_{ii}(k)}(w_{i}-q_{i}^{\prime }\varvec{\hat{\beta }}^{(1)}(k))w_{i} \end{aligned}$$

(13)

and

$$\begin{aligned} \varvec{\hat{\beta }}_{(i)}^{(1)\prime }(k)Q_{(i)}^{\prime }Q_{(i)} \varvec{\hat{\beta }}_{(i)}^{(1)}(k)= & {} \varvec{\hat{\beta }} ^{(1)\prime }(k)Q^{\prime }Q\varvec{\hat{\beta }}^{(1)}(k)+\frac{ 1-2h_{ii}(k)}{(1-h_{ii}(k))^{2}}\varvec{\hat{\beta }}^{(1)\prime }(k)q_{i}q_{i}^{\prime }\varvec{\hat{\beta }}^{(1)}(k) \nonumber \\&-\frac{1-2h_{ii}(k)}{(1-h_{ii}(k))^{2}}\varvec{\hat{\beta }}^{(1)\prime }(k)q_{i}w_{i}-\frac{1-2h_{ii}(k)}{(1-h_{ii}(k))^{2}}w_{i}q_{i}^{\prime } \varvec{\hat{\beta }}^{(1)}(k) \nonumber \\&-\frac{h_{ii}(k)^{2}}{(1-h_{ii}(k))^{2}}w_{i}^{2}+\frac{2k}{1-h_{ii}(k)} \varvec{\hat{\beta }}^{(1)\prime }(k)(Q^{\prime }Q+kI)^{-1} \nonumber \\&\times q_{i}w_{i}-\frac{2k}{1-h_{ii}(k)}\varvec{\hat{\beta }} ^{(1)\prime }(k)(Q^{\prime }Q+kI)^{-1}q_{i}q_{i}^{\prime }\varvec{\hat{ \beta }}^{(1)}(k)\nonumber \\&+\frac{1}{(1-h_{ii}(k))^{2}}q_{i}^{\prime }(Q^{\prime }Q+kI)^{-1}Q^{\prime }Q(Q^{\prime }Q+kI)^{-1}q_{i}\nonumber \\&\times \left( w_{i}-q_{i}^{\prime }\varvec{\hat{\beta }}^{(1)}(k)\right) ^{2} \end{aligned}$$

(14)

Hence, we obtain $\chi _{(i)}^{2}(k)$ from Eqs. (13) and (14) as $\square$

$$\begin{aligned} \chi _{(i)}^{2}(k)= & {} \mathbf {w}_{(i)}^{\prime }\mathbf {w}_{(i)}-2 \varvec{\hat{\beta }}^{(1)\prime }(k)Q^{\prime }\mathbf {w}+[2-\frac{ 1-2h_{ii}(k)}{(1-h_{ii}(k))^{2}}+\frac{2h_{ii}(k)}{1-h_{ii}(k)}]\varvec{ \hat{\beta }}^{(1)\prime }(k)q_{i}w_{i} \\&+\left[ \frac{2}{1-h_{ii}(k)}-\frac{1-2h_{ii}(k)}{(1-h_{ii}(k))^{2}} \right] w_{i}q_{i}^{\prime }\varvec{\hat{\beta }}^{(1)}(k)\\&-\left[ \frac{2}{1-h_{ii}(k)} -\frac{1-2h_{ii}(k)}{(1-h_{ii}(k))^{2}}\right] \\&\times \varvec{\hat{\beta }}^{(1)\prime }(k)q_{i}q_{i}^{\prime } \varvec{\hat{\beta }}^{(1)}(k)- \left[ \frac{2h_{ii}(k)}{1-h_{ii}(k)}+\frac{ h_{ii}(k)^{2}}{(1-h_{ii}(k))^{2}}\right] w_{i}^{2} \\&+\varvec{\hat{\beta }} ^{(1)\prime }(k)Q^{\prime }Q\varvec{\hat{\beta }}^{(1)}(k)+\frac{2k}{1-h_{ii}(k)}\varvec{\hat{\beta }}^{(1)\prime } (k)(Q^{\prime }Q+kI)^{-1}q_{i}w_{i}\\&-\frac{2k}{1-h_{ii}(k)}\varvec{\hat{\beta }} ^{(1)\prime }(k)(Q^{\prime }Q+kI)^{-1} \\&\times q_{i}q_{i}^{\prime }\varvec{\hat{\beta }}^{(1)}(k)+\frac{1}{ (1-h_{ii}(k))^{2}}q_{i}^{\prime }(Q^{\prime }Q+kI)^{-1}Q^{\prime }Q(Q^{\prime }Q+kI)^{-1}q_{i} \\&\times \left( w_{i}-q_{i}^{\prime }\varvec{\hat{\beta }}^{(1)}(k)\right) ^{2}. \end{aligned}$$

$\chi _{(i)}^{2}(k)$ reduces to

$$\begin{aligned} \chi _{(i)}^{2}(k)= & {} \mathbf {w}_{(i)}^{\prime }\mathbf {w}_{(i)}-2 \varvec{\hat{\beta }}^{(1)\prime }(k)Q^{\prime }\mathbf {w}+\frac{1}{ (1-h_{ii}(k))^{2}}\varvec{\hat{\beta }}^{(1)\prime }(k)q_{i}w_{i}+\frac{1 }{(1-h_{ii}(k))^{2}} \\&\times w_{i}q_{i}^{\prime }\varvec{\hat{\beta }}^{(1)}(k)-\frac{1}{ (1-h_{ii}(k))^{2}}\varvec{\hat{\beta }}^{(1)\prime }(k)q_{i}q_{i}^{\prime }\varvec{\hat{\beta }}^{(1)}(k)+\frac{h_{ii}(k)(h_{ii}(k)-2)}{ (1-h_{ii}(k))^{2}}w_{i}^{2} \\&+\varvec{\hat{\beta }}^{(1)\prime }(k)Q^{\prime }Q\varvec{\hat{\beta }}^{(1)}(k)+\frac{2k}{1-h_{ii}(k)}\varvec{\hat{\beta }}^{(1)\prime }(k)(Q^{\prime }Q+kI)^{-1}q_{i}w_{i} \\&-\frac{2k}{1-h_{ii}(k)}\varvec{\hat{\beta }}^{(1)\prime }(k)(Q^{\prime }Q+kI)^{-1}q_{i}q_{i}^{\prime }\varvec{\hat{\beta }}^{(1)}(k) \\&+\frac{1}{(1-h_{ii}(k))^{2}}q_{i}^{\prime }(Q^{\prime }Q+kI)^{-1}Q^{\prime }Q(Q^{\prime }Q+kI)^{-1}q_{i}\left( w_{i}-q_{i}^{\prime }\varvec{\hat{\beta }} ^{(1)}(k)\right) ^{2}. \end{aligned}$$

The difference $\Delta \chi _{i}^{2}(k)=\chi ^{2}(k)-\chi _{(i)}^{2}(k)$, after algebraic simplifications, equals

$$\begin{aligned} \Delta \chi _{i}^{2}(k)= & {} \frac{1}{(1-h_{ii}(k))^{2}}(w_{i}-q_{i}^{\prime } \varvec{\hat{\beta }}^{(1)}(k))^{2}-\frac{2k}{1-h_{ii}(k)}\varvec{ \hat{\beta }}^{(1)\prime }(k)(Q^{\prime }Q+kI)^{-1}q_{i} \\&\times \left( w_{i}-q_{i}^{\prime }\varvec{\hat{\beta }}^{(1)}(k)\right) {-}\frac{1}{ (1-h_{ii}(k))^{2}}q_{i}^{\prime }(Q^{\prime }Q{+}kI)^{-1}Q^{\prime }Q(Q^{\prime }Q{+}kI)^{-1}q_{i} \\&\times \left( w_{i}-q_{i}^{\prime }\varvec{\hat{\beta }}^{(1)}(k)\right) ^{2}. \end{aligned}$$

Using the definition of $w_{i}$, $q_{i}$ and Q, the proof is completed.