Local polynomial expectile regression

Abstract

This paper studies local polynomial estimation of expectile regression. Expectiles and quantiles both provide a full characterization of a (conditional) distribution function, but have each their own merits and inconveniences. Local polynomial fitting as a smoothing technique has a major advantage of being simple, allowing for explicit expressions and henceforth advantages when doing inference theory. The aim of this paper is twofold: to study in detail the use of local polynomial fitting in the context of expectile regression and to contribute to the important issue of bandwidth selection, from theoretical and practical points of view. We discuss local polynomial expectile regression estimators and establish an asymptotic normality result for them. The finite-sample performance of the estimators, combined with various bandwidth selectors, is investigated in a simulation study. Some illustrations with real data examples are given.

Appendix

1.1 A.1 Proof of Theorem 1

The proof of this theorem is similar in setup as the one provided by Fan et al. (1994) to study nonparametric regression based on i.i.d. observations. The main idea of the proof is to approximate the quantity to be minimized in (11) by a quadratic function whose minimizer is asymptotically normal, and then to show that $(\widehat{\tau }_\omega (x),\widehat{\tau }_\omega ^{(1)} (x), \cdots , \widehat{\tau }_\omega ^{(p)} (x) )^{\text{ T }}$ lies close enough to that minimizer to share the latter’s asymptotic behaviour. The convexity lemma (Pollard 1991) plays a role in the above approximation. We give the details of the proof below.

Recall that, for x a given point, $\beta _0=\tau _\omega (x), \beta _1=\tau _\omega ^{(1)}(x),\cdots , \beta _p=\frac{\tau _\omega ^{(p)}(x)}{p!}$ and $\widehat{\beta }_0=\widehat{\tau }_\omega (x),\widehat{\beta }_1=\widehat{\tau }_\omega ^{(1)}(x),\cdots ,\widehat{\beta }_p=\frac{\widehat{\tau }_\omega ^{(p)}(x)}{p!}$ with $(\widehat{\beta }_0,\cdots ,\widehat{\beta }_p)$ minimizing

$$\begin{aligned} \sum _{i=1}^n Q_\omega \left( Y_i-\sum _{j=0}^p\beta _j(X_i-x)^j\right) K\left( \frac{X_i-x}{h}\right) . \end{aligned}$$

Let

$$\begin{aligned} \begin{array}{l} K_i=K\left( \frac{X_i-x}{h}\right) \\ \mathbf {Z}_i=\left( 1,\frac{X_i-x}{h},\left( \frac{X_i-x}{h}\right) ^2,\ldots ,\left( \frac{X_i-x}{h}\right) ^p\right) ^{\text{ T }} \\ Y_i^*=Y_i-\tau _\omega (x)-\tau _\omega ^{(1)}(x)(X_i-x)-\cdots -\frac{\tau _\omega ^{(p)}(x)}{p!}(X_i-x)^p \\ \widehat{\varvec{\theta }}=\sqrt{nh}\left( \widehat{\beta }_0-\tau _\omega (x),\cdots ,h^p\left( \widehat{\beta }_p-\frac{\tau ^{(p)}_\omega (x)}{p!}\right) \right) ^{\text{ T }} . \end{array} \end{aligned}$$

For $(\theta _0,\ldots ,\theta _p)^{\text{ T }} =\varvec{\theta }\in \mathbb {R}^{p+1}$, $\widehat{\varvec{\theta }}$ minimizes the function

$$\begin{aligned} G_n(\varvec{\theta })&=\sum _{i=1}^n\left[ Q_\omega \left( Y^*_i-\varvec{\theta }^{\text{ T }} \frac{\varvec{Z}_i}{\sqrt{nh}}\right) -Q_\omega (Y_i^*)\right] K_i. \end{aligned}$$

Note that the function $G_n(\varvec{\theta })$ is convex in $\varvec{\theta }$ (the second derivative is $\ge 0$ for all $\varvec{\theta }$). It is sufficient to prove that this function converges pointwise to its conditional expectation, since it follows from the convexity lemma of Pollard (1991) that the convergence is also uniform on any compact set of $\varvec{\theta }$.

We next approximate $G_n(\cdot )$ by a quadratic function whose minimizing value has an asymptotic normal distribution. Two terms contribute to the approximation. One is a quadratic function obtained via a Taylor expansion of the expected value, and the other term is random and linear in $\varvec{\theta }$. Write

$$\begin{aligned} G_n(\varvec{\theta })=\mathrm {E}_{Y|X}[G_n(\varvec{\theta })|\mathcal {X}] -\frac{2}{\sqrt{nh}}\left( \sum _{i=1}^nL_\omega (Y_i^*)\varvec{Z}_iK_i-\mathrm {E}_{Y|X} \left[ L_\omega (Y_i^*)|X_i\right] \varvec{Z}_iK_i\right) ^{\text{ T }} \varvec{\theta } + R_n(\varvec{\theta }) \end{aligned}$$

(A.1)

with

$$\begin{aligned} R_n(\varvec{\theta })=G_n(\varvec{\theta })-\mathrm {E}_{Y|X}[G_n(\varvec{\theta })|\mathcal {X}]+\frac{2}{\sqrt{nh}}\left( \sum _{i=1}^n L_\omega (Y^*_i)\varvec{Z}_iK_i-\mathrm {E}_{Y|X}[L_\omega (Y_i^*)|X_i]\varvec{Z}_iK_i \right) ^{\text{ T }} \varvec{\theta }. \end{aligned}$$

(A.2)

Let M be a real number such that the interval $[-M,M]$ contains the support of K. By Taylor expansion,

$$\begin{aligned} \tau _\omega (X_i)=\tau _\omega (x)+\tau _\omega ^{(1)}(x)(X_i-x)+\cdots +\frac{\tau _\omega ^{(p+1)}(x)}{(p+1)!}(X_i-x)^{p+1}+\xi _{n,i} \quad \text {for}\quad |X_i-x|\le Mh \end{aligned}$$

with $\xi _{n,i}=o_P\left( |X_i-x|^{p+1}\right) =o_P(h^{p+1})$ holds uniformly as $X_i\rightarrow x$, i.e. $\max _{\{i:|X_i-x|\le Mh\}}||\xi _{n,i}||_{\infty }=o_P(h^{p+1})$ since $\tau _{\omega }(.)$ has a continuous $(p+2)$th derivative.

We have

$$\begin{aligned}&\mathrm {E}_{Y|X}[G_n(\varvec{\theta })|\mathcal {X}]\\&\quad =\mathrm {E}_{Y|X}\left. \left[ \sum _{i=1}^n\left[ Q_\omega \left( Y^*_i-\varvec{\theta }^{\text{ T }} \frac{\varvec{Z}_i}{\sqrt{nh}}\right) -Q_\omega (Y_i^*)\right] K_i\right| \mathcal {X}\right] \\&\quad =\sum _{i=1}^n\left. \left[ \mathrm {E}_{Y|X}\left[ Q_\omega \left( Y^*_i-\varvec{\theta }^{\text{ T }} \frac{\varvec{Z}_i}{\sqrt{nh}}\right) \right| X_i\right] -\mathrm {E}_{Y|X}\left[ Q_\omega (Y_i^*)|X_i\right] \right] K_i\\&\quad =\sum _{i=1}^n\left. \left( \mathrm {E}_{Y|X}\left[ Q_\omega \left( Y_i-\tau _\omega (x)-\tau _\omega ^{(1)}(x)(X_i-x)-\cdots -\frac{\tau _\omega ^{(p)}(x)}{p!}(X_i-x)^p-\varvec{\theta }^{\text{ T }} \frac{\varvec{Z}_i}{\sqrt{nh}}\right) \right| X_i\right] \right. \\&\quad \left. -\mathrm {E}_{Y|X}\left[ Q_\omega \left. \left( Y_i-\tau _\omega (x)-\tau _\omega ^{(1)}(x)(X_i-x)-\cdots -\frac{\tau _\omega ^{(p)}(x)}{p!}(X_i-x)^p\right) \right| X_i\right] \right) K_i\\&\quad =\sum _{i=1}^n\left. \left[ \varphi \left( \tau _\omega (X_i)-\tau _\omega (x)-\tau _\omega ^{(1)}(x)(X_i-x)-\cdots -\frac{\tau _\omega ^{(p)}(x)}{p!}(X_i-x)^p -\varvec{\theta }^{\text{ T }} \frac{\varvec{Z}_i}{\sqrt{nh}}\right| X_i\right) \right. \\&\quad \left. -\varphi \left. \left( \tau _\omega (X_i)-\tau _\omega (x)-\tau _\omega ^{(1)}(x)(X_i-x)-\cdots -\frac{\tau _\omega ^{(p)}(x)}{p!}(X_i-x)^p\right| X_i\right) \right] K_i\\&\quad =-\sum _{i=1}^n\varphi ^{(1)} \left. \left( \tau _\omega (X_i)-\tau _\omega (x)-\tau _\omega ^{(1)}(x)(X_i-x)-\cdots -\frac{\tau _\omega ^{(p)}(x)}{p!}(X_i-x)^p \right| X_i\right) \frac{\varvec{\theta }^{\text{ T }} \varvec{Z}_i}{\sqrt{nh}} K_i\\&\quad +\frac{1}{2}\sum _{i=1}^n \varphi ^{(2)} \left. \left( \tau _\omega (X_i)-\tau _\omega (x)-\tau _\omega ^{(1)}(x)(X_i-x)-\cdots -\frac{\tau _\omega ^{(p)}(x)}{p!}(X_i-x)^p\right| X_i\right) \\&\quad \frac{(\varvec{\theta }^{\text{ T }} \varvec{Z}_i)^2}{nh}K_i(1+o_P(1)). \end{aligned}$$

Moreover,

$$\begin{aligned}&\varphi ^{(2)} \left. \left( \tau _\omega (X_i)-\tau _\omega (x)-\tau _\omega ^{(1)}(x)(X_i-x)-\cdots -\frac{\tau _\omega ^{(p)}(x)}{p!}(X_i-x)^p\right| X_i\right) \\&\quad = \varphi ^{(2)} \left. \left( \tau _\omega (x)+\tau _\omega ^{(1)}(x)(X_i-x)+\cdots +\frac{\tau _\omega ^{(p+1)}(x)}{(p+1)!}(X_i-x)^{p+1}+\xi _{n,i} \right. \right. \\&\quad \left. \left. - \tau _\omega (x)-\tau _\omega ^{(1)}(x)(X_i-x)-\cdots -\frac{\tau _\omega ^{(p)}(x)}{p!}(X_i-x)^p)\right| X_i\right) \\&\quad =\varphi ^{(2)} \left. \left( \frac{\tau _\omega ^{(p+1)}(x)}{(p+1)!}(X_i-x)^{p+1}+\xi _{n,i}\right| X_i\right) \\&\quad =\varphi ^{(2)} \left. \left( 0\right| X_i\right) +O_P(h^{p+1}). \end{aligned}$$

It follows that

$$\begin{aligned} \mathrm {E}_{Y|X}[G_n(\varvec{\theta })|\mathcal {X}]= & {} \frac{-2}{\sqrt{nh}}\sum _{i=1}^n\mathrm {E}_{Y|X}\left[ L_\omega \left( Y_i^*|X_i\right) \right] (\varvec{\theta }^{\text{ T }} \varvec{Z}_i)K_i \\&+\frac{1}{2nh}\varvec{\theta }^{\text{ T }} \left( \sum _{i=1}^n K_i \gamma (\omega ,X_i)\varvec{Z}_i\varvec{Z}_i^{\text{ T }} \right) \varvec{\theta }(1+o_P(1)). \end{aligned}$$

Thus, we have

$$\begin{aligned} \frac{1}{nh}\sum _{i=1}^n K_i\gamma (\omega ,X_i)\varvec{Z}_i\varvec{Z}_i^{\text{ T }} = \frac{1}{nh}\sum _{i=1}^n K_i\gamma (\omega ,X_i) \, \begin{pmatrix} 1 &{}\frac{X_i-x}{h}&{}\left( \frac{X_i-x}{h}\right) ^2&{}\cdots &{}\left( \frac{X_i-x}{h}\right) ^p\\ \frac{X_i-x}{h} &{}\left( \frac{X_i-x}{h}\right) ^2&{}\left( \frac{X_i-x}{h}\right) ^3&{}\cdots &{}\left( \frac{X_i-x}{h}\right) ^{p+1}\\ \vdots &{}\vdots &{}\vdots &{}\ddots &{}\vdots \\ \left( \frac{X_i-x}{h}\right) ^{p}&{} \left( \frac{X_i-x}{h}\right) ^{p+1} &{} \left( \frac{X_i-x}{h}\right) ^{p+2} &{} \cdots &{} \left( \frac{X_i-x}{h}\right) ^{2p} \end{pmatrix}. \end{aligned}$$

Denoting $\widetilde{S}_{n,j}=\frac{1}{nh}\sum _{i=1}^n\gamma (\omega ,X_i)\left( \frac{X_i-x}{h}\right) ^jK\left( \frac{X_i-x}{h}\right)$, for $j=0,1,\ldots ,2p$, it follows from the fact that K has bounded support (see e.g. Fan and Gijbels 1996) that

$$\begin{aligned}&\widetilde{S}_{n,j}= \mathrm {E}_X[\widetilde{S}_{n,j}]+ O_P\left( \sqrt{\text {Var}_X(\widetilde{S}_{n,j})}\right) \\&\quad \mathrm {E}_X[\widetilde{S}_{n,j}]=\frac{n}{nh}\int \gamma (\omega ,v)f_X(v)\left( \frac{v-x}{h}\right) ^j K\left( \frac{v-x}{h}\right) {\mathrm{d}}v =\int \gamma (\omega ,x+uh)f_X(x+uh)u^jK\left( u\right) {\mathrm{d}}u\\&\quad =f_X(x)\gamma (\omega ,x)\mu _j + o(1), \end{aligned}$$

where the last equality comes from the dominated convergence theorem where we assumed that $h\rightarrow 0$ and $f_X(.)$ is continuous in a neighbourhood of x.

A similar argument leads to

$$\begin{aligned}&\text {Var}_X[\widetilde{S}_{n,j}]= \mathrm {E}_X[\widetilde{S}_{n,j}^2]-\mathrm {E}_X[\widetilde{S}_{n,j}]^2 \le \frac{n}{n^2h^2}\int \left( \gamma (\omega ,v)\left( \frac{v-x}{h}\right) ^jK\left( \frac{v-x}{h}\right) \right) ^2f_X(v){\mathrm{d}}v \\&\quad =\frac{1}{nh}\int \left( \gamma (\omega ,x+uh)u^jK\left( u\right) \right) ^2f_X(x+uh){\mathrm{d}}u =o(1). \end{aligned}$$

With this result and the definition of the matrix $\mathbf {S}$, we have

$$\begin{aligned} \frac{1}{nh}\sum _{i=1}^n K_i\gamma (\omega ,X_i)\varvec{Z}_i\varvec{Z}_i^{\text{ T }}&=\gamma (\omega ,x)f_X(x)\mathbf {S}+o_P(1). \end{aligned}$$

We then obtain that

$$\begin{aligned} \mathrm {E}_{Y|X}[G_n(\varvec{\theta })|\mathcal {X}]= & {} \frac{-2}{\sqrt{nh}}\sum _{i=1}^n\mathrm {E}_{Y|X}\left[ L_\omega \left( Y_i^*|X_i\right) \right] (\varvec{\theta }^{\text{ T }} \varvec{Z}_i)K_i \\&+\frac{1}{2nh}\varvec{\theta }^{\text{ T }} \left( \sum _{i=1}^n K_i \gamma (\omega ,X_i)\varvec{Z}_i\varvec{Z}_i^{\text{ T }} \right) \varvec{\theta }(1+o_P(1))\\= & {} \frac{-2}{\sqrt{nh}}\sum _{i=1}^n\mathrm {E}_{Y|X}\left[ L_\omega \left( Y_i^*|X_i\right) \right] (\varvec{\theta }^{\text{ T }} \varvec{Z}_i)K_i +\frac{1}{2}\varvec{\theta }^{\text{ T }} \gamma (\omega ,x)f_X(x)\mathbf {S}\varvec{\theta }(1+o_P(1)). \end{aligned}$$

Next we show that $R_n(\varvec{\theta })=o_P(1)$ (for the definition of $R_n(\varvec{\theta })$ see (A.2)). We start by rewriting and approximating this quantity as follows:

$$\begin{aligned}&R_n(\varvec{\theta })\nonumber \\&=G_n(\varvec{\theta })-\mathrm {E}_{Y|X}[G_n(\varvec{\theta })|\mathcal {X}]+\frac{2}{\sqrt{nh}}\left( \sum _{i=1}^n L_\omega (Y^*_i)\varvec{Z}_iK_i-\mathrm {E}_{Y|X}[L_\omega (Y_i^*)|X_i]\varvec{Z}_iK_i \right) ^{\text{ T }} \varvec{\theta }\nonumber \\&=\sum _{i=1}^n\left[ Q_\omega \left( Y^*_i-\varvec{\theta }^{\text{ T }} \frac{\varvec{Z}_i}{\sqrt{nh}}\right) -Q_\omega (Y_i^*)\right] K_i\nonumber \\&\quad +\frac{2}{\sqrt{nh}}\sum _{i=1}^n\mathrm {E}_{Y|X}\left[ L_\omega \left( Y_i^*|X_i\right) \right] (\varvec{\theta }^{\text{ T }} \varvec{Z}_i)K_i -\frac{1}{2}\varvec{\theta }^{\text{ T }} \gamma (\omega ,x)f_X(x)\mathbf {S}\varvec{\theta }(1+o_P(1))\nonumber \\&\quad +\frac{2}{\sqrt{nh}}\left( \sum _{i=1}^n L_\omega (Y^*_i)\varvec{Z}_iK_i-\mathrm {E}_{Y|X}[L_\omega (Y_i^*)|X_i]\varvec{Z}_iK_i \right) ^{\text{ T }} \varvec{\theta }\nonumber \\&=\sum _{i=1}^n\left[ Q_\omega \left( Y^*_i-\varvec{\theta }^{\text{ T }} \frac{\varvec{Z}_i}{\sqrt{nh}}\right) -Q_\omega (Y_i^*)\right] K_i-\frac{1}{2}\varvec{\theta }^{\text{ T }} \gamma (\omega ,x)f_X(x)\mathbf {S}\theta (1+o_P(1))\nonumber \\&\quad +\frac{2}{\sqrt{nh}}\left( \sum _{i=1}^n L_\omega (Y^*_i)\varvec{Z}_iK_i\right) ^{\text{ T }} \varvec{\theta }\nonumber \\&=\sum _{i=1}^n\left[ Q_\omega \left( Y^*_i-\varvec{\theta }^{\text{ T }} \frac{\varvec{Z}_i}{\sqrt{nh}}\right) -Q_\omega (Y_i^*)+\frac{2}{\sqrt{nh}} L_\omega (Y^*_i)\varvec{\theta }^{\text{ T }} \varvec{Z}_i\right] K_i-\frac{1}{2}\varvec{\theta }^{\text{ T }} \gamma (\omega ,x)f_X(x)\mathbf {S}\theta (1+o_P(1)). \end{aligned}$$

By using Assumption (A3), we obtain

$$\begin{aligned}&\text {Var}_{X,Y}[R_n(\varvec{\theta })]\le n \mathrm {E}_{Y,X}\left[ \left( Q_\omega \left( Y^*_1-\varvec{\theta }^{\text{ T }} \frac{\varvec{Z}_1}{\sqrt{nh}}\right) -Q_\omega (Y_1^*) +\frac{2}{\sqrt{nh}}\left( L_\omega (Y^*_1)\varvec{Z}_1 \right) ^{\text{ T }} \varvec{\theta }\right) ^2\right] K_1^2\\&\quad \le n \int \int \left( Q_\omega \left( y_v^*-\varvec{\theta }^{\text{ T }} \frac{\varvec{z}}{\sqrt{nh}}\right) -Q_\omega (y^*_v) +\frac{2}{\sqrt{nh}}\left( L_\omega (y_v^*)\varvec{z} \right) ^{\text{ T }} \varvec{\theta }\right) ^2\\&\quad H(y|v) {\mathrm{d}}y K^2\left( \frac{v-x}{h}\right) f_X(v){\mathrm{d}}v\\&\quad = o\left( n\int \left( \varvec{\theta }^{\text{ T }} \frac{\varvec{z}}{\sqrt{nh}}\right) ^2 K^2\left( \frac{v-x}{h}\right) f_X(v){\mathrm{d}}v\right) =o(1) \end{aligned}$$

with $\mathbf {z}=\left( 1,\frac{v-x}{h},\left( \frac{v-x}{h}\right) ^2,\cdots ,\left( \frac{v-x}{h}\right) ^p\right) ^{\text{ T }}$ and $y_v^*=y-\tau _\omega (x)-\tau _\omega ^{(1)}(x)(v-x)-\cdots -\frac{\tau _\omega ^{(p)}(x)}{p!}(v-x)^p$.

It follows from the definition of $R_n(\varvec{\theta })$ (in (A.2)) that for any $\varvec{\theta }\in \mathbf {R}^{p+1}$, $\omega \in (0,1)$ and $x \in \mathbb {R}^{p+1}$, $\mathrm {E}_{X,Y}[R_n(\varvec{\theta })]= 0$. Therefore $R_n(\varvec{\theta })=o_P(1)$. Indeed, for any constant $\epsilon >0$ and by the inequality of Chebyshev,

$$\begin{aligned} P\left[ \left| R_n(\varvec{\theta })\right|>\epsilon \right] = P\left[ \left| R_n(\varvec{\theta })-\mathrm {E}_{X,Y}\left[ R_n(\varvec{\theta })\right] \right| >\epsilon \right] \\ \le \frac{1}{\epsilon ^2} \text {Var}_{X,Y}\left[ R_n(\varvec{\theta })\right] =o(1). \end{aligned}$$

For the quantity $G_n(\varvec{\theta })$ in (A.1), we thus obtain

$$\begin{aligned} G_n(\varvec{\theta })&=\mathrm {E}_{Y|X}[G_n(\varvec{\theta })|\mathcal {X}]-\frac{2}{\sqrt{nh}}\left( \sum _{i=1}^nL_\omega (Y_i^*)\varvec{Z}_iK_i-\mathrm {E}_{Y|X}\left[ L_\omega (Y_i^*)|X_i\right] \varvec{Z}_iK_i\right) ^{\text{ T }} \varvec{\theta } + R_n(\varvec{\theta })\\&=\frac{-2}{\sqrt{nh}}\sum _{i=1}^n\mathrm {E}_{Y|X}\left[ L_\omega \left( Y_i^*|X_i\right) \right] (\varvec{\theta }^{\text{ T }} \varvec{Z}_i)K_i +\frac{1}{2}\varvec{\theta }^{\text{ T }} \gamma (\omega ,x)f_X(x)\mathbf {S}\varvec{\theta }(1+o_P(1))\\&\quad -\frac{2}{\sqrt{nh}}\left( \sum _{i=1}^nL_\omega (Y_i^*)\varvec{Z}_iK_i-\mathrm {E}_{Y|X}\left[ L_\omega (Y_i^*)|X_i\right] \varvec{Z}_iK_i\right) ^{\text{ T }} \varvec{\theta } + R_n(\varvec{\theta })\\&=\frac{1}{2}\varvec{\theta }^{\text{ T }} \gamma (\omega ,x)f_X(x)\mathbf {S}\varvec{\theta }-\frac{2}{\sqrt{nh}}\left( \sum _{i=1}^nL_\omega (Y_i^*)\varvec{Z}_iK_i\right) ^{\text{ T }} \varvec{\theta } + r_n(\varvec{\theta })\\&=\frac{1}{2}\varvec{\theta }^{\text{ T }} \gamma (\omega ,x)f_X(x)\mathbf {S}\varvec{\theta }+\mathbf {W}_n^{\text{ T }} \varvec{\theta } + r_n(\varvec{\theta }) \end{aligned}$$

with $r_n(\varvec{\theta })=o_P(1)$ for each fixed $\varvec{\theta }$ and

$$\begin{aligned} \mathbf {W}_n=\frac{-2}{\sqrt{nh}}\left( \sum _{i=1}^nL_\omega (Y_i^*)\varvec{Z}_iK_i\right) . \end{aligned}$$

It easy to see that $\mathbf {W}_n$ has a bounded second moment and hence is stochastically bounded. For $c>0$ and by Assumption (A1) ($\varphi (t|z)$ is bounded), we have, with $\varvec{z}_u=\left( 1,u,u^2,\cdots ,u^p\right) ^{\text{ T }}$, $y_u^*=y-\tau _\omega (x)-\tau _\omega ^{(1)}(x)(hu)-\cdots -\frac{\tau _\omega ^{(p)}(x)}{p!}(hu)^p$, using Hölder’s inequality and Assumption (A3),

$$\begin{aligned} \mathrm {E}_{X,Y}[\mathbf {W}_n\mathbf {W}_n^{\text{ T }} ]= & {} \frac{4}{nh}\mathrm {E}_{X,Y}\left[ \left( \sum _{i=1}^nL_w(Y_i^*)\varvec{Z}_iK_i\right) \left( \sum _{i=1}^nL_w(Y_i^*)\varvec{Z}_iK_i\right) ^{\text{ T }} \right] \\\le & {} \frac{4c}{nh}\mathrm {E}_{X,Y}\left[ \left( \sum _{i=1}^nL_w(Y_i^*)^2K_i^2\varvec{Z}_i\varvec{Z}_i^{\text{ T }} \right) ^{\text{ T }} \right] \\= & {} \frac{4c}{h}\int \int L_w(y_v^*)^2\varvec{z}\varvec{z}^{\text{ T }} f_{Y|X}(y|v){\mathrm{d}}yK^2\left( \frac{v-x}{h}\right) f_X(v){\mathrm{d}}v\\= & {} \frac{4c}{h}\int \int L_w(y-\tau _\omega (x)-\tau _\omega ^{(1)}(x)(v-x)-\cdots -\frac{\tau _\omega ^{(p)}(x)}{p!}(v-x)^p)^2\varvec{z}\varvec{z}^{\text{ T }}\\&\quad f_{Y|X}(y|v){\mathrm{d}}yK^2\left( \frac{v-x}{h}\right) f_X(v){\mathrm{d}}v\\= & {} 4c\int \int L_w(y_u^*)^2f_{Y|X}(y|u){\mathrm{d}}y\varvec{z}_u\varvec{z}_u^{\text{ T }} K^2(u)f_X(u+xh){\mathrm{d}}u\\= & {} 4c\int \int \left| L_w(y_u^*)\right| ^{2}f_{Y|X}(y|u){\mathrm{d}}y\varvec{z}_u\varvec{z}_u^{\text{ T }} K^2(u)f_X(u+xh){\mathrm{d}}u\\\le & {} 4c\int \left( \int \left| L_w(y_u^*)\right| ^{2+\delta }f_{Y|X}(y|u){\mathrm{d}}y\right) ^{\frac{2}{2+\delta }}\varvec{z}_u\varvec{z}_u^{\text{ T }} K^2(u)f_X(u+xh){\mathrm{d}}u\\= & {} O\left( \mathrm {E}_X\left[ (K)^2\varvec{z}_u\varvec{z}_u^{\text{ T }}\right] \right) = O(1) \end{aligned}$$

which also implies that $\mathrm {E}_{Y,X}[\varvec{W}_n]=O(1)$ as a result of Jensen’s inequality.

Note that

$$\begin{aligned} G_n(\varvec{\theta })-\mathbf {W}_n^{\text{ T }} \varvec{\theta } \end{aligned}$$

is a convex function of $\varvec{\theta }$ which converges in probability to the convex function $\frac{1}{2}\varvec{\theta }^{\text{ T }} \gamma (w,x)f_X(x)\mathbf {S}\varvec{\theta }$.

By the convexity lemma, Pollard (1991), for any compact subset $\varLambda \in \mathbb {R}^{p+1}$

$$\begin{aligned} \sum _{\varvec{\theta }\in \varLambda }|r_n(\varvec{\theta })|=o_P(1). \end{aligned}$$

So the quadratic approximation to the convex function $G_n(\varvec{\theta })$ holds uniformly for $\varvec{\theta }$ in any compact set. Then, using the convexity assumption again, the minimizer $\widehat{\varvec{\theta }}$ of $G_n(\varvec{\theta })$ converges in probability to the minimizer of the quadratic function $-(\gamma (\omega ,x)f_X(x)\mathbf {S})^{-1}\mathbf {W}_n$

$$\begin{aligned} \widehat{\varvec{\theta }}+(\gamma (\omega ,x)f_X(x)\mathbf {S})^{-1}\mathbf {W}_n = o_P(1). \end{aligned}$$

In matrix notation, we have

$$\begin{aligned}&\sqrt{nh}\begin{pmatrix} \widehat{\beta }_0-\tau _\omega (x)\\ h(\widehat{\beta }_1-\tau ^{(1)}_\omega (x))\\ \vdots \\ h^p\left( \widehat{\beta }_p-\frac{\tau ^{(p)}_\omega (x)}{p!}\right) \end{pmatrix} - \frac{2}{\sqrt{nh}\gamma (\omega ,x)f_X(x)}\begin{pmatrix}\mu _0 &{} \mu _1 &{} \cdots &{} \mu _p \\ \mu _1 &{} \mu _2 &{} \cdots &{} \mu _{p+1} \\ \vdots &{} \vdots &{} \ddots &{} \vdots \\ \mu _p &{} \mu _{p+1} &{} \cdots &{} \mu _{2p}\end{pmatrix}^{-1}\left( \sum _{i=1}^nL_\omega (Y_i^*)\varvec{Z}_iK_i\right) =o_P(1)\\ \end{aligned}$$

with

$$\begin{aligned} \mathbf {W}_n^*&=\left( \sum _{i=1}^nL_\omega (Y_i^*)\varvec{Z}_iK_i\right) =\sum _{i=1}^n L_\omega (Y_i^*)\begin{pmatrix} 1\\ \frac{X_i-x}{h}\\ \vdots \\ \left( \frac{X_i-x}{h}\right) ^p \end{pmatrix}K\left( \frac{X_i-x}{h}\right) \end{aligned}$$

and hence

$$\begin{aligned} \mathbf {S}^{-1}\mathbf {W}_n^*&=\sum _{i=1}^nL_\omega (Y_i^*)\begin{pmatrix}\mu _0 &{} \mu _1 &{} \cdots &{} \mu _p \\ \mu _1 &{} \mu _2 &{} \cdots &{} \mu _{p+1} \\ \vdots &{} \vdots &{} \ddots &{} \vdots \\ \mu _p &{} \mu _{p+1} &{} \cdots &{} \mu _{2p}\end{pmatrix}^{-1} \begin{pmatrix} 1\\ \frac{X_i-x}{h}\\ \vdots \\ \left( \frac{X_i-x}{h}\right) ^p \end{pmatrix}K\left( \frac{X_i-x}{h}\right) . \end{aligned}$$

So

$$\begin{aligned}&\sqrt{nh}\begin{pmatrix} \widehat{\beta }_0-\tau _\omega (x)\\ h(\widehat{\beta }_1-\tau ^{(1)}_\omega (x))\\ \vdots \\ h^p\left( \widehat{\beta }_p-\frac{\tau ^{(p)}_\omega (x)}{p!}\right) \end{pmatrix} \nonumber \\&- \frac{2}{\sqrt{nh}\gamma (\omega ,x)f_X(x)}\sum _{i=1}^nL_\omega (Y_i^*)\begin{pmatrix}\mu _0 &{} \mu _1 &{} \cdots &{} \mu _p \\ \mu _1 &{} \mu _2 &{} \cdots &{} \mu _{p+1} \\ \vdots &{} \vdots &{} \ddots &{} \vdots \\ \mu _p &{} \mu _{p+1} &{} \cdots &{} \mu _{2p}\end{pmatrix}^{-1} \begin{pmatrix} 1 \\ \frac{X_i-x}{h}\\ \vdots \\ \left( \frac{X_i-x}{h}\right) ^p \end{pmatrix}K\left( \frac{X_i-x}{h}\right) =o_P(1). \end{aligned}$$

(A.3)

The $(j+1)$th component (for $j=0,1,\ldots ,p$) of the above equality is

$$\begin{aligned}&\sqrt{nh}\left( h^j\left( \frac{\widehat{\tau }^{(j)}_\omega (x)}{j!}-\frac{\tau ^{(j)}_\omega (x)}{j!}\right) -\frac{2}{nh\gamma (\omega ,x)f_X(x)}\sum _{i=1}^nL_\omega (Y_i^*)(\mathbf {S}^{-1}\varvec{Z}_{i})_{j+1}K_i\right) =o_P(1)\\&\quad \sqrt{nh}\left( h^j\left( \frac{\widehat{\tau }^{(j)}_\omega (x)}{j!}-\frac{\tau ^{(j)}_\omega (x)}{j!} \right) -V_{n,j}\right) =o_P(1), \end{aligned}$$

denoting $V_{n,j}=\frac{U_{n,j}}{\gamma (\omega ,x)f_X(x)\text {det}(\mathbf {S})}$ and $U_{n,j}=2(nh)^{-1}\sum _{i=1}^nL_\omega (Y_i^*)(\text {adj}(\mathbf {S})\mathbf {Z}_{i})_{j+1}K_i$, where $\text {det}(\mathbf {S})$ is the determinant of $\mathbf {S}$ and $\text {adj}(\mathbf {S})$ is the adjugate matrix of $\mathbf {S}$.

Equivalently, for any $\epsilon >0$, we have

$$\begin{aligned} \mathrm {E}_{Y,X}\left[ \mathbf {P}\left. \left( \sqrt{nh}\left| h^j\left( \frac{\widehat{\tau }^{(j)}_\omega (x)}{j!}-\frac{\tau ^{(j)}_\omega (x)}{j!}\right) -V_{n,j}\right| \ge \epsilon \right| \mathcal {X} \right) \right] =o_P(1). \end{aligned}$$

This implies that

$$\begin{aligned} \mathbf {P}\left. \left( \sqrt{nh}\left| h^j\left( \frac{\widehat{\tau }^{(j)}_\omega (x)}{j!}-\frac{\tau ^{(j)}_\omega (x)}{j!}\right) -V_{n,j}\right| \ge \epsilon \right| \mathcal {X} \right) =o_P(1). \end{aligned}$$

Hence, the conditional asymptotic normality follows from that of $U_{n,j}$, which is established with the help of Lemmas 1 and 2 stated in Section A.2. The proofs of the lemmas are provided in Section S6 of the Supplementary Material part. $\square$

1.2 A.2 Two lemmas

Lemma 1

Under the assumptions of Theorem 1, we have

$$\begin{aligned} \mathrm {E}_{Y|X}[U_{n,j}|\mathcal {X}]=dh^{p+1}(1+o_P(1)) \qquad \qquad \text{ and } \qquad \qquad \text {Var}_{Y|X}[U_{n,j}|\mathcal {X}]=\frac{v^2}{nh}(1+o_P(1)) \end{aligned}$$

where $U_{n,j}=2(nh)^{-1}\sum _{i=1}^nL_\omega (Y_i^*)(\text {adj}(\mathbf {S})\mathbf {Z}_{i})_{j+1}K_i$,

$$\begin{aligned} d&=\frac{1}{(p+1)!}\tau _\omega ^{(p+1)}(x)\gamma (\omega ,x) (\text {adj}(\mathbf {S})\mathbf {c}_p)_{j+1}f_X(x)+\frac{1}{(p+2)!}\tau _\omega ^{(p+2)}(x)h\gamma (\omega ,x) (\text {adj}(\mathbf {S})\widetilde{\mathbf {c}}_p)_{j+1}f_X(x)\nonumber \\&\quad +\frac{1}{(p+1)!}\tau _\omega ^{(p+1)}(x)h\gamma (\omega ,x) (\text {adj}(\mathbf {S})\widetilde{\mathbf {c}}_p)_{j+1}f_X^{(1)}(x) \end{aligned}$$

(A.4)

$$\begin{aligned}&\text{ and } \qquad \qquad v^2=f_X(x)\int \left( 2L_\omega (y-\tau _\omega (x))\right) ^{2}f_{Y|X}(y|x){\mathrm{d}}y\int (\text {adj}(\mathbf {S})\mathbf {z}_v)_{j+1}^2 K^{2}\left( v\right) {\mathrm{d}}v \end{aligned}$$

(A.5)

with $\mathbf {z}_v=\left( 1,v,v^2,\cdots ,v^p\right) ^{\text{ T }}$.

Lemma 2

Under Assumptions (A1)—(A5), we have

$$\begin{aligned} P\left[ \sqrt{nh}\frac{(U_{n,j}-dh^{p+1})}{v}\le t |\mathcal {X}\right] =\varPhi (t)+o_P(1), \end{aligned}$$

with d and v define as in (A.4) and (A.5), respectively.