Cross-sample entropy estimation for time series analysis: a nonparametric approach

Abstract

Cross-sample entropy (CSE) allows to analyze the association level between two time series that are not necessarily stationary. The current criteria to estimate the CSE are based on the normality assumption, but this condition is not necessarily satisfied in reality. Also, CSE calculation is based on a tolerance and an embedding dimension parameter, which are defined rather subjectively. In this paper, we define a new way of estimating the CSE with a nonparametric approach. Specifically, a residual-based bootstrap-type estimator is considered for long-memory and heteroskedastic models. Subsequently, the established criteria are redefined for the approach of interest for generalization purposes. Finally, a simulation study serves to evaluate the performance of this estimation technique. An application to foreign exchange market data before and after the 1999 Asian financial crisis was considered to study the synchrony level of the CAD/USD and SGD/USD foreign exchange rate time series. A bootstrap-type method allowed to obtain a more realistic estimation of the cross-sample entropy (CSE) statistics. Specifically, estimated CSE was slightly different than that obtained in previous studies, but for both periods the synchrony level using CSE between the time series was higher after the 1999 Asian financial crisis.

Appendices

Appendix A: Cross-sample entropy variance

The CSE variance is obtained by Richman and Moorman [33]. Let $\Upsilon =(N-m)^{2}{\bar{A}}_m(r)$ and fixing $\Psi =(N-m)^{2}{\bar{B}}_m(r)$ (i.e., $\Psi$ is not a random variable), where ${\bar{A}}_m$ and ${\bar{B}}_m$ are defined in (1.2), we get

$$\begin{aligned} CP=\frac{{\bar{A}}_m(r)}{{\bar{B}}_m(r)}=\frac{\Upsilon }{\Psi }, \end{aligned}$$

which corresponds to an estimation of conditional probability about coincidence between u and v for $m+1$ points, given that coincidences between u and v for m points exist. According to Liu et al. [26], we get

$$\begin{aligned} \Upsilon =\displaystyle {\sum _{i,j=1}^{N-m}} U_{ij}, \end{aligned}$$

where

$$\begin{aligned} U_{ij}=\left\{ \begin{array}{lcc} 1, &{} \text {if} &{} d_{\infty }(x_{i,m+1},y_{i,m+1}) \le r, \\ 0, &{} \text {if} &{} d_{\infty }(x_{i,m+1},y_{i,m+1}) > r. \end{array}\right. \end{aligned}$$

Considering $\sigma _{CP}^{2}$ as the variance of the estimator CP, we get

$$\begin{aligned} \sigma _{CP}^{2}= & {} Var\bigg (\frac{\Upsilon }{\Psi }\bigg )\\= & {} \frac{1}{\Psi ^{2}}\displaystyle {\sum _{i=1}^{N-m}\sum _{j=1}^{N-m}\sum _{k=1}^{N-m}\sum _{\ell =1}^{N-m}}Cov(U_{i,j},U_{k,\ell }), \end{aligned}$$

as for $i=k$ and $j=\ell$ the covariance between $U_{i,j}$ and $U_{k,\ell }$ is $Cov(U_{i,j},U_{k,\ell })=Var(U_{k,\ell })=CP(1-CP)$. If $i \ne k$ and $j \ne \ell$, $U_{ij}$ and $U_{k,\ell }$ are independent and $Cov(U_{i,j},U_{k,\ell })=0$.

If vectors overlap, i.e., if $\min \{ |i-k|,|j-\ell | \} \le m$, we get

$$\begin{aligned}&Cov(U_{i,j},U_{k,\ell })=U_{i,j} U_{k,\ell }-CP^{2}\\&\quad =\left\{ \begin{array}{lcc} 1-CP^{2}, &{} \text {if exists coincidence in }m+1\text { points}, \\ -CP^{2}, &{} \quad \text {otherwise}. \end{array}\right. \end{aligned}$$

Then, $\sigma _{CP}^{2}$ is estimated by

$$\begin{aligned} {\widehat{\sigma }}_{CP}^{2}=\frac{CP(1-CP)}{\Psi }+\frac{1}{\Psi ^{2}}\big [M_{\Upsilon }-M_{\Psi }(CP)^{2}\big ], \end{aligned}$$

where $M_{\Upsilon }$ is the number of pairs of matching templates of length $m+1$ that vectors $x_{i,m+1}$ and $y_{j,m+1}$ overlapped and $M_{\Psi }$ is the number of pairs of matching templates of length m that vectors $x_{i,m}$ and $y_{j,m}$ overlapped. Using the Delta method, the CSE variance $\sigma _{CSE}^{2}$ is estimated as

$$\begin{aligned} \sigma _{CSE}^{2}\approx [g'(CP)]^{2}\sigma _{CP}^{2}=\frac{\sigma _{CP}^{2}}{CP^{2}}, \end{aligned}$$

where $g(t)=-\log (t)$, $t>0$.

Appendix B: Whittle estimator

The methodology to approximate the maximum likelihood estimator (MLE) is based on the calculation of the periodogram by means of the fast Fourier transform and the use of the approximation of the Gaussian log-likelihood function due to Whittle [45] and Bisaglia and Guégan [7]. Suppose that the sample vector $\mathbf{Y}=(y_1,y_2,\ldots ,y_n)$ is normally distributed with zero mean and autocovariance given by

$$\begin{aligned} \gamma (k-j)= & {} \int _{-\pi }^{\pi }f(\lambda )e^{i\lambda (k-j)}d\lambda , \end{aligned}$$

$k,j=1,\ldots ,n$, where $f(\lambda )$ is the spectral density of the ARFIMA model (2.4) defined by

$$\begin{aligned} f(\lambda )=\frac{\sigma ^2}{2\pi }\left( 2\sin \frac{\lambda }{2}\right) ^{-2d}\frac{|\Theta (e^{-i\lambda })|^2}{|\Phi (e^{-i\lambda })|^2} \end{aligned}$$

and is associated with the parameter set $\mathbf{\Omega }=(\alpha _1,\ldots ,\alpha _p,d,\beta _1,\ldots ,\beta _q)$ of (2.4). The log-likelihood function of the process Y is given by

$$\begin{aligned} L(\varvec{\Omega }|\mathbf{Y})=-\frac{1}{2n}[\log |\varvec{\Delta }|-\mathbf{Y}^{\top }\varvec{\Delta }^{-1} \mathbf{Y}]. \end{aligned}$$

(5.1)

where $\varvec{\Delta }=[\gamma (k-j)]$. For calculating (5.1), two asymptotic approximations are made for the terms $\log |\varvec{\Delta }|$ and $\mathbf{Y}^{\top }\varvec{\Delta }^{-1} \mathbf{Y}$ to obtain

$$\begin{aligned} L(\varvec{\Omega }|\mathbf{Y})\approx -\frac{1}{4\pi }\left[ \int _{-\pi }^{\pi }\log [2\pi f(\lambda )]d\lambda + \int _{-\pi }^{\pi }\frac{I(\lambda )}{f(\lambda )}d\lambda \right] ,\nonumber \\ \end{aligned}$$

(5.2)

as $n\rightarrow \infty$, where

$$\begin{aligned} I(\lambda )=\frac{1}{2\pi n}\Bigg |\sum _{j=1}^{n}y_j e^{i\lambda j}\Bigg |^2 \end{aligned}$$

is the periodogram. Thus, a discrete version of (5.2) is the Riemann approximation of the integral given by

$$\begin{aligned} L(\varvec{\Omega }|\mathbf{Y})\approx -\frac{1}{2n}\left[ \sum _{j=1}^{n}\log f(\lambda _j) + \sum _{j=1}^{n}\frac{I(\lambda _j)}{f(\lambda _j)}\right] ,\nonumber \\ \end{aligned}$$

(5.3)

where $\lambda _j=2\pi j/n$ are the Fourier frequencies. To find the estimator of parameter vector $\varvec{\Omega }$, we use the minimization of $L(\varvec{\Omega }|\mathbf{Y})$ produced by the nlm function of R software [31]. This nonlinear minimization function minimizes $L(\varvec{\Omega }|\mathbf{Y})$ using a Newton-type algorithm. Under regularity conditions according to Theorem 2 of Contreras-Reyes and Palma [11], the Whittle estimator $\widehat{\varvec{\Omega }}$ that maximizes the log-likelihood function given in (5.3) is consistent [5] and distributed normally [13].

Appendix C: Quasi-maximum likelihood estimator

Let $\{ y_t \}$ be a FIGARCH(p, d, q)-(a, b) process described in (2.6–2.8), with a parameter set $\varvec{\Omega }=(\varvec{\Omega }_{1},\varvec{\Omega }_{2})^{\top }$, where $\varvec{\Omega }_{1}=(\phi _{1},\ldots ,\phi _{p},d,\theta _{1},\ldots ,\theta _{q})^{\top }$ and $\varvec{\Omega }_{2}=(\alpha _{0},\ldots ,\alpha _{a},\beta _{1},\ldots ,\beta _{b})^{\top }$, and $\{ y_t \}$ is associated with the sample vector $\mathbf{Y}=(y_1,y_2,\ldots ,y_n)$. An approximated MLE or quasi-MLE ${\widehat{\Omega }}$ for $\{ y_t \}$ is obtained by maximizing the conditional log-likelihood function

$$\begin{aligned} \begin{aligned} L(\varvec{\Omega }|\mathbf{Y}) = -\frac{1}{2n}\sum _{t=1}^{n}\Bigg [ \log \sigma _{t}^{2}+\frac{\varepsilon _{t}^{2}}{\sigma _{t}^{2}} \Bigg ]. \end{aligned} \end{aligned}$$

(5.4)

Let $\widehat{\varvec{\Omega }}=\widehat{\varvec{\Omega }}_n$ be the exact point where the function (5.4) is maximized. Under regularity conditions discussed in Baillie et al. [5] and Palma [30], we have that $\widehat{\varvec{\Omega }}$ is a consistent estimator and

$$\begin{aligned} \sqrt{n}(\widehat{\varvec{\Omega }}-\varvec{\Omega }_{0})\mathop {\buildrel {\mathcal {D}}\over \longrightarrow }N_{\kappa }(0,\varvec{\Sigma }^{-1}) \end{aligned}$$

as $n\rightarrow \infty$, where ${\mathcal {D}}$ denotes convergence in distribution, $\kappa =p+q+a+b+2$ and $\varvec{\Sigma }=\text {diag}(\varvec{\Sigma }_{1},\varvec{\Sigma }_{2})$ with

$$\begin{aligned} \varvec{\Sigma }_{1}= & {} {\mathbb {E}}\Bigg [ \frac{1}{\sigma _{t}^{2}}\frac{\partial \varepsilon _{t}}{\partial \varvec{\Omega }_{1}}\frac{\partial \varepsilon _{t}}{\partial \varvec{\Omega }_{1}^{\top }}\\&+\frac{1}{2\sigma _{t}^{4}}\frac{\partial \sigma _{t}^{2}}{\partial \varvec{\Omega }_{1}}\frac{\partial \sigma _{t}^{2}}{\partial \varvec{\Omega }_{1}^{\top }} \Bigg ], \end{aligned}$$

and

$$\begin{aligned} \varvec{\Sigma }_{2}={\mathbb {E}}\Bigg [ \frac{1}{2\sigma _{t}^{4}}\frac{\partial \sigma _{t}^{2}}{\partial \varvec{\Omega }_{2}}\frac{\partial \sigma _{t}^{2}}{\partial \varvec{\Omega }_{2}^{\top }} \Bigg ]. \end{aligned}$$