A computationally efficient approach to estimating species richness and rarefaction curve

Abstract

In ecological and educational studies, estimators of the total number of species and rarefaction curve based on empirical samples are important tools. We propose a new method to estimate both rarefaction curve and the number of species based on a ready-made numerical approach such as quadratic optimization. The key idea in developing the proposed algorithm is based on nonparametric empirical Bayes estimation incorporating an interpolated rarefaction curve through quadratic optimization with linear constraints based on g-modeling in Efron (Stat Sci 29:285–301, 2014). Our proposed algorithm is easily implemented and shows better performances than existing methods in terms of computational speed and accuracy. Furthermore, we provide a criterion of model selection to choose some tuning parameters in estimation procedure and the idea of confidence interval based on asymptotic theory rather than resampling method. We present some asymptotic result of our estimator to validate the efficiency of our estimator theoretically. A broad range of numerical studies including simulations and real data examples are also conducted, and the gain that it produces has been compared to existing methods.

Appendix

1.1 A.1 Proof of Lemma  1

(i):

The proof of $E({\hat{c}})=c$:

The given conditions $\mathbf{f} = (\int \{1-(1-p)^m\} d {\hat{H}}_J(p))_{1\le m \le s}$ and $\mathbf{f}$ is an unbiased estimator of $A(m) = \int \{1-(1-p)^m\}dH_J(p)$ are equivalent to

$$\begin{aligned} \sum _{j=1}^J p_j^m \{E({\hat{w}}_j^*) - w_j \} \equiv \sum _{j=1}^J p_j^{m-1} \delta _j =0 \end{aligned}$$

for $1\le m \le s$ and $\delta _j =p_j \{E({\hat{w}}_j^*) - w_j \}$. This can be expressed

$$\begin{aligned} \mathcal{P} \varvec{\delta } =\mathbf{0}. \end{aligned}$$

where $\mathcal{P} =(p_{mj})_{1\le m \le s, 1\le j \le J} =(p_j^m)_{0\le m \le s-1, 1\le j \le J}$ is a Vandermonde matrix and $\varvec{\delta } = (\delta _j)_{1\le j \le J}$. The Vandemonde matrix has a full rank, so if $J \le s$, then the rank of $\mathcal{P}$ is J which leads to ${\varvec{\delta }} =\mathbf{0}$. We have $p_j E({\hat{w}}_j^*) = p_j w_j$ resulting in $E({\hat{w}}_j^*) = w_j$ due to $p_j > 0$. Finally, we prove $E({\hat{c}}) = E(\sum _{j=1}^J {\hat{w}}_j^*) = \sum _{j=1}^J w_j =c$.

(ii):

The proof of $\hbox {var}({\hat{c}})=O \left( c \frac{\xi _1}{\xi _M} \right)$:

For a sequence $m = [\theta s]$ which is the integer part of $\theta s$ for some $0< \theta <1$, we can take a $\delta >0$ such that $\min _{ p \in [\epsilon _1, \epsilon _2] } \{1-(1-p)^m\} \ge (1-(1-\epsilon _1)^m) \ge \delta >0$ from $(\mathbf{A2})$ in Assumption A. Since we only need to consider the first M coordinates from (A3), so we define M dimensional vectors such as $\hat{\mathbf{w}}^*_M =({\hat{w}}_1^*, \ldots , {\hat{w}}_M^*)^{\mathrm{T}}$ and $\mathbf{1}_M=(1,1,\ldots , 1)^{\mathrm{T}}$ as well as covariance matrix ${\varvec{\varSigma }}_M = \hbox {var}(\hat{\mathbf{w}}_M^*)$. We also define $\mathbf{a}_M =(a_1,\ldots , a_M)^{\mathrm{T}}$ for $a_i = 1-(1-p_i)^m$ for $1\le i \le M$. Then we have

$$\begin{aligned} \delta {\hat{c}} = \delta \mathbf{1}_M^{\mathrm{T}}\hat{\mathbf{w}}_M^* \le \mathbf{a}_M ^{\mathrm{T}}\hat{\mathbf{w}}_M^* \le \mathbf{1}_M^{\mathrm{T}}\hat{\mathbf{w}}_M^* = {\hat{c}}, \end{aligned}$$

Let $(\mathbf{e}_j, \xi _j)$ be the eigenvector and eigenvalue of ${\varvec{\varSigma }}_{M}$ where $\xi _1 \ge \cdots \ge \xi _M >0$. We have $\hbox {var}(f_m) = \mathbf{a}_M^{\mathrm{T}}{\varvec{\varSigma }}_M \mathbf{a}_M$ where $f_m=\mathbf{a}_M ^{\mathrm{T}}\hat{\mathbf{w}}_M^*$ and $\hbox {var}({\hat{c}}) = \mathbf{1}_M^{\mathrm{T}}{\varvec{\varSigma }}_M \mathbf{1}_M$. Using these and $a_i \ge \delta$ leading to $\mathbf{1}^{\mathrm{T}}_M \mathbf{1}_M \le \frac{1}{\delta ^2} \mathbf{a}^{\mathrm{T}}_M \mathbf{a}_M$, we derive

$$\begin{aligned} \frac{\hbox {var}({\hat{c}})}{\hbox {var}(f_m)}= & {} \frac{\mathbf{1}_M^{\mathrm{T}}{\varvec{\varSigma }}_M \mathbf{1}_M}{\mathbf{a}_M^{\mathrm{T}}{\varvec{\varSigma }}_M \mathbf{a}_M} = \frac{\xi _1 }{\xi _M}\frac{\mathbf{1}^{\mathrm{T}}_M \mathbf{1}_M}{ \mathbf{a}^{\mathrm{T}}_M \mathbf{a}_M } \le \frac{\xi _1}{\xi _{M}} \frac{1}{\delta ^2} =O\left( \frac{\xi _1}{\xi _M}\right) . \end{aligned}$$

Since we have

$$\begin{aligned} \hbox {var}(f_m) \le c \max _{1\le j \le s} \left\{ 1- \frac{{s-m \atopwithdelims ()j}}{ {s \atopwithdelims ()j}}\right\} ^2 \sum _{j=1}^s g(j) \le c \{1-(1-\epsilon _2)^s\} \le c, \end{aligned}$$

we prove $\hbox {var}( {\hat{c}}) \le \hbox {var}(f_m) O(\frac{\xi _M}{\xi _1}) =O \left( c \frac{\xi _1}{\xi _M} \right)$.