Abstract
Random-effects meta-analysis serves to integrate the results of multiple studies with methods such as moment estimation and likelihood estimation duly proposed. These existing methods are based on asymptotic normality with respect to the number of studies. However, the test and interval estimation deviate from the nominal significance level when integrating a small number of studies. Although a method for constructing more conservative intervals has been recently proposed, the exact distribution of test statistic for the overall treatment effect is not well known. In this paper, we provide an almost-exact distribution of the test statistic in random-effects meta-analysis and propose the test and interval estimation using the almost-exact distribution. Simulations demonstrate the accuracy of estimation and application to existing meta-analysis using the method proposed here. With known variance parameters, the estimation performance using the almost-exact distribution always achieves the nominal significance level regardless of the number of studies and heterogeneity. We also propose some methods to construct a conservative interval estimation, even when the variance parameters are unknown, and present their performances via simulation and an application to Alzheimer’s disease meta-analysis.
Similar content being viewed by others
References
Akkouchi, M. (2005). On the convolution of gamma distributions. Soochow Journal of Mathematics, 31, 205–211.
Biggerstaff, B., Jackson, D. (2008). The exact distribution of Cochran’s heterogeneity statistic in one-way random effects meta-analysis. Statistics in Medicine, 27, 6093–6110.
Box, G. (1954). Some theorems on quadratic forms applied in the study of analysis of variance problems, I. Effect of inequality of variance in the one-way classification. The Annals of Mathematical Statistics, 25, 290–302.
Chen, R., Chan, P., Chu, H. (2017). Treatment effects between monotherapy of donepezil versus combination with memantine for alzheimer disease: A meta-analysis. PLOS ONE, 12, 1–14.
Cornell, J., Mulrow, C., Localio, R. (2014). Random-effects meta-analysis of inconsistent effects: A time for change. Annals of Internal Medicine, 160, 267–270.
DerSimonian, R., Kacker, R. (2007). Random-effects model for meta-analysis of clinical trials: An update. Contemporary Clinical Trials, 28, 105–114.
DerSimonian, R., Laird, N. (1986). Meta-analysis in clinical trials. Controlled Clinical Trials, 7, 177–188.
Follmann, D., Proschan, M. (1999). Valid inference in random effects meta-analysis. Biometrics, 55, 732–737.
Hardy, R., Thompson, S. (1996). A likelihood approach to meta-analysis with random effects. Statistics in Medicine, 15, 619–629.
Hartung, J. (1999). An alternative method for meta-analysis. Biometrical Journal, 41, 901–916.
Higgins, J., Thompson, S. (2002). Quantifying heterogeneity in a meta-analysis. Statistics in Medicine, 21, 1539–1558.
Imhof, J. (1961). Computing the distribution of quadratic forms in normal variables. Biometrika, 48, 419–426.
Jackson, D., Bowden, J., Baker, R. (2010). How does the dersimonian and laird procedure for random effects meta-analysis compare with its more efficient but harder to compute counterparts? Journal of Statistical Planning and Inference, 140, 961–970.
Knapp, G., Hartung, J. (2003). Improved tests for a random effects meta-regression with a single covariate. Statistics in Medicine, 22, 2693–2710.
Li, M., Liu, Y., Li, P. (2022). Empirical likelihood meta-analysis with publication bias correction under Copas-like selection model. Annals of the Institute of Statistical Mathematics, 74(1), 93–112.
Michael, H., Thornton, S., Xie, M. (2019). Exact inference on the random-effects model for meta-analyses with few studies. Biometrics, 75, 485–493.
Noma, H., Nagashima, K., Furukawa, T. (2020). Permutation inference methods for multivariate meta-analysis. Biometrics, 76, 337–347.
Paule, R., Mandel, J. (1982). Consensus values and weighting factors. Journal of Research of the National Bureau of Standards, 87, 377–385.
Rover, C., Knapp, G., Friede, T. (2015). Hartung-knapp-sidik-jonkman approach and its modification for random-effects meta-analysis with few studies. BMC medical research methodology, 15, 99.
Sanchez-Meca, J., Marin-Martinez, F. (2008). Confidence intervals for the overall effect size in random-effects meta-analysis. Psychological Methods, 13, 31–48.
Stijnen, T., Hamza, T., Ozdemir, P. (2010). Random effects meta-analysis of event outcome in the framework of the generalized linear mixed model with applications in sparse data. Statistics in Medicine, 29, 3046–3067.
Thompson, S., Sharp, S. (1999). Explaining heterogeneity in meta-analysis: A comparison of methods. Statistics in Medicine, 18, 2693–2708.
Whitehead, A., Whitehead, J. (1991). A general parametric approach to the meta-analysis of randomized clinical trials. Statistics in Medicine, 10, 1665–1677.
Acknowledgements
We thank the editor, the associate editor and the two anonymous referees for their constructive comments. This work was supported by JSPS KAKENHI (Grant number JP21K11783).
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
A Theoretical details and numerical calculation
A Theoretical details and numerical calculation
1.1 A.1 Proof of Theorem 1
First, we should note that \({\hat{\theta }}_{DL}\) and \({\hat{\tau }}^2_{\mathrm{u}}\) are not mutually independent. This situation is quite complicated, unlike the case of sample mean and unbiased variance from normal data with equal error-variances, such as Student’s t test. For a such reason, in order to derive the distribution of \(T_{DL}\), we consider the marginalization
by decomposing the joint PDF of \((T_{DL},{\hat{\tau }}^2_{\mathrm{u}})\). Next, we discuss the mean and variance of \(T_{DL}|{\hat{\tau }}^2_{\mathrm{u}}\) (i.e., the conditional random variable of \(T_{DL}\) given \({\hat{\tau }}^2_{\mathrm{u}}\)). Given \({\hat{\tau }}^2_{\mathrm{u}} = x\), we have the constraint condition
imposed on \({\hat{\theta }}_k\), \(k=1,\dots ,K\). By solving (7) on \({\hat{\theta }}_{k'}\) (\(k'=1,\dots ,K\)), we obtain two cases
where \({\bar{\theta }}_{y,- k'}\) and \(v^2_{- k'}\) are an extended inverse-variance weighted average and its square average version except the \(k'\)-th element \({\hat{\theta }}_{k'}\), respectively, that is,
\(\xi _{k'}({\bar{\theta }}_{0,- k'},v^2_{- k'};x)\) is a quantity written as
See Supplemental material Sect. 5 for further details of solving (7).
For simplicity, let \(k' = K\). The PDF of \({\hat{\theta }}_{k'}|{\hat{\tau }}_u^2\) can be expressed as
where \(\Omega = \{ ({\hat{\theta }}_1, \dots , {\hat{\theta }}_K) \in {\mathbb {R}}^n| {\hat{\tau }}_u^2=x \}\), \({\bar{z}}_{-(K-1)} = {\bar{\theta }}_{0,-(K-1)}(z_1, \dots , z_{K-2}, z_K)\), \(\xi _{-(K-1)}({\mathbf {z}}) = \xi _{-(K-1)}(z_1, \dots , z_{K-2},x, z_K)\) and \(\phi _k\) is the PDF of \(N(\theta , \sigma _k^2+\tau ^2)\). See Supplemental material Sect. 6 for further derivation details of \(f_{{\hat{\theta }}_{K}|{\hat{\tau }}_u^2}\). We have \(E[{\hat{\theta }}_{K}|{\hat{\tau }}^2_u] = \theta\) because the distribution of \({\hat{\theta }}_{K}|{\hat{\tau }}_u^2\) is symmetry. That is, two cases of \({\hat{\theta }}_{K}\) obtained under the constraint condition (7) must occur fairly in probabilistic events. Hence, we observe that \(T_{DL}|{\hat{\tau }}^2_{\mathrm{u}}\) can be transformed to
using U which takes either 1 or -1 with equal probability, where
The distribution of \(\varepsilon ({\bar{\theta }}_{0,- k'},v^2_{- k'};x)\) is determined by two quantities \({\bar{\theta }}_{0,- k'}\) and \(v^2_{- k'}\) but cannot be expressed by some known distribution. On the other hand, because \(Y({\bar{\theta }}_{x,- k'},{\bar{\theta }}_{0,- k'};x)\) is composed of a linear sum of \({\bar{\theta }}_{x,- k'}\) and \({\bar{\theta }}_{0,- k'}\), \(Y({\bar{\theta }}_{x,- k'},{\bar{\theta }}_{0,- k'};x)\) is normally distributed and we find that it has mean \(m_{- k'}(x)\) and variance \(v^2_{- k'}(x)\). Replacing \({\hat{\theta }}_k\) with \({\hat{\theta }}_k=\theta +\sqrt{\sigma ^2_k+\tau ^2}z_k\) using mutually independent random numbers \(z_1\), \(\dots\), \(z_K\sim N(0,1)\), three quantities \({\bar{\theta }}_{x,- k'}\), \({\bar{\theta }}_{0,- k'}\) and \(v^2_{- k'}\) to determine the distribution of \(T_{DL}|_{{\hat{\tau }}_{\mathrm{u}}^2=x}\) are written as
We obtain the result of this theorem by replacing \({\bar{\theta }}_{0,- k'}\) and \(v^2_{- k'}\) in \(\xi _{k'}({\bar{\theta }}_{0,- k'},v^2_{- k'};x)\) with
for \(k'=K\) and using the relation \(\gamma _0^{(1)}-\gamma ^{(1)}_{0,- K}=\sigma _K^{-2}\). \(\square\)
1.2 A.2 Proof of Theorem 2
An idea of Theorem 2 is to approximate the distribution of \(T_{DL}|_{{\hat{\tau }}^2_\mathrm{u}=x}\) by a normal distribution, because the exact conditional distribution is a complicated but symmetric distribution. Concretely, the mean and variance of \(T_{DL}|_{{\hat{\tau }}^2_\mathrm{u}=x}\) are, respectively, obtained as
and
where the form of \(E[{\hat{\theta }}_i {\hat{\theta }}_j|{\hat{\tau }}_{\mathrm{u}}^2=x]\) is
and the joint PDF of \(({\hat{\theta }}_1, \dots , {\hat{\theta }}_{K-1}, {\hat{\tau }}^2_{\mathrm{u}})\) is written as
where \({\bar{t}}_{- K} = \left( \sum _{k=1}^{K-1} t_k/\sigma _k^2\right) /\gamma _{0,- K}^{(1)}\) and \(\overline{t^2}_{- K} = \left( \sum _{k=1}^{K-1} t^2_k/\sigma _k^2\right) /\gamma _{0,- K}^{(1)}\) Thus, the conditional mean \(\mu (x)\) and variance \(\nu ^2(x)\) in Theorem 2 are obtained in this manner.
Finally, we show that the distribution of \(T_{DL}|{\hat{\tau }}^2_{\mathrm{u}}\) can be approximated by normal distribution. For simplicity, assume \(\sigma _1^2 \le \cdots \le \sigma _K^2\), so that we have \(\sum _k^2 (\sigma _k^2 + x)^{-1} \ge K (\sigma _K^2 + x)^{-1}\). Denote a sum of normally distributed random variables as an approximate value of \(T_{DL}|_{{\hat{\tau }}_{\mathrm{u}}^2=x}\) by
Then, we have
where c is a constant. In Euclidean distance, \(T_{DL}|_{{\hat{\tau }}^2_{\mathrm{u}}}\) converges to \({\tilde{T}}_{DL}|_{{\hat{\tau }}_{\mathrm{u}}^2}\) (as \(K \rightarrow \infty\)). This means that \(\lim _{K \rightarrow \infty }|f_{T_{DL}|_{{\hat{\tau }}_{\mathrm{u}}^2}}(t|x)-f_{{\tilde{T}}_{DL}|_{{\hat{\tau }}_{\mathrm{u}}^2}}(t|x)|=0\) in the PDFs of two random variables by the continuity of the function \(f_{T_{DL}|_{{\hat{\tau }}_{\mathrm{u}}^2}}(t|x)\). Also, it is clear that \(f_{{\tilde{T}}_{DL}|_{{\hat{\tau }}_{\mathrm{u}}^2}}(t|x)\) and \(\phi (t;\mu (x),\nu ^2(x))\) are equivalent for sufficiently large K by showing that they have identical mean and variance. Therefore, the proof of Theorem 2 is complete. \(\square\)
1.3 A.3 Numerical calculation for Theorem 2
We have two problems in numerical calculation for Theorem 2. One is to calculate the PDF of between-study variance \({\hat{\tau }}^2_{\mathrm{u}}\), and another is to calculate the variance of \(T_{DL}|{\hat{\tau }}^2_{\mathrm{u}}\). The estimate of between-study variance, \({\hat{\tau }}^2_{\mathrm{u}}\), is the sum of weighted chi-square distribution, as shown by Lemma 1. The numerical calculation of the PDF of weighted chi-square distribution is proposed by many authors, for example, Imhof (1961), Akkouchi (2005), and so on. However, these methods are not necessarily accurate when the number of studies K is small. Therefore, we calculate the PDF by generating random numbers that follow the chi-square distribution with 1 degree of freedom using simulation. That is, if \(x_k(\ell ), k=1, \dots , K, \ell =1, \dots , L\) are random numbers that independently follow a chi-square distribution with 1 degree of freedom, then the random number \(y_1, \dots , y_L\) of L estimates of the between-study variance represents
Thus, the cumulative distribution function of the between-study variance estimate \({\hat{\tau }}^2_{\mathrm{u}}\) is
The PDF \(f_{{\hat{\tau }}^2_{\mathrm{u}}}\) is also calculated using the numerical differentiation, \(f_{{\hat{\tau }}^2_{\mathrm{u}}}(x) = (F_{{\hat{\tau }}^2_{\mathrm{u}}}(x+h)-F_{{\hat{\tau }}^2_{\mathrm{u}}}(x))/h\), where h is a sufficiently small constant. In our simulation, we set \(L = 100,000\) for calculating \(f_{{\hat{\tau }}^2_{\mathrm{u}}}\).
In the numerical calculation of the variance of \(T_{DL}|{\hat{\tau }}^2_{\mathrm{u}}\), Monte Carlo integration with importance sampling is quite useful and a handy tool in this case. That is, if \(z_{k}^{(\ell )}, \ell =1,\dots , L, k=1, \dots , K-1\) are random numbers sampled from standard normal distribution N(0, 1), the mean of \({\hat{\theta }}_i {\hat{\theta }}_j|{\hat{\tau }}_{\mathrm{u}}^2\) can be approximated as
where
Thus, the conditional variance \(\nu ^2(x)\) is calculated based on (9) given sufficiently large L. In our simulation, we set \(L = 10,000\) for calculating \(E[{\hat{\theta }}_i {\hat{\theta }}_j|{\hat{\tau }}_{\mathrm{u}}^2=x]\). Using the two PDFs \(f_{{\hat{\tau }}^2_{\mathrm{u}}}\) and \(f_{T_{DL}|{\hat{\tau }}^2_{\mathrm{u}}}\), the PDF (5) of test statistic \(T_{DL}\) can be calculated, where the trapezoidal or Simpson’s rules are useful for numerical integration using these PDFs.
1.4 A.4 Simulation of conditional test statistic \(T_{DL}|{\hat{\tau }}_u^2=x\)
We compare the difference between the exact conditional distribution of \(T_{DL}|{\hat{\tau }}_u^2=x\) and its normal approximation under some x. The setting of \(\theta\) and \(\sigma _k^2\) are the same as Sect. 4.2. The number of study is set to small case \(K=5\) and the heterogeneity is set to small and large \(I^2=0.3, 0.9\). x is set to (1, 2) for comparing various situations. We sample 100,000 data of \((z_1, \dots , z_{k-1})\) from standard normal distribution and calculate the exact distribution of \(T_{DL}|{\hat{\tau }}_u^2=x\). The almost-exact distribution of \(T_{DL}|{\hat{\tau }}_u^2=x\) can be calculated by the method in Appendix A.3.
We show the simulation histogram based on the exact conditional distribution and the corresponding almost-exact PDF in Fig. 3. The almost-exact distribution of \(T_{DL}|{\hat{\tau }}_u^2=x\) approximates its exact distribution well in the small number of studies. This provides that it is reasonable to use the almost-exact distribution of \(T_{DL}|{\hat{\tau }}_u^2=x\) for our simulations.
About this article
Cite this article
Hanada, K., Sugimoto, T. Inference using an exact distribution of test statistic for random-effects meta-analysis. Ann Inst Stat Math 75, 281–302 (2023). https://doi.org/10.1007/s10463-022-00844-4
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10463-022-00844-4