Abstract
Future climate is typically projected using multi-model ensembles, but the ensemble mean is unlikely to be optimal if models’ skill at reproducing historical climate is not considered. Moreover, individual climate models are not independent. Here, we examine the interplay between the benefits of optimising an ensemble for the performance of its mean and the the effect this has on ensemble spread as an uncertainty estimate. Using future Australian precipitation change as a case study, we perform optimal subset selection based on present-day precipitation, sea surface temperature and/or 500 hPa eastward wind climatologies. We use either one, two, or all three variables as predictors. Out-of-sample projection skill is assessed using a model-as-truth approach (rather than observations). For multiple variables, multi-objective optimisation is used to obtain Pareto-optimal subsets (an ensemble of model subsets), to gauge the uncertainty in optimisation arising from the multiple constraints. We find that the spread of climate model subset averages typically under-represents the true projection uncertainty (overconfidence), but that the situation can be significantly improved using mixture distributions for uncertainty estimation. The single best predictor, present-day precipitation, gives the most accurate results but is still overconfident—a consequence of calibrating too specifically. It is only when all three constraints are used that projection skill is improved and overconfidence is eliminated, but at the cost of a poorer best estimate relative to one predictor. We thus identify an important trade-off between accuracy and precision, depending on the number of predictors, which is likely relevant for any subset selection or weighting strategy.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Abramowitz G (2010) Model independence in multi-model ensemble prediction. Aust Meteorol Oceanogr J 59:3–6 (Doi: 10.1.1.222.5811).
Abramowitz G, Bishop CH (2015) Climate model dependence and the ensemble dependence transformation of CMIP projections. J Clim 28(6):2332–2348. https://doi.org/10.1175/JCLI-D-14-00364.1
Annan J, Hargreaves J (2011) Understanding the CMIP3 multimodel ensemble. J Clim 24(16):4529–4538. https://doi.org/10.1175/2011JCLI3873.1
Annan JD, Hargreaves JC (2017) On the meaning of independence in climate science. Earth Syst Dyn 8(1):211. https://doi.org/10.5194/esd-8-211-2017
Bishop CH, Abramowitz G (2013) Climate model dependence and the replicate Earth paradigm. Clim Dyn 41(3–4):885–900. https://doi.org/10.1007/s00382-012-1610-y
Boé J, Hall A, Qu X (2009) September sea-ice cover in the Arctic Ocean projected to vanish by 2100. Nat Geosci 2(5):341–343. https://doi.org/10.1038/ngeo467
Boé J (2018) Interdependency in multimodel climate projections: component replication and result similarity. Geophys Res Lett 45(6):2771–2779. https://doi.org/10.1002/2017GL076829
Borodina A, Fischer EM, Knutti R (2017) Emergent constraints in climate projections: a case study of changes in high-latitude temperature variability. J Clim 30(10):3655–3670. https://doi.org/10.1175/JCLI-D-16-0662.1
Box GE, Draper NR (1987) Empirical model-building and response surfaces. Wiley, New York (ISBN 0-471-81033-9)
Caldwell PM, Bretherton CS, Zelinka MD, Klein SA, Santer BD, Sanderson BM (2014) Statistical significance of climate sensitivity predictors obtained by data mining. Geophys Res Lett 41(5):1803–1808. https://doi.org/10.1002/2014GL059205
Deb K (2014) Multi-objective optimization. Search methodologies. Springer, Boston, pp 403–449. https://doi.org/10.1007/978-1-4614-6940-7_15
Gurobi Optimization, Inc. (2015) Gurobi optimizer reference manual. http://www.gurobi.com. Accessed 04 Apr 2018
Herger N, Abramowitz G, Knutti R, Angélil O, Lehmann K, Sanderson BM (2018) Selecting a climate model subset to optimise key ensemble properties. Earth Syst Dyn 9:135–151. https://doi.org/10.5194/esd-9-135-2018
IPCC (2013) Summary for policymakers. In: Climate change 2013: the physical science basis. In: Stocker TF, Qin D, Plattner GK, Tignor M, Allen SK, Boschung J, Nauels A, Xia Y, Bex V, Midgley PM (eds) Contribution of Working Group I to the Fifth Assessment Report of the Intergovernmental Panel on Climate Change. Cambridge University Press, Cambridge
King AD, Donat MG, Alexander LV, Karoly DJ (2015) The ENSO-Australian rainfall teleconnection in reanalysis and CMIP5. Clim Dyn 44(9–10):2623–2635. https://doi.org/10.1007/s00382-014-2159-8
Knutti R, Abramowitz G, Collins M, Eyring V, Gleckler P, Hewitson B, Mearns L (2010a) Good practice guidance paper on assessing and combining multi model climate projections. IPCC Expert Meeting on Assessing and Combining Multi Model Climate Projections, p 15
Knutti R, Furrer R, Tebaldi C, Cermak J, Meehl GA (2010b) Challenges in combining projections from multiple climate models. J Clim 23(10):2739–2758. https://doi.org/10.1175/2009JCLI3361.1
Knutti R, Masson D, Gettelman A (2013) Climate model genealogy: generation CMIP5 and how we got there. Geophys Res Lett 40(6):1194–1199. https://doi.org/10.1002/grl.50256
Knutti R, Sedláček J, Sanderson BM, Lorenz R, Fischer EM, Eyring V (2017) A climate model projection weighting scheme accounting for performance and interdependence. Geophys Res Lett 44(4):1909–1918. https://doi.org/10.1002/2016GL072012
Langenbrunner B, Neelin JD (2017) Pareto-optimal estimates of california precipitation change. Geophys Res Lett 44(24):12436–12446. https://doi.org/10.1002/2017GL075226
Leduc M, Laprise R, de Elìa R, Šeparović L (2016) Is institutional democracy a good proxy for model independence? J Clim 29(23):8301–8316. https://doi.org/10.1175/JCLI-D-15-0761.1
Lorenz R, Herger N, Sedláček J, Eyring V, Fischer E M, Knutti R (2018) Prospects and caveats of weighting climate models for summer maximum temperature projections over North America. J Geophys Res Atmos. https://doi.org/10.1029/2017JD027992.
Mitchell JE (2002) Branch-and-cut algorithms for combinatorial optimization problems. Handbook of applied optimization, pp 65–77
Pareto V (1906) Manuale di economia politica, vol 13. Societa Editrice, Milano
Perkins SE, Pitman AJ (2009) Do weak AR4 models bias projections of future climate changes over Australia? Clim Change 93(3–4):527–558. https://doi.org/10.1007/s10584-008-9502-1
Pincus R, Batstone CP, Hofmann RJP, Taylor KE, Glecker PJ (2008) Evaluating the present-day simulation of clouds, precipitation, and radiation in climatemodels. J Geophys Res 113:D14209. https://doi.org/10.1029/2007JD009334
Pitman AJ, Perkins SE (2008) Regional projections of future seasonal and annual changes in rainfall and temperature over Australia based on skill-selected AR4 models. Earth Interact 12(12):1–50. https://doi.org/10.1175/2008EI260.1
Power S, Haylock M, Colman R, Wang X (2006) The predictability of interdecadal changes in ENSO activity and ENSO teleconnections. J Clim 19(19):4755–4771. https://doi.org/10.1175/JCLI3868.1
Risbey JS, Pook MJ, McIntosh PC, Wheeler MC, Hendon HH (2009) On the remote drivers of rainfall variability in Australia. Mon Weather Rev 137(10):3233–3253. https://doi.org/10.1175/2009MWR2861.1
Sanderson BM, Knutti R, Caldwell P (2015a) Addressing interdependency in a multimodel ensemble by interpolation of model properties. J Clim 28:5150–5170. https://doi.org/10.1175/JCLI-D-14-00361.1
Sanderson BM, Knutti R, Caldwell P (2015b) A representative democracy to reduce interdependency in a multimodel ensemble. J Clim 28(13):5171–5194. https://doi.org/10.1175/JCLI-D-14-00362.1
Sanderson BM, Wehner M, Knutti R (2017) Skill and independence weighting for multi-model assessments. Geosci Model Dev 10(6):2379–2395. https://doi.org/10.5194/gmd-10-2379-2017
Taylor KE, Stouffer RJ, Meehl GA (2012) An overview of CMIP5 and the experiment design. Bull Am Meteorol Soc 93(4):485–498. https://doi.org/10.1175/BAMS-D-11-00094.1
Tebaldi C, Knutti R (2007) The use of the multi-model ensemble in probabilistic climate projections. Philos Trans R Soc Lond A Math Phys Eng Sci 365(1857):2053–2075. https://doi.org/10.1098/rsta.2007.2076
Weigel AP, Liniger MA, Appenzeller C (2007) The discrete Brier and ranked probability skill scores. Mon Weather Rev 135(1):118–124. https://doi.org/10.1175/MWR3280.1
Weigel AP, Knutti R, Liniger MA, Appenzeller C (2010) Risks of model weighting in multimodel climate projections. J Clim 23(15):4175–4191. https://doi.org/10.1175/2010JCLI3594.1
Acknowledgements
We would like to thank Karsten Lehmann for help with the mathematical programming solver Gurobi. Thanks to Jan Sedláček for providing access to the CMIP5 archive based at ETHZ. Nadja Herger, Steven Sherwood, and Oliver Angélil acknowledge the support of the Australian Research Council Centre of Excellence for Climate System Science (CE110001028). Gab Abramowitz and Scott A. Sisson acknowledge the Australian Research Council Centre of Excellence for Climate Extremes (CE170100023). Scott A. Sisson acknowledges the support of the Australian Research Council Centre of Excellence for Mathematical and Statistical Frontiers (CE140100049). ETH Zurich acknowledges support by the European Union’s Horizon 2020 research and innovation programme under Grant agreement no. 641816 (CRESCENDO) We acknowledge the World Climate Research Programme’s Working Group on Coupled Modelling, which is responsible for CMIP, and we thank the climate modeling groups (listed in Table S1 in the SI) for producing and making available their model output. For CMIP the US Department of Energy’s Program for Climate Model Diagnosis and Intercomparison provides coordinating support and led development of software infrastructure in partnership with the Global Organization for Earth System Science Portals. CMIP data can be obtained from http://cmip-pcmdi.llnl.gov/cmip5/. We use Python for data analysis and visualisation.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
About this article
Cite this article
Herger, N., Abramowitz, G., Sherwood, S. et al. Ensemble optimisation, multiple constraints and overconfidence: a case study with future Australian precipitation change. Clim Dyn 53, 1581–1596 (2019). https://doi.org/10.1007/s00382-019-04690-8
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00382-019-04690-8