Abstract
There is now a wide range of forecasts and observations of seasonal climatic conditions that can be used across a range of application sectors, including hydrological risk forecasting, planning and management. As we rely more on seasonal climate forecasts, it becomes essential to also assess its quality to ensure its intended use. In this study, we provide the most comprehensive assessment of seasonal temperature and precipitation ensemble forecasts of the EUROSIP multi-model forecasting system over Europe. The forecasts from the four individual climate models within the EUROSIP are assessed using both deterministic and probabilistic approaches. One equally and two unequally Weighted Multi-Models (WMMs) are also constructed from the individual models, for both climate variables, and their respective forecasts are also assessed. Consistent with existing literature, we find limited seasonal climate prediction skill over Europe. A simple equally WMM system performs better than both unequally WMM combination systems. However, the equally WMM system does not always outperform the single best model within the EUROSIP multi-model. Based on the results, it is recommended to assess seasonal temperature and precipitation forecast of individual climate models as well as their multi-model mean for a comprehensive overview of the forecast skill.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Adler RF, Huffman GJ, Chang A, Ferraro R, Xie P-P, Janowiak J, Rudolf B, Schneider U, Curtis S, Bolvin D, Gruber A, Susskind J, Arkin P, Nelkin E (2003) The version-2 global precipitation climatology project (GPCP) monthly precipitation analysis (1979-present). J Hydrometeorol 4(6):1147–1167. https://doi.org/10.1175/1525-7541(2003)004%3c1147:TVGPCP%3e2.0.CO;2
Alexander MA, Bladé I, Newman M, Lanzante JR, Lau N-C, Scott JD (2002) The atmospheric bridge: the influence of ENSO teleconnections on air–sea interaction over the global oceans. J Clim 15(16):2205–2231. https://doi.org/10.1175/1520-0442(2002)015%3c2205:TABTIO%3e2.0.CO;2
Arribas A, Glover M, Maidens A, Peterson K, Gordon M, MacLachlan C, Graham R, Fereday D, Camp J, Scaife AA, Xavier P, McLean P, Colman A, Cusack S (2011) The GloSea4 ensemble prediction system for seasonal forecasting. Mon Weather Rev 139(6):1891–1910. https://doi.org/10.1175/2010MWR3615.1
Athanasiadis PJ, Bellucci A, Scaife AA, Hermanson L, Materia S, Sanna A, Borrelli A, MacLachlan C, Gualdi S (2017) A multisystem view of wintertime NAO seasonal predictions. J Clim 30(4):1461–1475
Balmaseda M, Anderson D (2009) Impact of initialization strategies and observations on seasonal forecast skill. Geophys Res Lett 36(1). https://doi.org/10.1029/2008GL035561
Bazile R, Boucher M-A, Perreault L, Leconte R (2017) Verification of ECMWF system4 for seasonal hydrological forecasting in a northern climate. Hydrol Earth Syst Sci Discuss 2017:1–22. https://doi.org/10.5194/hess-2017-387
Bell CJ, Gray LJ, Charlton-Perez AJ, Joshi MM, Scaife AA (2009) Stratospheric communication of El Niño teleconnections to European winter. J Clim 22(15):4083–4096. https://doi.org/10.1175/2009JCLI2717.1
Bellprat O, Massonnet F, Siegert S, Prodhomme C, Macias-Gómez D, Guemas V, Doblas-Reyes F (2017) Uncertainty propagation in observational references to climate model scales. Rem Sens Environ 203:101–108. https://doi.org/10.1016/j.rse.2017.06.034
Benjamini Y, Hochberg Y (1995) Controlling the false discovery rate: a practical and powerful approach to multiple testing. J R Stat Soc Ser B (Methodological) 57(1):289–300
Boucher M-A, Perreault L, Anctil F (2009) Tools for the assessment of hydrological ensemble forecasts obtained by neural networks. J Hydroinf 11(3–4):297–307. https://doi.org/10.2166/hydro.2009.037
Brankovič C, Palmer TN (2000) Seasonal skill and predictability of ECMWF PROVOST ensembles. Quart J R Meteorol Soc 126(567):2035–2067. https://doi.org/10.1002/qj.49712656704
Bröcker J, Smith LA (2008) From ensemble forecasts to predictive distribution functions. Tellus A 60(4):663–678. https://doi.org/10.1111/j.1600-0870.2008.00333.x
Candille G, Talagrand O (2005) Evaluation of probabilistic prediction systems for a scalar variable. Quart J R Meteorol Soc 131(609):2131–2150. https://doi.org/10.1256/qj.04.71
Chapelle O, Vapnik V, Bengio Y (2002) Model selection for small sample regression. Mach Learn 48(1):9–23
Colman RA (2005) BMRC atmospheric model (BAM) version 3.0: comparison with mean climatology. Research Report No. 108, Bur. Met. Australia
Crochemore L, Ramos M-H, Pappenberger F (2016) Bias correcting precipitation forecasts to improve the skill of seasonal streamflow forecasts. Hydrol Earth Syst Sci 20(9):3601–3618. https://doi.org/10.5194/hess-20-3601-2016
Dee DP, Uppala SM, Simmons AJ, Berrisford P, Poli P, Kobayashi S, Andrae U, Balmaseda MA, Balsamo G, Bauer P, Bechtold P, Beljaars ACM, van de Berg L, Bidlot J, Bormann N, Delsol C, Dragani R, Fuentes M, Geer AJ, Haimberger L, Healy SB, Hersbach H, Hólm EV, Isaksen L, Kallberg P, Köhler M, Matricardi M, McNally AP, Monge-Sanz BM, Morcrette J-J, Park B-K, Peubey C, de Rosnay P, Tavolato C, Thépaut J-N, Vitart F (2011) The ERA-Interim reanalysis: configuration and performance of the data assimilation system. Quart J R Meteorol Soc 137(656):553–597. https://doi.org/10.1002/qj.828
DelSole T (2007) A bayesian framework for multimodel regression. J Clim 20(12):2810–2826. https://doi.org/10.1175/JCLI4179.1
DelSole T, Yang X, Tippett MK (2013) Is unequal weighting significantly better than equal weighting for multi-model forecasting? Quart J R Meteorol Soc 139(670):176–183. https://doi.org/10.1002/qj.1961
Doblas-Reyes FJ, Déqué M, Piedelievre J-P (2000) Multi-model spread and probabilistic seasonal forecasts in PROVOST. Quart J R Meteorol Soc 126(567):2069–2087. https://doi.org/10.1002/qj.49712656705
Doblas-Reyes FJ, García-Serrano J, Lienert F, Biescas AP, Rodrigues LRL (2013) Seasonal climate predictability and forecasting: status and prospects. Wiley Interdiscip Rev: Clim Change 4(4):245–268. https://doi.org/10.1002/wcc.217
Doblas-Reyes FJ, Pavan V, Stephenson DB (2003) The skill of multi-model seasonal forecasts of the wintertime North Atlantic Oscillation. Clim Dyn 21(5):501–514. https://doi.org/10.1007/s00382-003-0350-4
Efron B (1983) Estimating the error rate of a prediction rule: improvement on cross-validation. J Am Stat Assoc 78(382):316–331
EUROSIP (2016) EUROSIP operational history. Accessed 4 Aug 2017
Ferro CAT (2014) Fair scores for ensemble forecasts. Quart J R Meteorol Soc 140(683):1917–1923. https://doi.org/10.1002/qj.2270
Ferro CAT, Richardson DS, Weigel AP (2008) On the effect of ensemble size on the discrete and continuous ranked probability scores. Meteorol Appl 15(1):19–24. https://doi.org/10.1002/met.45
Fortin V, Favre A-C, Saïd M (2006) Probabilistic forecasting from ensemble prediction systems: improving upon the best-member method by using a different weight and dressing kernel for each member. Quart J R Meteorol Soc 132(617):1349–1369. https://doi.org/10.1256/qj.05.167
Frías MD, Herrera S, Cofiño AS, Gutiérrez JM (2010) Assessing the skill of precipitation and temperature seasonal forecasts in Spain: windows of opportunity related to ENSO events. J Clim 23(2):209–220. https://doi.org/10.1175/2009JCLI2824.1
Fricker TE, Ferro CAT, Stephenson DB (2013) Three recommendations for evaluating climate predictions. Meteorol Appl 20(2):246–255. https://doi.org/10.1002/met.1409
Gneiting T (2011) Quantiles as optimal point forecasts. Int J Forecast 27(2):197–207. https://doi.org/10.1016/j.ijforecast.2009.12.015
Gneiting T, Raftery AE, Westveld III, Goldman T (2005) Calibrated probabilistic forecasting using ensemble model output statistics and minimum CRPS estimation. Mon Weather Rev 133(5):1098–1118. https://doi.org/10.1175/MWR2904.1
Graham RJ, Evans ADL, Mylne KR, Harrison MSJ, Robertson KB (2000) An assessment of seasonal predictability using atmospheric general circulation models. Quarterly Journal of the Royal Meteorological Society 126(567):2211–2240. https://doi.org/10.1002/qj.49712656712
Guemas V, Blanchard-Wrigglesworth E, Chevallier M, Day JJ, Déqué M, Doblas-Reyes FJ, Fućkar NS, Germe A, Hawkins E, Keeley S, Koenigk T, Mélia D S y, Tietsche S (2016) A review on Arctic sea–ice predictability and prediction on seasonal to decadal time-scales. Quart J R Meteorol Soc 142(695):546–561. https://doi.org/10.1002/qj.2401
Hagedorn R, Doblas-Reyes FJ, Palmer TN (2005) The rationale behind the success of multi-model ensembles in seasonal forecasting—I. Basic concept. Tellus A 57(3):219–233. https://doi.org/10.1111/j.1600-0870.2005.00103.x
Hawkins E, Sutton R (2009) The potential to narrow uncertainty in regional climate predictions. Bull Am Meteorol Soc 90(8):1095–1107. https://doi.org/10.1175/2009BAMS2607.1
Hawkins E, Sutton R (2011) The potential to narrow uncertainty in projections of regional precipitation change. Clim Dyn 37(1):407–418. https://doi.org/10.1007/s00382-010-0810-6
Hersbach H (2000) Decomposition of the continuous ranked probability score for ensemble prediction systems. Weather Forecast 15(5):559–570. https://doi.org/10.1175/1520-0434(2000)015%3c0559:DOTCRP%3e2.0.CO;2
Kharin VV, Zwiers FW (2002) Climate predictions with multimodel ensembles. J Clim 15(7):793–799
Kim H-M, Webster PJ, Curry JA (2012) Seasonal prediction skill of ECMWF System 4 and NCEP CFSv2 retrospective forecast for the Northern Hemisphere Winter. Clim Dyn 39(12):2957–2973. https://doi.org/10.1007/s00382-012-1364-6
Krikunov AV, Kovalchuk SV (2015) Dynamic selection of ensemble members in multi-model hydrometeorological ensemble forecasting. Procedia Computer Science 66 (Supplement C), 220–227. 4th International Young Scientist Conference on Computational Science
Krishnamurti TN, Kishtawal CM, Zhang Z, LaRow T, Bachiochi D, Williford E, Gadgil S, Surendran S (2000) Multimodel ensemble forecasts for weather and seasonal climate. J Clim 13(23):4196–4216. https://doi.org/10.1175/1520-0442(2000)013%3c4196:MEFFWA%3e2.0.CO;2
Kumar A (2009) Finite samples and uncertainty estimates for skill measures for seasonal prediction. Mon Weather Rev 137(8):2622–2631
Kumar A, Chen M, Wang W (2013) Understanding prediction skill of seasonal mean precipitation over the tropics. J Clim 26(15):5674–5681. https://doi.org/10.1175/JCLI-D-12-00731.1
Lavell A, Oppenheimer M, Diop C, Hess J, Lempert R, Li J, Muir-Wood R, Myeong S (2012) Climate change: new dimensions in disaster risk, exposure, vulnerability, and resilience. In Field C, Barros V, Stocker T, Qin D, Dokken D, Ebi K, Mastrandrea M, Mach K, Plattner G-K, Allen S, Tignor M, Midgley P (eds) Managing the risks of extreme events and disasters to advance climate change adaptation, Cambridge University Press, Cambridge, pp 25–64. https://doi.org/10.1596/978-0-8213-8845-7
Lavers D, Luo L, Wood EF (2009) A multiple model assessment of seasonal climate forecast skill for applications. Geophys Res Lett 36(23). https://doi.org/10.1029/2009GL041365
Lloyd-Hughes B, Saunders MA (2002) Seasonal prediction of European spring precipitation from El Niño-southern oscillation and local sea-surface temperatures. Int J Climatol 22(1):1–14. https://doi.org/10.1002/joc.723
Lorenz EN (1963) Deterministic nonperiodic flow. J Atmos Sci 20(2):130–141. https://doi.org/10.1175/1520-0469(1963)020%3c0130:DNF%3e2.0.CO;2
MacLachlan C, Arribas A, Peterson KA, Maidens A, Fereday D, Scaife AA, Gordon M, Vellinga M, Williams A, Comer RE, Camp J, Xavier P, Madec G (2015) Global Seasonal forecast system version 5 (GloSea5): a high-resolution seasonal forecast system. Quart J R Meteorol Soc 141(689):1072–1084. https://doi.org/10.1002/qj.2396
Manzanas R, Lucero A, Weisheimer A, Gutiérrez A (2017) Can bias correction and statistical downscaling methods improve the skill of seasonal precipitation forecasts? Clim Dyn. https://doi.org/10.1007/s00382-017-3668-z
Matheson JE, Winkler RL (1976) Scoring rules for continuous probability distributions. Manag Sci 22(10):1087–1096. https://doi.org/10.1287/mnsc.22.10.1087
Molinaro AM, Simon R, Pfeiffer RM (2005) Prediction error estimation: a comparison of resampling methods. Bioinformatics 21(15):3301–3307
Molteni F, Stockdale T, Balmaseda M, Balsamo G, Buizza R, Ferranti L, Magnusson L, Mogensen K, Palmer T, Vitart F (2011) The new ECMWF seasonal forecast system (System 4). ECMWF Tech Memorandum 656:49
Murphy AH (1969) On the “ranked probability score”. J Appl Meteorol 8(6):988–989. https://doi.org/10.1175/1520-0450(1969)008%3c0988:OTPS%3e2.0.CO;2
National Academies of Sciences, E. and Medicine (2016) Attribution of extreme weather events in the context of climate change. Washington, DC: The National Academies Press. https://doi.org/10.17226/21852
Palmer T, Doblas-Reyes F, Hagedorn R, Weisheimer A (2005) Probabilistic prediction of climate using multi-model ensembles: from basics to applications. Trans R Soc B, Philos. https://doi.org/10.1098/rstb.2005.1750
Palmer TN, Doblas-Reyes FJ, Hagedorn R, Alessandri A, Gualdi S, Andersen U, Feddersen H, Cantelaube P, Terres J-M, Davey M, Graham R, Délécluse P, Lazar A, Déqué M, Guérémy J-F, Díez E, Orfila B, Hoshen M, Morse AP, Keenlyside N, Latif M, Maisonnave E, Rogel P, Marletto V, Thomson MC (2004) Development of a European multimodel ensemble system for seasonal-to-interannual prediction (DEMETER). Bull Am Meteorol Soc 85(6):853–872
Peng P, Kumar A, van den Dool H, Barnston AG (2002) An analysis of multimodel ensemble predictions for seasonal climate anomalies. J Geophys Res: Atmosp 107(D23):ACL-18. https://doi.org/10.1029/2002JD002712
Prodhomme C, Doblas-Reyes F, Bellprat O, Dutra E (2016) Impact of land-surface initialization on sub-seasonal to seasonal forecasts over Europe. Clim Dyn 47(3):919–935. https://doi.org/10.1007/s00382-015-2879-4
Robertson AW, Lall U, Zebiak SE, Goddard L (2004) Improved combination of multiple atmospheric GCM ensembles for seasonal prediction. Mon Weather Rev 132(12):2732–2744. https://doi.org/10.1175/MWR2818.1
Rodrigues LRL, Doblas-Reyes FJ, dos Santos Coelho CA (2014) Multi-model calibration and combination of tropical seasonal sea surface temperature forecasts. Clim Dyn 42(3):597–616. https://doi.org/10.1007/s00382-013-1779-8
Rodrigues LRL, García-Serrano J, Doblas-Reyes F (2014) Seasonal forecast quality of the West African monsoon rainfall regimes by multiple forecast systems. J Geophys Res: Atmos 119(13):7908–7930. https://doi.org/10.1002/2013JD021316
Saha S, Moorthi S, Wu X, Wang J, Nadiga S, Tripp P, Behringer D, Hou Y-T, Chuang H, Iredell M, Ek M, Meng J, Yang R, Mendez MP, van den Dool H, Zhang Q, Wang W, Chen M, Becker E (2014) The NCEP climate forecast system version 2. J Clim 27(6):2185–2208. https://doi.org/10.1175/JCLI-D-12-00823.1
Scaife AA, Arribas A, Blockley E, Brookshaw A, Clark RT, Dunstone N, Eade R, Fereday D, Folland CK, Gordon M, Hermanson L, Knight JR, Lea DJ, MacLachlan C, Maidens A, Martin M, Peterson AK, Smith D, Vellinga M, Wallace E, Waters J, Williams A (2014) Skillful long-range prediction of European and North American winters. Geophys Res Lett 41(7):2514–2519. https://doi.org/10.1002/2014GL059637
Senan R, Orsolini YJ, Weisheimer A, Vitart F, Balsamo G, Stockdale TN, Dutra E, Doblas-Reyes FJ, Basang D (2016) Impact of springtime Himalayan-Tibetan Plateau snowpack on the onset of the Indian summer monsoon in coupled seasonal forecasts. Clim Dyn 47(9):2709–2725. https://doi.org/10.1007/s00382-016-2993-y
Shi W, Schaller N, MacLeod D, Palmer TN, Weisheimer A (2015) Impact of hindcast length on estimates of seasonal climate predictability. Geophys Res Lett 42(5):1554–1559
Siegert S, Bellprat O, Ménégoz M, Stephenson DB, Doblas-Reyes FJ (2017) Detecting improvements in forecast correlation skill: statistical testing and power analysis. Mon Weather Rev 145(2):437–450
Soares MB, Dessai S (2015) Exploring the use of seasonal climate forecasts in Europe through expert elicitation. Clim Risk Manag 10:8–16
Sordo C, Frías MD, Herrera S, Gutiérrez JM (2008) Interval-based statistical validation of operational seasonal forecasts in Spain conditioned to El Niño-Southern Oscillation events. J Geophys Res: Atmos 113(D17). https://doi.org/10.1029/2007JD009536
Starks PJ, Ross JD, Heathman GC (2003) Modelling the spatial and temporal distribution of soil moisture at watershed scales using remote sensing and GIS. In: Singhroy VH, Hansen DT, Pierce RR, Johnson AI (eds) Spatial methods for solution of environmental and hydrologic problems—science, policy and standardization. ASTM International, West Conshohocken, pp 58–75
Stockdale T (2013) The EUROSIP system - a multi-model approach. In Seminar on Seasonal prediction: science and applications, 3–7 September 2012, Shinfield Park, Reading, pp 257–268. ECMWF: ECMWF
Tebaldi C, Knutti R (2007) The use of the multi-model ensemble in probabilistic climate projections. Philos Trans R Soc Lond A: Math Phys Eng Sci 365(1857):2053–2075. https://doi.org/10.1098/rsta.2007.2076
Torralba V, Doblas-Reyes FJ, MacLeod D, Christel I, Davis M (2017) Seasonal climate prediction: a new source of information for the management of wind energy resources. J Appl Meteorol Climatol 56(5):1231–1247. https://doi.org/10.1175/JAMC-D-16-0204.1
Trigo RM, Osborn TJ, Corte-Real JM (2002) The North Atlantic Oscillation influence on Europe: climate impacts and associated physical mechanisms. Clim Res 20:9–17. https://doi.org/10.3354/cr020009
van den Hurk B, Doblas-Reyes F, Balsamo G, Koster RD, Seneviratne SI, Jr HC (2012) Soil moisture effects on seasonal temperature and precipitation forecast scores in europe. Clim Dyn 38(1):349–362
Velázquez JA, Petit T, Lavoie A, Boucher M-A, Turcotte R, Fortin V, Anctil F (2009) An evaluation of the Canadian global meteorological ensemble prediction system for short-term hydrological forecasting. Hydrol Earth Syst Sci 13(11):2221–2231. https://doi.org/10.5194/hess-13-2221-2009
Vitart F, Huddleston MR, Déqué M, Peake D, Palmer TN, Stockdale TN, Davey MK, Ineson S, Weisheimer A (2007) Dynamically-based seasonal forecasts of Atlantic tropical storm activity issued in June by EUROSIP. Geophys Res Lett 34(16). https://doi.org/10.1029/2007GL030740
Voldoire A, Sachez-Gomez E, y Melia DS, Decharme B, Cassou C, Sénési S, Valcke S, Beau I, Alias A, Chevallier M, Déqué M, Deshayes J, Douville H, Fernandez E, Madec G, Maisonnave E, Moine M-P, Planton S, Saint-Martin D, Szopa S, Tyteca S, Alkama R, Belamari S, Braun A, Coquart L, Chauvin F (2013) The CNRM-CM5.1 global climate model: description and basic evaluation. Clim Dyn 40(9):2091–2121. https://doi.org/10.1007/s00382-011-1259-y
Wanders N, Wada Y (2015) Decadal predictability of river discharge with climate oscillations over the 20th and early 21st century. Geophys Res Lett 42(24):10689–10695. https://doi.org/10.1002/2015GL066929
Wanders N, Wood EF (2016) Improved sub-seasonal meteorological forecast skill using weighted multi-model ensemble simulations. Environ Res Lett 11(9):094007. https://doi.org/10.1088/1748-9326/11/9/094007
Wehrli K, Bhend J, Liniger MA (2017) Systematic quality assessment of an operational seasonal forecasting system. Tech Rep MeteoSwiss 263:52
Weigel AP, Knutti R, Liniger MA, Appenzeller C (2010) Risks of model weighting in multimodel climate projections. Journal of Climate 23(15):4175–4191
Weigel AP, Liniger MA, Appenzeller C (2008) Can multi-model combination really enhance the prediction skill of probabilistic ensemble forecasts? Quart J R Meteorol Soc 134(630):241–260. https://doi.org/10.1002/qj.210
Wilks DS (2006) Comparison of ensemble-MOS methods in the Lorenz ’96 setting. Meteorol Appl 13(3):243–256. https://doi.org/10.1017/S1350482706002192
Yuan X, Ma F, Wang L, Zheng Z, Ma Z, Ye A (2016) An experimental seasonal hydrological forecasting system over the yellow river basin—part 1: Understanding the role of initial hydrological conditions. Hydrol Earth Syst Sci 20(6):2437–2451. https://doi.org/10.5194/hess-20-2437-2016
Yun WT, Stefanova L, Krishnamurti TN (2003) Improvement of the multimodel superensemble technique for seasonal forecasts. J Clim 16(22):3834–3840. https://doi.org/10.1175/1520-0442(2003)016%3c3834:IOTMST%3e2.0.CO;2
Acknowledgements
The authors thank Prof. Francisco J. Doblas Reyes, Alicia Sanchez Lorente, Veronica Torralba and Louis-Philippe Caron for discussions and suggestions during the forming of this paper. Thanks to Nicolau Manubens, whose incredible technical support allowed steady implementation of the experiments in this study. Thanks are also due to the anonymous reviewers and their critical reviews. Their comments and suggestions improved the content of this paper. The research leading to these results has received funding from the EU H2020 Framework Programme under grant agreement #641811 (IMPREX).
Author information
Authors and Affiliations
Corresponding author
Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
About this article
Cite this article
Mishra, N., Prodhomme, C. & Guemas, V. Multi-model skill assessment of seasonal temperature and precipitation forecasts over Europe. Clim Dyn 52, 4207–4225 (2019). https://doi.org/10.1007/s00382-018-4404-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00382-018-4404-z