Abstract
In this paper we propose a new methodology for evaluating prediction intervals (PIs). Typically , PIs are evaluated with reference to confidence values. However, other metrics should be considered, since high values are associated to too wide intervals that convey little information and are of no use for decision-making. We propose to compare the error distribution (predictions out of the interval) and the maximum mean absolute error (MAE) allowed by the confidence limits. Along this paper PIs based on neural networks for short-term load forecast are compared using two different strategies: (1) dual perturb and combine (DPC) algorithm and (2) conformal prediction. We demonstrated that depending on the real scenario (e.g., time of day) different algorithms perform better. The main contribution is the identification of high uncertainty levels in forecast that can guide the decision-makers to avoid the selection of risky actions under uncertain conditions. Small errors mean that decisions can be made more confidently with less chance of confronting a future unexpected condition.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Armstrong, J., Collopy, F.: Error measures for generalizing about forecasting methods: empirical comparisons. Int. J. Forecast. 8(1), 69–80. http://www.sciencedirect.com/science/article/pii/016920709290008W (1992)
Christoffersen, P.F.: Evaluating interval forecasts. Int. Econ. Rev. 39(4), 841–862 (1998)
Gama, J., Rodrigues, P.: Stream-based electricity load forecast. In: Knowledge Discovery in Databases: PKDD 2007. Lecture Notes in Computer Science, vol. 4702, pp. 446–453. Springer, Berlin. http://dx.doi.org/10.1007/978-3-540-74976-9_45 (2007)
Geurts, P., Wehenkel, L.: Closed-form dual perturb and combine for tree-based models. In: Raedt, L.D., Wrobel, S. (eds.) Proceedings of the Twenty-Second International Conference on Machine Learning (ICML 2005), Bonn, Germany, August 7–11, 2005. ACM International Conference Proceeding Series, vol. 119, pp. 233–240. ACM. http://doi.acm.org/10.1145/1102351.1102381 (2005)
Hernández, L., Baladrón, C., Aguiar, J.M., Carro, B., Sánchez-Esguevillas, A., Lloret, J.: Artificial neural networks for short-term load forecasting in microgrids environment. Energy 75(0), 252–264. http://www.sciencedirect.com/science/article/pii/S0360544214008871 (2014)
Hippert, H., Pedreira, C., Souza, R.: Neural networks for short-term load forecasting: a review and evaluation. IEEE Trans. Power Syst. 16(1), 44–55 (2001)
Khosravi, A., Nahavandi, S., Creighton, D.C., Atiya, A.F.: Comprehensive review of neural network-based prediction intervals and new advances. IEEE Trans. Neural Netw. 22(9), 1341–1356 (2011)
O\(\hat{\rm a}\)Connor, M., Remus, W., Griggs, K.: The asymmetry of judgemental confidence intervals in time series forecasting. Int. J. Forecast. 17(4), 623–633. http://www.sciencedirect.com/science/article/pii/S0169207001001030 (2001)
Papadopoulos, H., Haralambous, H.: Reliable prediction intervals with regression neural networks. Neural Netw. 24(8), 842–851. http://www.sciencedirect.com/science/article/pii/S089360801100150X (2011). Artificial Neural Networks: Selected Papers from ICANN 2010
Proedrou, K., Nouretdinov, I., Vovk, V., Gammerman, A.: Transductive confidence machines for pattern recognition. ECML 2002, pp. 381–390. Springer, Berlin (2001)
Quan, H., Srinivasan, D., Khosravi, A.: Short-term load and wind power forecasting using neural network-based prediction intervals. IEEE Trans. Neural Netw. Learn. Syst. 25(2), 303–315 (2014)
Quan, H., Srinivasan, D., Khosravi, A.: Uncertainty handling using neural network-based prediction intervals for electrical load forecasting. Energy 73(0), 916–925. http://www.sciencedirect.com/science/article/pii/S0360544214008032 (2014)
Rodrigues, P.P., Gama, J.: A system for analysis and prediction of electricity-load streams. Intell. Data Anal. 13(3), 477–496 (2009)
Saunders, C., Gammerman, A., Vovk, V.: Transduction with confidence and credibility. Proc. Int. Jt. Conf. Artif. Intell. 16, 722–726 (1999)
Acknowledgments
This work was supported by NORTE-07-0124-FEDER-000056 financed by ON.2\(-\)O Novo Norte, under the National Strategic Reference Framework, through the Development Fund, and by national funds, through FCT. Additionally, it was funded by European Commission through MAESTRA (ICT-2013-612944).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this paper
Cite this paper
Almeida, V., Gama, J. (2016). Measures for Combining Prediction Intervals Uncertainty and Reliability in Forecasting. In: Burduk, R., Jackowski, K., Kurzyński, M., Woźniak, M., Żołnierek, A. (eds) Proceedings of the 9th International Conference on Computer Recognition Systems CORES 2015. Advances in Intelligent Systems and Computing, vol 403. Springer, Cham. https://doi.org/10.1007/978-3-319-26227-7_14
Download citation
DOI: https://doi.org/10.1007/978-3-319-26227-7_14
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-26225-3
Online ISBN: 978-3-319-26227-7
eBook Packages: EngineeringEngineering (R0)