Abstract
A prior distribution of weights for Multilayer feedforward neural network in Bayesian point of view plays a central role toward generalization. In this context, we propose a new prior law for weights parameters which motivate the network regularization more than \(l_ {1}\) and \(l_ {2}\) early proposed. To train the network, we have based on Hamiltonian Monte Carlo, it is used to simulate the prior and the posterior distribution. The generated samples are used to approximate the gradient of the evidence which allows to re-estimate the hyperparameters that balance a trade off between the likelihood term and regularized term, on the other hand we use the obtained posterior samples to estimate the network output. The case problem studied in this paper includes a regression and classification tasks. The obtained results illustrate the advantages of our approach in term of error rate compared to old approach, unfortunately our method consomme time before convergence.
Similar content being viewed by others
References
Angelov P (2014) Outside the box: an alternative data analytics framework. J Autom Mob Robot Intell Syst 8(2):29–35
Angelov P, Sadeghi-Tehran P, Ramezani R (2011) An approach to automatic real-time novelty detection, object identification, and tracking in video streams based on recursive density estimation and evolving takagi-sugeno fuzzy systems. Int J Intell Syst 26(3):189–205
Angelov P, Kasabov N (2005) Evolving computational intelligence systems. In Proceedings of the 1st international workshop on genetic fuzzy systems, pp 76–82
Bache K, Lichman M (2013) UCI machine learning repository. https://archive.ics.uci.edu/ml/datasets.html. Accessed 2013
Bishop CM (1995) Neural networks for pattern recognition. Oxford University Press, Oxford
Buntine WL, Weigend AS (1991) Bayesian back-propagation. Complex Syst 5(6):603–643
de Campos Souza PV, Silva GRL, Torres LCB (2018) Uninorm based regularized fuzzy neural networks. In: 2018 IEEE conference on evolving and adaptive intelligent systems (EAIS), pp 1–8, IEEE
de Campos Souza PV, Torres LCB, Guimaraes AJ, Araujo VS, Araujo VJS, Rezende TS (2019) Data density-based clustering for regularized fuzzy neural networks based on nullneurons and robust activation function. Soft Comput. https://doi.org/10.1007/s00500-019-03792-z
Duane S, Kennedy AD, Pendleton BJ, Roweth D (1987) Hybrid Monte Carlo. Phys Lett B 195(2):216–222
Ekonomou L, Christodoulou CA, Mladenov V (2016) An artificial neural network software tool for the assessment of the electric field around metal oxide surge arresters. Neural Comput Appl 27(5):1143–1148
Fan Q, Zurada JM, Wu W (2014) Convergence of online gradient method for feedforward neural networks with smoothing l1/2 regularization penalty. Neurocomputing 131:208–216
Kocadağlı O, Aşıkgil B (2014) Nonlinear time series forecasting with bayesian neural networks. Expert Syst Appl 41(15):6596–6610
Lan S (2013) Advanced Bayesian computational methods through geometric techniques. University of California, Irvine
Lan S, Stathopoulos V, Shahbaba B, Girolami M (2015) Markov chain Monte Carlo from lagrangian dynamics. J Comput Graph Stat 24(2):357–378
Leimkuhler B, Reich S (2004) Simulating hamiltonian dynamics, vol 14. Cambridge University Press, Cambridge
Lughofer E (2013) On-line assurance of interpretability criteria in evolving fuzzy systems-achievements, new concepts and open issues. Inf Sci 251:22–46
MacKay DJ (1992a) The evidence framework applied to classification networks. Neural Comput 4(5):720–736
MacKay DJ (1992b) A practical bayesian framework for backpropagation networks. Neural Comput 4(3):448–472
Marwala T (2007) Bayesian training of neural networks using genetic programming. Pattern Recognit Lett 28(12):1452–1458
Neal RM (1993) Probabilistic inference using Markov chain Monte Carlo methods. Department of Computer Science, University of Toronto Toronto, Ontario, Canada
Neal RM et al (2011) Mcmc using hamiltonian dynamics. Handb Markov Chain Monte Carlo 2(11):2
Neal RM (2012) Bayesian learning for neural networks, vol 118. Springer Science & Business Media, Berlin
Niu D-X, Shi H-F, Wu DD (2012) Short-term load forecasting using bayesian neural networks learned by hybrid monte carlo algorithm. Appl Soft Comput 12(6):1822–1827
Precup R-E, Filip H-I, Rădac M-B, Petriu EM, Preitl S, Dragoş C-A (2014) Online identification of evolving takagi-sugeno-kang fuzzy models for crane systems. Appl Soft Comput 24:1155–1163
Ramchoun H, Idrissi MAJ, Ghanou Y, Ettaouil M (2017) New modeling of multilayer perceptron architecture optimization with regularization: an application to pattern classification. IAENG Int J Comput Sci 44(3):261–269
Ramchoun H, Ettaouil M (2018) Hamiltonian monte carlo based on evidence framework for bayesian learning to neural network. Soft Comput. https://doi.org/10.1007/s00500-018-3138-5
Tibshirani R (1996) Regression shrinkage and selection via the lasso. J R Stat Soc Ser B (Methodological) 58:267–288
Vivarelli F, Williams CK (2001) Comparing bayesian neural network algorithms for classifying segmented outdoor images. Neural Netw 14(4–5):427–437
Wu W, Fan Q, Zurada JM, Wang J, Yang D, Liu Y (2014) Batch gradient method with smoothing l1/2 regularization for training of feedforward neural networks. Neural Netw 50:72–78
Zhang H, Tang Y (2017) Online gradient method with smoothing \(l_{0}\) regularization for feedforward neural networks. Neurocomputing 224:1–8
Zhang H, Tang Y, Liu X (2015) Batch gradient training method with smoothing \(l_{0}\) regularization for feedforward neural networks. Neural Comput Appl 26(2):383–390
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflicts of interest.
Ethical approval
This article does not contain any studies with human participants or animals performed by any of the authors.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Ramchoun, H., Ettaouil, M. New prior distribution for Bayesian neural network and learning via Hamiltonian Monte Carlo. Evolving Systems 11, 661–671 (2020). https://doi.org/10.1007/s12530-019-09288-3
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12530-019-09288-3