Abstract
We present a new learning algorithm for feed-forward neural networks based on the standard Backpropagation method using an adaptive global learning rate. The adaption is based on the evolution of the error criteria but in contrast to most other approaches, our method uses the error measured on the validation set instead of the training set to dynamically adjust the global learning rate. At no time the examples of the validation set are directly used for training the network in order to maintain its original purpose of validating the training and to perform ”early stopping”. The proposed algorithm is a heuristic method consisting of two phases. In the first phase the learning rate is adjusted after each iteration such that a minimum of the error criteria on the validation set is quickly attained. In the second phase, this search is refined by repeatedly reverting to previous weight configurations and decreasing the global learning rate. We experimentally show that the proposed method rapidly converges and that it outperforms standard Backpropagation in terms of generalization when the size of the training set is reduced.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Rumelhart, D.E., McClelland, J.L.: Parallel Distributed Processing: Explorations in the Microstructure of Cognition. Foundations, vol. 1. MIT Press, Cambridge, MA, USA (1986)
Chan, L.W., Fallside, F.: An adaptive learning algorithm for backpropagation networks. Computer Speech and Language 2, 205–218 (1987)
Salomon, R.: Improved convergence rate of back-propagation with dynamic adaptation of the learning rate. In: Schwefel, H.-P., Männer, R. (eds.) Parallel Problem Solving from Nature. LNCS, vol. 496, pp. 269–273. Springer, Heidelberg (1991)
Battiti, R.: Accelerated backpropagation learning: Two optimization methods. Complex Systems 3, 331–342 (1989)
Vogl, T., Mangis, J., Rigler, J., Zink, W., Alkon, D.: Accelerating the convergence of the back-propagation method. Biological Cybernetics 59, 257–263 (1988)
Hsin, H.C., Li, C.C., Sun, M., Sclabassi, R.: An adaptive training algorithm for back-propagation neural networks. In: International Conference on Systems, Man and Cybernetics, vol. 2, pp. 1049–1052 (1992)
Palgianakos, V., Vrahatis, M., Magoulas, G.: Nonmonotone methods for backpropagation training with adaptive learning rate. In: International Joint Conference on Neural Networks, vol. 3, pp. 1762–1767 (1999)
LeCun, Y., Simard, P., Pearlmutter, B.: Automatic learning rate maximization by on-line estimation of the hessian’s eigenvectors. In: Hanson, S., Cowan, J., Giles, L. (eds.) Advances in Neural Information Processing Systems, vol. 5, Morgan Kaufmann Publishers, San Mateo, CA (1993)
Magoulas, G.D., Vrahatis, M.N., Androulakis, G.S.: Effective back-propagation with variable stepsize. Neural Networks 10, 69–82 (1997)
Magoulas, G.D., Vrahatis, M.N., Androulakis, G.S.: Improving the convergence of the backpropagation algorithm using learning rate adaptation methods. Neural Computation 11(7), 1769–1796 (1999)
Jacobs, R.A.: Increased rates of convergence through learning rate adaption. Neural Networks 1, 295–307 (1988)
Silva, F.M., Almeida, L.B.: Acceleration techniques for the backpropagation algorithm. In: Proceedings of the EURASIP Workshop 1990 on Neural Networks, pp. 110–119. Springer, London (1990)
Riedmiller, M., Braun, H.: A direct adaptive method for faster backpropagation learning: The RPROP algorithm. In: Proc. of the IEEE Intl. Conf. on Neural Networks, San Francisco, CA, pp. 586–591. IEEE Computer Society Press, Los Alamitos (1993)
Luenberger, D.G.: Introduction to linear and nonlinear programming. Addison-Wesley, Reading (1973)
Fahlman, S.E.: Faster-learning variations on back-propagation: An empirical study. In: Touretzky, D.S., Hinton, G.E., Sejnowski, T.J. (eds.) Connectionist Models Summer School, pp. 38–51. Morgan Kaufmann Publishers, San Mateo, CA (1988)
Leonard, J., Kramer, M.: Improvement of the backpropagation algorithm for training neural networks. Computers & Chemical Engineering 14(3), 337–341 (1990)
Schraudolph, N.: Local gain adaptation in stochastic gradient descent. In: ICANN. Ninth International Conference on Artificial Neural Networks, vol. 2, pp. 569–574 (1999)
Harmon, M., Baird III, L.: Multi-player residual advantage learning with general function approximation. Technical report, Wright Laboratory, WL/AACF, Wright-Patterson Air Force Base, OH (1996)
Almeida, L.B., Langlois, T., Amaral, J.D., Plakhov, A.: Parameter adaptation in stochastic optimization. In: Saad, D. (ed.) On-Line Learning in Neural Networks, Cambridge University Press, Cambridge (1999)
Sutton, R.S.: Adapting bias by gradient descent: an incremental version of belta-bar-delta. In: Proceedings of the Tenth International Conference on Machine Learning, Cambridge, MA, pp. 171–176 (1992)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2007 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Duffner, S., Garcia, C. (2007). An Online Backpropagation Algorithm with Validation Error-Based Adaptive Learning Rate. In: de Sá, J.M., Alexandre, L.A., Duch, W., Mandic, D. (eds) Artificial Neural Networks – ICANN 2007. ICANN 2007. Lecture Notes in Computer Science, vol 4668. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-74690-4_26
Download citation
DOI: https://doi.org/10.1007/978-3-540-74690-4_26
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-74689-8
Online ISBN: 978-3-540-74690-4
eBook Packages: Computer ScienceComputer Science (R0)