Abstract
Multilayer perceptrons (MLP) has been proven to be very successful in many applications including classification. The activation function is the source of the MLP power. Careful selection of the activation function has a huge impact on the network performance. This paper gives a quantitative comparison of the four most commonly used activation functions, including the Gaussian RBF network, over ten real different datasets. Results show that the sigmoid activation function substantially outperforms the other activation functions. Also, using only the needed number of hidden units in the MLP, we improved its conversion time to be competitive with the RBF networks most of the time.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Lu, B., Evans, B.L.: Channel Equalization by Feedforward Neural Networks. In: IEEE Int. Symposium on Circuits and Systems, Orlando, FL, vol. 5, pp. 587–590 (1999)
Michie, D., Spiegelhalter, D.J., Taylor, C.C.: Machine Learning, Neural and Statistical Classification. Elis Horwood, London (1994)
Piekniewski, F., Rybicki, L.: Visual Comparison of Performance for Different Activation Functions in MLP Networks. In: IJCNN 2004 & FUZZ-IEEE, Budapest, vol. 4, pp. 2947–2953 (2004)
Dorffner, G.: A Unified Framework for MLPs and RBFNs: Introducing Conic Section Function Networks. Cybernetics and Systems 25(4), 511–554 (1994)
Haykin, S.: Neural Networks A Comprehensive Introduction. Prentice Hall, New Jersey (1999)
Huang, G., Chen, Y., Babri, H.A.: Classification Ability of Single Hidden Layer Feedforward Neural Networks. IEEE Transactions on Neural Networks 11(3), 799–801 (2000)
Le Cun, Y., Touresky, D., Hinton, G., Sejnowski, T.: A Theoretical Framework for Backpropagation. The Connectionist Models Summer School, 21–28 (1988)
Li, Y., Pont, M.J., Jones, N.B.: A Comparison of the Performance of Radial Basis Function and Multi-layer Perceptron Networks in Condition Monitoring and Fault Diagnosis. In: The International Conference on Condition Monitoring, Swansea, pp. 577–592 (1999)
Arahal, M.R., Camacho, E.F.: Application of the Ran Algorithm to the Problem of Short Term Load Forecasting. Technical Report, University of Sevilla, Sevilla (1996)
Finan, R.A., Sapeluk, A.T., Damper, R.I.: Comparison of Multilayer and Radial Basis Function Neural Networks for Text-Dependent Speaker Recognition. In: IEEE Int. Conf. on Neural Networks, Washington DC, vol. 4, pp. 1992–1997 (1996)
Karkkainen, T.: MLP in Layer-Wise Form with Applications to Weight Decay. Neural Computation 14(6), 1451–1480 (2002)
Werbos, P.J.: Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences. Doctoral Thesis, Applied Mathematics. Harvard University. Boston (1974)
Wang, D., Huang, G.: Protein Sequence Classification Using Extreme Learning Machine. In: IJCNN 2005, Montréal, vol. 3, pp. 1406–1411 (2005)
Duch, W., Jankowski, N.: Survey of Neural Transfer Functions. Neural Computing Surveys 2, 163–212 (1999)
Duch, W., Jankowski, N.: Transfer functions: Hidden Possibilities for Better Neural Networks. In: 9th European Symposium on Artificial Neural Network, Bruges, pp. 81–94 (2001)
Hu, Y., Hwang, J.: Handbook of Neural Network Signal Processing, 3rd edn. CRC-Press, Florida (2002)
Zurada, J.M.: Introduction to Artificial Neural Systems. PWS Publishing, Boston (1999)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Shenouda, E.A.M.A. (2006). A Quantitative Comparison of Different MLP Activation Functions in Classification. In: Wang, J., Yi, Z., Zurada, J.M., Lu, BL., Yin, H. (eds) Advances in Neural Networks - ISNN 2006. ISNN 2006. Lecture Notes in Computer Science, vol 3971. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11759966_125
Download citation
DOI: https://doi.org/10.1007/11759966_125
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-34439-1
Online ISBN: 978-3-540-34440-7
eBook Packages: Computer ScienceComputer Science (R0)