Abstract
The relationship between generalization ability, neural network size and function complexity have been analyzed in this work. The dependence of the generalization process on the complexity of the function implemented by neural architecture is studied using a recently introduced measure for the complexity of the Boolean functions. Furthermore an association rule discovery (ARD) technique was used to find associations among subsets of items in the whole set of simulations results. The main result of the paper is that for a set of quasi-random generated Boolean functions it is found that large neural networks generalize better on high complexity functions in comparison to smaller ones, which performs better in low and medium complexity functions.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Haykin, S.: Neural Networks: A Comprehensive Foundation. Macmillan/IEEE Press (1994)
Baum, E.B., Haussler, D.: What size net gives valid generalization? Neural Computation 1, 151–160 (1989)
Lawrence, S., Giles, C.L., Tsoi, A.C.: What Size Neural Network Gives Optimal Generalization? Convergence Properties of Backpropagation. In: Technical Report UMIACS-TR-96-22 and CS-TR-3617, Institute for Advanced Computer Studies, Univ. of Maryland (1996)
Caruana, R., Lawrence, S., Giles, C.L.: Overfitting in Neural Networks: Backpropagation, Conjugate Gradient, and Early Stopping. In: Leen, T.K., Dietterich, T.G., Tresp, V. (eds.) Advances in Neural Information Processing Systems, vol. 13, pp. 402–408. MIT Press, Cambridge (2001)
Krogh, A., Hertz, J.A.: A simple weight decay can improve generalization. In: Moody, J.E., Hanson, S.J., Lippmann, R.P. (eds.) Advances in Neural Information Processing Systems, vol. 4, pp. 950–957. Morgan Kaufmann, San Mateo (1992)
Prechelt, L.: Automatic Early Stopping Using Cross Validation: Quantifying the Criteria. Neural Networks 11, 761–767 (1998)
Setiono, R.: Feedforward neural network construction using cross-validation. Neural Computation 13, 2865–2877 (2001)
Bartlett, P.L.: For valid generalization the size of the weights is more important than the size of the network. In: Mozer, M.C., Jordan, M.I., Petsche, T. (eds.) Advances in Neural Information Processing Systems, vol. 9, pp. 134–140. MIT Press, Cambridge (1997)
Franco, L., Anthony, M.: On a generalization complexity measure for Boolean functions. In: Proceedings of the 2004 IEEE International Joint Conference on Neural Networks, pp. 973–978. IEEE Press, Los Alamitos (2004)
Franco, L.: Generalization ability of Boolean functions implemented in feedforward neural networks. Neurocomputing (2006) (In Press)
Franco, L., Anthony, M.: The influence of oppositely classified examples on the generalization complexity of Boolean functions. IEEE Transactions on Neural Networks (2006) (In Press)
Wegener, I.: The complexity of Boolean functions. Wiley and Sons Inc., Chichester (1987)
Siu, K.Y., Roychowdhury, V.P., Kailath, T.: Depth-Size Tradeoffs for Neural Computation. IEEE Transactions on Computers 40, 1402–1412 (1991)
Franco, L., Cannas, S.A.: Non glassy ground-state in a long-range antiferromagnetic frustrated model in the hypercubic cell. Physica A 332, 337–348 (2004)
Franco, L., Cannas, S.A.: Generalization and Selection of Examples in Feedforward Neural Networks. Neural Computation 12(10), 2405–2426 (2000)
Franco, L., Cannas, S.A.: Generalization Properties of Modular Networks: Implementing the Parity Function. IEEE Transactions on Neural Networks 12, 1306–1313 (2001)
Becquet, C., Blachon, S., Jeudy, B., Boulicaut, J.F., Gandrillon, O.: Strong association rules mining for large-scale gene-expression data analysis: A case study on human SAGE data. Genome Biology 3, 1–16 (2002)
Creighton, C., Hanash, S.: Mining gene expressions databases for association rules. Bioinformatics 19, 79–86 (2003)
Agrawal, R., Imielinski, T., Swami, A.: Mining associations rules between sets of items in large databases. In: Proceedings of the ACM SIGMOD international conference on Management of data, Washignton D.C, pp. 207–216 (1993)
Brian, S., Motwani, R., Silverstein, C.: Beyond Market baskets: Generalizing associations rules to correlations. In: Proceedings of the ACM SIGMOD conference, Tucson, pp. 265–276 (1997)
Franco, L., Jerez, J.M., Bravo, J.M.: Role of function complexity and network size in the generalization ability of feedforward networks. In: Cabestany, J., Prieto, A.G., Sandoval, F. (eds.) IWANN 2005. LNCS, vol. 3512, pp. 1–8. Springer, Heidelberg (2005)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Gómez, I., Franco, L., Subirats, J.L., Jerez, J.M. (2006). Neural Network Architecture Selection: Size Depends on Function Complexity. In: Kollias, S.D., Stafylopatis, A., Duch, W., Oja, E. (eds) Artificial Neural Networks – ICANN 2006. ICANN 2006. Lecture Notes in Computer Science, vol 4131. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11840817_13
Download citation
DOI: https://doi.org/10.1007/11840817_13
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-38625-4
Online ISBN: 978-3-540-38627-8
eBook Packages: Computer ScienceComputer Science (R0)