Summary
Capability of generalization in learning from examples can be modeled using regularization, which has been developed as a tool for improving stability of solutions of inverse problems. Theory of inverse problems has been developed to solve various tasks in applied science such as acoustics, geophysics and computerized tomography. Such problems are typically described by integral operators. It is shown that learning from examples can be reformulated as an inverse problem defined by an evaluation operator. This reformulation allows one to characterize optimal solutions of learning tasks and design learning algorithms based on numerical solutions of systems of linear equations.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Aizerman M A, Braverman E M, Rozonoer L I (1964) Theoretical foundations of potential function method in pattern recognition learning. Automation and Remote Control 28:821-837
Aronszajn N (1950) Theory of reproducing kernels. Transactions of AMS 68:33-404
Berg C, Christensen J P R, Ressel P (1984) Harmonic Analysis on Semigroups. New York, Springer-Verlag
Bertero M (1989) Linear inverse and ill-posed problems. Advances in Electronics and Electron Physics 75:1-120
Bjorck A (1996) Numerical methods for least squares problem. SIAM
Boser B, Guyon I, Vapnik V N (1992) A training algorithm for optimal margin clasifiers. In Proceedings of the 5th Annual ACM Workshop on Computational Learning Theory (Ed. Haussler D), pp. 144-152. ACM Press
Cortes C, Vapnik V N (1995) Support-vector networks. Machine Learning 20:273-297
Courant R, Hilbert D (1962) Methods of Mathematical Physics, vol. 2. New York, Wiley
Cristianini N, Shawe-Taylor J (2000) An Introduction to Support Vector Machines. Cambridge, Cambridge University Press
Cucker F, Smale S (2002) On the mathematical foundations of learning. Bulletin of the AMS 39:1-49
De Mol C (1992) A critical survey of regularized inversion method. In Inverse Problems in Scattering and Imaging (Eds. Bertero M, Pike E R), pp. 346-370. Bristol, Adam Hilger
De Vito E, Rosasco L, Caponnetto A, De Giovannini U, Odone F (2005) Learning from examples as an inverse problem. Journal of Machine Learning Research 6:883-904
Di Chiro G, Brooks R A (1979) The 1979 Nobel prize in physiology or medicine. Science 206:1060-1062
Dunford N, Schwartz J T (1963) Linear Operators. Part II: Spectral Theory. New York, Interscience Publishers
Engl H W, Hanke M, Neubauer A (2000) Regularization of Inverse Problems. Dordrecht, Kluwer
Friedman A (1982) Modern Analysis. New York, Dover
Girosi F (1998) An equivalence between sparse approximation and support vector machines. Neural Computation 10:1455-1480 (AI Memo No 1606, MIT)
Girosi F, Jones M, Poggio T (1995) Regularization theory and neural network architectures. Neural Computation 7:219-269
Groetch C W (1977) Generalized Inverses of Linear Operators. New York, Dekker
Hadamard J (1902) Sur les problèmes aux dérivées partielles et leur signification physique. Bulletin of University of Princeton 13:49
Hansen P C (1998) Rank-Deficient and Discrete Ill-Posed Problems. Philadelphia, SIAM
Kecman V (2001) Learning and Soft Computing. Cambridge, MIT Press.
Kůrková V (2003) High-dimensional approximation by neural networks. Chapter 4 in Advances in Learning Theory: Methods, Models and Applications (Eds. Suykens J et al.), pp. 69-88. Amsterdam, IOS Press
Kůrková V (2004) Learning from data as an inverse problem. In: COMP-STAT 2004 - Proceedings on Computational Statistics (Ed. Antoch J), pp. 1377-1384. Heidelberg, Physica-Verlag/Springer
Kůrková V (2005) Neural network learning as an inverse problem. Logic Journal of IGPL 13:551-559
Kůrková V, Sanguineti M (2005) Error estimates for approximate optimization by the extended Ritz method. SIAM Journal on Optimization 15:461-487
Kůrková V, Sanguineti M (2005) Learning with generalization capability by kernel methods with bounded complexity. Journal of Complexity 21:350-367
Moore E H (1920) Abstract. Bull. Amer. Math. Soc. 26:394-395
Parzen E (1966) An approach to time series analysis. Annals of Math. Statistics 32:951-989
Penrose R (1955) A generalized inverse for matrices. Proc. Cambridge Philos. Soc. 51:406-413
Pinkus A (1985) n-width in Approximation Theory. Berlin, Springer-Verlag
Poggio T, Girosi F (1990) Networks for approximation and learning. Proceedings IEEE 78:1481-1497
Poggio T, Smale S (2003) The mathematics of learning: dealing with data. Notices of the AMS 50:536-544
Popper K (1968) The Logic of Scientific Discovery. New York, Harper Torch Book
Rummelhart D E, Hinton G E, Williams R J (1986) Learning internal representations by error propagation. In Parallel Distributed Processing (Eds. Rummelhart D E, McClelland J L), pp. 318-362. Cambridge, MIT Press
Russo L (2004) The Forgotten Revolution. Berlin, Springer-Verlag
Schölkopf B, Smola A J (2002) Learning with Kernels - Support Vector Machines, Regularization, Optimization and Beyond. Cambridge, MIT Press
Smith D W, McIntyre R (1982) Husserl and Intentionality: A Study of Mind, Meaning, and Language. Dordrecht and Boston, D. Reidel Publishing Co.
Strichartz R S (2003) A Quide to Distribution Theory and Fourier Transforms. Singapore, World Scientific
Tikhonov A N, Arsenin V Y (1977) Solutions of Ill-posed Problems. Washington, D.C., W.H. Winston
Wahba G (1990) Splines Models for Observational Data. Philadelphia, SIAM
Werbos P J (1995) Backpropagation: Basics and New Developments. In The Handbook of Brain Theory and Neural Networks (Ed. Arbib M), pp. 134-139. Cambridge, MIT Press
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2007 Springer-Verlag Berlin Heidelberg
About this chapter
Cite this chapter
Kůrková, V. (2007). Generalization in Learning from Examples. In: Duch, W., Mańdziuk, J. (eds) Challenges for Computational Intelligence. Studies in Computational Intelligence, vol 63. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-71984-7_13
Download citation
DOI: https://doi.org/10.1007/978-3-540-71984-7_13
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-71983-0
Online ISBN: 978-3-540-71984-7
eBook Packages: EngineeringEngineering (R0)