Abstract
We consider the problem of on-line prediction competitive with a benchmark class of continuous but highly irregular prediction rules. It is known that if the benchmark class is a reproducing kernel Hilbert space, there exists a prediction algorithm whose average loss over the first N examples does not exceed the average loss of any prediction rule in the class plus a “regret term” of O(N − 1/2). The elements of some natural benchmark classes, however, are so irregular that these classes are not Hilbert spaces. In this paper we develop Banach-space methods to construct a prediction algorithm with a regret term of O(N \(^{\rm -1/{\it p}}\)), where p∈(2,∞) and p–2 reflects the degree to which the benchmark class fails to be a Hilbert space.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Adams, R.A., Fournier, J.J.F.: Sobolev Spaces, 2nd edn. Pure and Applied Mathematics, vol. 140. Academic Press, Amsterdam (2003)
Aronszajn, N.: Theory of reproducing kernels. Transactions of the American Mathematical Society 68, 337–404 (1950)
Auer, P., Cesa-Bianchi, N., Gentile, C.: Adaptive and self-confident on-line learning algorithms. Journal of Computer and System Sciences 64, 48–75 (2002)
Cesa-Bianchi, N., Conconi, A., Gentile, C.: On the generalization ability of on-line learning algorithms. IEEE Transactions on Information Theory 50, 2050–2057 (2004)
Cesa-Bianchi, N., Long, P.M., Warmuth, M.K.: Worst-case quadratic loss bounds for on-line prediction of linear functions by gradient descent. IEEE Transactions on Neural Networks 7, 604–619 (1996)
Clarkson, J.A.: Uniformly convex spaces. Transactions of the American Mathematical Society 40, 396–414 (1936)
Figiel, T.: On the moduli of convexity and smoothness. Studia Mathematica 56, 121–155 (1976), available free of charge at: http://matwbn.icm.edu.pl
Hanner, O.: On the uniform convexity of L p and l p. Arkiv för Matematik 3, 239–244 (1956)
Karatzas, I., Shreve, S.E.: Brownian Motion and Stochastic Calculus, 2nd edn. Springer, New York (1991)
Kivinen, J., Warmuth, M.K.: Exponential Gradient versus Gradient Descent for linear predictors. Information and Computation 132, 1–63 (1997)
Kolmogorov, A.N.: Grundbegriffe der Wahrscheinlichkeitsrechnung. Springer, Berlin (1933); English translation: Foundations of the theory of probability. Chelsea, New York (1950)
Lindenstrauss, J.: On the modulus of smoothness and divergent series in Banach spaces. Michigan Mathematical Journal 10, 241–252 (1963)
Liokumovich, V.I.: The existence of B-spaces with non-convex modulus of convexity (in Russian). ИзвестиЯ вьıсших УЧебньІх заведений. Математика 12, 43–50 (1973)
Nikolsky, S.M.: On imbedding, continuation and approximation theorems for differentiable functions of several variables. Russian Mathematical Surveys 16(5), 55–104 (1961), Russian original in: Успехи математических наук 16(5), 63–114 (1961)
Tyrrell Rockafellar, R.: Convex Analysis. Princeton University Press, Princeton (1970)
Rudin, W.: Functional Analysis, 2nd edn. McGraw-Hill, Boston (1991)
Shafer, G., Vovk, V.: Probability and Finance: It’s Only a Game! Wiley, New York (2001)
Sorenson, H.W.: Least-squares estimation: from Gauss to Kalman. IEEE Spectrum 7, 63–68 (1970)
Steinwart, I.: On the influence of the kernel on the consistency of support vector machines. Journal of Machine Learning Research 2, 67–93 (2001)
Vovk, V.: Competing with wild prediction rules. Technical Report arXiv:cs.LG/0512059, arXiv.org., e-Print archive (January 2006)
Vovk, V.: On-line regression competitive with reproducing kernel Hilbert spaces. In: Cai, J.-Y., Cooper, S.B., Li, A. (eds.) TAMC 2006. LNCS, vol. 3959, pp. 452–463. Springer, Heidelberg (2006); Full version: Technical Report arXiv:cs.LG/0511058, arXiv.org e-Print archive (January 2006)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2006 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Vovk, V. (2006). Competing with Wild Prediction Rules. In: Lugosi, G., Simon, H.U. (eds) Learning Theory. COLT 2006. Lecture Notes in Computer Science(), vol 4005. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11776420_41
Download citation
DOI: https://doi.org/10.1007/11776420_41
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-35294-5
Online ISBN: 978-3-540-35296-9
eBook Packages: Computer ScienceComputer Science (R0)