Abstract
The majority of results in computational learning theory are concerned with concept learning, i.e. with the special case of function learning for classes of functions with range {0, 1}. Much less is known about the theory of learning functions with a larger range such as ℕ or ℝ. In particular relatively few results exist about the general structure of common models for function learning, and there are only very few nontrivial function classes for which positive learning results have been exhibited in any of these models.
We introduce in this paper the notion of a binary branching adversary tree for function learning, which allows us to give a somewhat surprising equivalent characterization of the optimal learning cost for learning a class of real-valued functions (in terms of a max-min definition which does not involve any “learning” model).
Another general structural result of this paper relates the cost for learning a union of function classes to the learning costs for the individual function classes.
Furthermore, we exhibit an efficient learning algorithm for learning convex piecewise linear functions from ℝd into ℝ. Previously, the class of linear functions from ℝd into ℝ was the only class of functions with multidimensional domain that was known to be learnable within the rigorous framework of a formal model for online learning.
Finally we give a sufficient condition for an arbitrary class\(\mathcal{F}\) of functions from ℝ into ℝ that allows us to learn the class of all functions that can be written as the pointwise maximum ofk functions from\(\mathcal{F}\). This allows us to exhibit a number of further nontrivial classes of functions from ℝ into ℝ for which there exist efficient learning algorithms.
Article PDF
Similar content being viewed by others
References
Auer, P., & Long, P.M. (1994). Simulating access to hidden information while learning,Proceedings of the 26th Annual ACM Symposium on the Theory of Computing (pp. 263–272).
Auer, P., Long, P.M., Maass, W., & Woeginger, G.J. (1993). On the complexity of function learning,Proceedings of the Sixth Annual ACM Conference on Computational Learning Theory, pp. 392–401.
Angluin, D. (1988). Queries and concept learning,Machine Learning, 2(4):319–342.
Barland, I. (1992). Some ideas on learning with directional feedback. Master's thesis, Computer Science Department, UC Santa Cruz.
Berlekamp, E.R. (1968). Block coding for the binary symmetric channel with noiseless, delayless feedback, InError Correcting Codes (pp. 61–85), New York: Wiley.
Barzdin, J.M., & Frievald, R.V. (1972). On the prediction of general recursive functions,Soviet Math. Doklady, 13:1224–1228.
Cesa-Bianchi, N., Freund, Y., Helmbold, D.P., & Warmuth, M.K. (in press). On-line prediction and conversion strategies. InProceedings of the First Euro-COLT Workshop, The Institute of Mathematics and its Applications, to appear.
Cesa-Bianchi, N., Long, P.M., & Warmuth, M.K. (1993). Worst-case quadratic loss bounds for a generalization of the Widrow-Hoff rule. InProceedings of the 6th Annual Workshop on Comput. Learning Theory (pp. 429–438).
Dawid, A. (1984). Statistical theory: The sequential approach.Journal of the Royal Statistical Society (Series A), pp. 278–292.
Faber, V., & Mycielski, J. (1991). Applications of learning theorems.Fundamenta Informaticae, 15(2):145–167.
Feder, M., Merhav, N., & Gutman, M. (1992). Universal prediction of individual sequences.IEEE Transactions of Information Theory, 38:1258–1270.
Kimber, D., & Long, P.M. (1992). The learning complexity of smooth functions of a single variable. InProc. 5th Annu. Workshop on Comput. Learning Theory (pp. 153–159).
Kearns, M.J., Schapire, R.E., & Sellie, L.M. (1992). Toward efficient agnostic learning. InProc. 5th Annu. Workshop on Comput. Learning Theory (pp. 341–352).
Littlestone, N. (1988). Learning quickly when irrelevant attributes abound: A new linear-threshold algorithm,Machine Learning, 2:285–318.
Littlestone, N. (1989).Mistake Bounds and Logarithmic Linear-threshold Learning Algorithms PhD thesis. Technical Report UCSC-CRL-89-11, University of California Santa Cruz.
Littlestone, N., Long, P.M., & Warmuth, M.K. (1991). On-line learning of linear functions. InProc. of the 23rd Symposium on Theory of Computing (pp. 465–475).
Littlestone, N., & Warmuth, M.K. (1991). The weighted majority algorithm, Technical Report UCSC-CRL-91-28, UC Santa Cruz. A preliminary version appeared inthe Proceedings of the 30th Annual IEEE Symposium of the Foundations of Computer Science.
Long, P.M., & Warmuth, M.K. (in press). Composite geometric concepts and polynomial predictability.Inform. Comput.
Maass, W. (1991). On-line learning with an oblivious environment and the power of randomization.Proc. 4th Annu. Workshop on Comput. Learning Theory (pp. 167–175).
Maass, W., & Turán, G. (1992). Lower bound methods and separation results for on-line learning models.Machine Learning, 9:107–145.
Mycielski, J. (1988). A learning algorithm for linear operators.Proceedings of the American Mathematical Society, 103(2):547–550.
Rivest, R.L., Meyer, A.R., Kleitman, D.J., Winklmann, K., & Spencer, J. (1980). Coping with errors in binary search procedures.Journal of Computer and System Sciences, 20:396–404.
Sauer, N. (1972). On the density of families of sets.J. Combinatorial Theory (A), 13:145–147.
Spencer, J. (1992). Ulam's searching game with a fixed number of lies.Theoretical Computer Science, 95(2):307–321.
Uspensky, J.V. (1948).Theory of Equations, McGraw-Hill.
Vovk, V. (1990). Aggregating strategies. InProc. 3rd Annu. Workshop on Comput. Learning Theory (pp. 371–383).
Vovk, V. (1992). Universal forecasting algorithms.Inform. Comput., 96(2):245–277.
Author information
Authors and Affiliations
Rights and permissions
About this article
Cite this article
Auer, P., Long, P.M., Maass, W. et al. On the complexity of function learning. Mach Learn 18, 187–230 (1995). https://doi.org/10.1007/BF00993410
Received:
Accepted:
Issue Date:
DOI: https://doi.org/10.1007/BF00993410