Skip to main content
SpringerLink
Log in

A Model Selection Method Based on Bound of Learning Coefficient

  • Conference paper
Artificial Neural Networks – ICANN 2006 (ICANN 2006)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 4132))

Included in the following conference series:

Abstract

To decide the optimal size of learning machines is a central issue in the statistical learning theory, and that is why some theoretical criteria such as the BIC are developed. However, they cannot be applied to singular machines, and it is known that many practical learning machines e.g. mixture models, hidden Markov models, and Bayesian networks, are singular. Recently, we proposed the Singular Information Criterion (SingIC), which allows us to select the optimal size of singular machines. The SingIC is based on the analysis of the learning coefficient. So, the machines, to which the SingIC can be applied, are still limited. In this paper, we propose an extension of this criterion, which enables us to apply it to many singular machines, and evaluate the efficiency in Gaussian mixtures. The results offer an effective strategy to select the optimal size.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. Amari, S., Ozeki, T.: Differential and algebraic geometry of multilayer perceptrons. IEICE Trans E84-A 1, 31–38 (2001)

    Google Scholar 

  2. Hartigan, J.A.: A failure of likelihood asymptotics for normal mixtures. In: Proc. of the Berkeley Conference in Honor of J.Neyman and J.Kiefer, vol. 2, pp. 807–810 (1985)

    Google Scholar 

  3. Akaike, H.: A new look at the statistical model identification. IEEE Trans. on Automatic Control 19, 716–723 (1974)

    Article  MATH  MathSciNet  Google Scholar 

  4. Schwarz, G.: Estimating the dimension of a model. Annals of Statistics 6(2), 461–464 (1978)

    Article  MATH  MathSciNet  Google Scholar 

  5. Rissanen, J.: Stochastic complexity and modeling. Annals of Statistics 14, 1080–1100 (1986)

    Article  MATH  MathSciNet  Google Scholar 

  6. Watanabe, S.: Algebraic analysis for non-identifiable learning machines. Neural Computation 13(4), 899–933 (2001)

    Article  MATH  Google Scholar 

  7. Yamazaki, K., Nagata, K., Watanabe, S.: A new method of model selection based on learning coefficient. In: Proc. of International Symposium on Nonlinear Theory and its Applications, pp. 389–392 (2005)

    Google Scholar 

  8. Aoyagi, M., Watanabe, S.: The generalization error of reduced rank regression in bayesian estimation. In: Proc. of ISITA, pp. 1068–1073 (2004)

    Google Scholar 

  9. Yamazaki, K., Watanabe, S.: Learning coefficient of hidden markov models. Technical Report of IEICE NC2005-14, 37–42 (2005)

    Google Scholar 

  10. Hosino, T., Watanabe, K., Watanabe, S.: Stochastic complexity of variational bayesian hidden markov models. In: Proc. of International Joint Conference on Neural Networks, pp. 1114–1119 (2005)

    Google Scholar 

  11. Watanabe, K., Watanabe, S.: Variational bayesian stochastic complexty of mixture models. MIT press, Cambridge (to appear)

    Google Scholar 

  12. Watanabe, S., Yamazaki, K., Aoyagi, M.: Kullback information of normal mixture is not an analytic function. Technical Report of IEICE NC2004-50, 41–46 (2004) (in Japanese)

    Google Scholar 

  13. Yamazaki, K., Watanabe, S.: Stochastic complexity of bayesian networks. In: Proc. of UAI, pp. 592–599 (2003)

    Google Scholar 

  14. Ogata, Y.: A monte carlo method for an objective bayesian procedure. Ann. Inst. Statis. Math. 42(3), 403–433 (1990)

    Article  MATH  MathSciNet  Google Scholar 

  15. Hukushima, K., Nemoto, K.: Exchange monte carlo method and application to spin glass simulations. Journal of Physical Society of Japan 65(6), 1604–1608 (1996)

    Article  MathSciNet  Google Scholar 

  16. Yamazaki, K., Watanabe, S.: Singularities in complete bipartite graph-type boltzmann machines and upper bounds of stochastic complexities. IEEE Trans. On Neural Networks 16(2), 312–324 (2005)

    Article  Google Scholar 

  17. Yamazaki, K., Watanabe, S.: Generalization errors in estimation of stochastic context-free grammar. In: The IASTED International Conference on ASC, pp. 183–188 (2005)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Precision and Intelligence Laboratory, Tokyo Institute of Technology, R2-5, 4259 Nagatsuta, Midori-ku, Yokohama, 226-8503, Japan

    Keisuke Yamazaki & Sumio Watanabe

  2. Dept. of Computational Intelligence and Systems Science, Tokyo Institute of Technology, R2-5, 4259 Nagatsuta, Midori-ku, Yokohama, 226-8503, Japan

    Kenji Nagata

  3. Fraunhofer FIRST, IDA, Kekuléstr. 7, 12489, Berlin, Germany

    Keisuke Yamazaki & Klaus-Robert Müller

Authors
  1. Keisuke Yamazaki
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kenji Nagata
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sumio Watanabe
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Klaus-Robert Müller
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. School of Electrical and Computer Engineering, Image, Video and Multimedia Systems Laboratory, National Technical University of Athens, 157 80, Zographou, GR, Greece

    Stefanos Kollias

  2. Department of Electrical and Computer Engineering, National Technical University of Athens, 15780, Zographou, Greece

    Andreas Stafylopatis

  3. Department of Informatics, Nicolaus Copernicus University, Toruń, Poland

    Włodzisław Duch

  4. Adaptive Informatics Research Centre, Helsinki University of Technology, P.O. Box 5400, 02015, HUT, Finland

    Erkki Oja

Rights and permissions

Reprints and permissions

Copyright information

© 2006 Springer-Verlag Berlin Heidelberg

About this paper

Cite this paper

Yamazaki, K., Nagata, K., Watanabe, S., Müller, KR. (2006). A Model Selection Method Based on Bound of Learning Coefficient. In: Kollias, S., Stafylopatis, A., Duch, W., Oja, E. (eds) Artificial Neural Networks – ICANN 2006. ICANN 2006. Lecture Notes in Computer Science, vol 4132. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11840930_38

Download citation

  • DOI: https://doi.org/10.1007/11840930_38

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-540-38871-5

  • Online ISBN: 978-3-540-38873-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation