Skip to main content
SpringerLink
Log in

Provably Fast Training Algorithms for Support Vector Machines

  • Published:
Theory of Computing Systems Aims and scope Submit manuscript

Abstract

Support Vector Machines are a family of algorithms for the analysis of data based on convex Quadratic Programming. We derive randomized algorithms for training SVMs, based on a variation of Random Sampling Techniques; these have been successfully used for similar problems. We formally prove an upper bound on the expected running time which is quasilinear with respect to the number of data points and polynomial with respect to the other parameters, i.e., the number of attributes and the inverse of a chosen soft margin parameter. [This is the combined journal version of the conference papers (Balcázar, J.L. et al. in Proceedings of 12th International Conference on Algorithmic Learning Theory (ALT’01), pp. 119–134, [2001]; Balcázar, J.L. et al. in Proceedings of First IEEE International Conference on Data Mining (ICDM’01), pp. 43–50, [2001]; and Balcázar, J.L. et al. in Proceedings of SIAM Workshop in Discrete Mathematics and Data Mining, pp. 19–29, [2002]).]

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Adler, I., Shamir, R.: A randomized scheme for speeding up algorithms for linear and convex programming with high constraints-to-variable ratio. Math. Program. 61, 39–52 (1993)

    Article  MathSciNet  Google Scholar 

  2. Balcázar, J.L., Dai, Y., Watanabe, O.: A Random sampling technique for training support vector machines: for a primal-form maximal-margin classifiers. In: Proceedings of 12th International Conference on Algorithmic Learning Theory (ALT’01). Lecture Notes in Computer Science, vol. 2225, pp. 119–134. Springer, London (2001)

    Chapter  Google Scholar 

  3. Balcázar, J.L., Dai, Y., Watanabe, O.: Provably fast training algorithms for support vector machines. In: Proceedings of First IEEE International Conference on Data Mining (ICDM’01), pp. 43–50. IEEE, Los Alamitos (2001)

    Chapter  Google Scholar 

  4. Balcázar, J.L., Dai, Y., Watanabe, O.: Provably fast support vector regression using random sampling. In: Proceedings of SIAM Workshop in Discrete Mathematics and Data Mining, pp. 19–29. SIAM, Philadelphia (2002)

    Google Scholar 

  5. Bennett, K.P., Bredensteiner, E.J.: Duality and geometry in SVM classifiers. In: Proceedings of 17th International Conference on Machine Learning (ICML’2000), pp. 57–64. Morgan Kaufmann, San Mateo (2000)

    Google Scholar 

  6. Bennett, K.P., Campbell, C.: Support Vector Machines: Hype or Hallelujah? SIGKDD Explorations 2, 2 (2000)

    Article  Google Scholar 

  7. Bennett, K.P., Mangasarian, O.L.: Robust linear programming discrimination of two linearly inseparable sets. Optim. Methods Softw. 1, 23–34 (1992)

    Article  Google Scholar 

  8. Bertsekas, D.P.: Nonlinear Programming. Athena Scientific, Belmont (1995)

    MATH  Google Scholar 

  9. Bi, J., Bennett, K.P.: Duality, geometry, and support vector regression. In: Proceedings of Advances in Neural Information Processing Systems (NIPS’02), pp. 539–600. MIT Press, Cambridge (2002)

    Google Scholar 

  10. Boser, B.E., Guyon, I.M., Vapnik, V.N.: A training algorithm for optimal margin classifiers. In: Proceedings of 5th Annual Conference on Computational Learning Theory (COLT’92), pp. 144–152. ACM, New York (1992)

    Google Scholar 

  11. Bradley, P.S., Mangasarian, O.L., Musicant, D.R.: Optimization methods in massive datasets. In: Abello, J., Pardalos, P.M., Resende, M.G.C. (eds.) Handbook of Massive Datasets, pp. 439–471. Kluwer Academic, Dordrecht (2002)

    Google Scholar 

  12. Cauwenberghs, G., Poggio, T.: Incremental and decremental support vector machine learning. In: Proceedings of Advances in Neural Information Processing Systems (NIPS’00), pp. 409–415. MIT Press, Cambridge (2000)

    Google Scholar 

  13. Clarkson, K.L.: A Las Vegas algorithm for linear programming when the dimension is small. In: Proceedings of 29th IEEE Symposium on Foundations of Computer Science (FOCS’88), pp. 452–456. IEEE, Los Alamitos (1988). (See [15] for a better version.)

    Chapter  Google Scholar 

  14. Clarkson, K.L.: Las Vegas algorithms for linear and integer programming. J. ACM 42, 488–499 (1995)

    Article  MATH  MathSciNet  Google Scholar 

  15. Cortes, C., Vapnik, V.: Support-vector networks. Mach. Learn. 20, 273–297 (1995)

    MATH  Google Scholar 

  16. Cristianini, N., Shawe-Taylor, J.: An Introduction to Support Vector Machines. Cambridge University Press, Cambridge (2000)

    Google Scholar 

  17. Cucker, F., Smale, S.: On the mathematical foundations of learning. Bull. Am. Math. Soc. 39(1), 1–49 (2001)

    Article  MathSciNet  Google Scholar 

  18. Evgeniou, T., Pontil, M., Poggio, T.: A unified framework for regularization networks and support vector machines. AI Memo No. 1654, AI Lab MIT (1999)

  19. Fine, S., Scheinberg, K.: Efficient SVM training using low-rank kernel representations. J. Mach. Learn. Res. 2, 243–264 (2001)

    Article  Google Scholar 

  20. Gärtner, B., Welzl, E.: A simple sampling lemma: Analysis and applications in geometric optimization. Discrete Comput. Geom. 25(4), 569–590 (2001)

    MATH  MathSciNet  Google Scholar 

  21. Hush, D., Scovel, C.: Polynomial-time decomposition algorithms for support vector machines. Mach. Learn. 51, 51–71 (2003)

    Article  MATH  Google Scholar 

  22. Keerthi, S.S., Gilbert, E.G.: Convergence of a generalized SMO algorithm for SVM classifier design. Mach. Learn. 46(1–3), 351–360 (2002)

    Article  MATH  Google Scholar 

  23. Lee, Y., Mangasarian, O.L.: RSVM: Reduced Support Vector Machines. In: CD Proceedings of the SIAM International Conference on Data Mining, Chicago, 5–7 April 2001. SIAM, Philadelphia (2001). (Available from http://www.cs.wisc.edu/ olvi/olvi.html.)

    Google Scholar 

  24. Lin, C.J.: Linear convergence of a decomposition method for support vector machines. Technical Report. http://www.csie.ntu.edu.tw/~cjlin/papers/linearconv.pdf (2001)

  25. Lin, C.J.: On the convergence of the decomposition method for support vector machines. IEEE Trans. Neural Networks 14, 1267–1281 (2002)

    MATH  Google Scholar 

  26. Martín, M.: On-line support vector machine regression. In: Proceedings of 13th European Conference on Machine Learning (ECML’02), pp. 282–294 (2002).

  27. Osuna, E., Freund, R., Girosi, F.: An improved training algorithm for support vector machines. In: Proceedings of IEEE Workshop on Neural Networks for Signal Processing, pp. 276–285 (1997)

  28. Pavlov, D., Mao, J., Dom, B.: Scaling up support vector machines using boosting algorithm. In: Proceedings of International Conference Pattern Rec., pp. 2219–2222 (2000)

  29. Platt, J.: Fast training of support vector machines using sequential minimal optimization. In: Schölkopf, B., Burges, C.J.C., Smola, A.J. (eds.) Advances in Kernel Methods—Support Vector Learning, pp. 185–208. MIT Press, Cambridge (1999)

    Google Scholar 

  30. Schölkopf, B., Burges, C., Vapnik, V.: Extracting support data for a given task. In: Proceedings of First International Conference on Knowledge Discovery and Data Mining (KDD’95), pp. 252–257 (1995)

  31. Schölkopf, B., Smola, A., Williamson, R.C.: A new support vector regression algorithm. In: Proceedings of Advances in Neural Information Processing Systems (NIPS’99), pp. 330–336. MIT Press, Cambridge (1999)

    Google Scholar 

  32. Schölkopf, B., Smola, A., Williamson, R.C., Bartlett, P.L.: New support vector algorithms. Neural Comput. 12, 1207–1245 (2000)

    Article  Google Scholar 

  33. Smola, A.J., Schölkopf, B.: A tutorial on support vector regression. NeuroCOLT Technical Report NC-TR-98-030, Royal Holloway College, University of London (1998)

Download references

Author information

Authors and Affiliations

  1. Departament de Llenguatges i Sistemes Informàtics, Univ. Politècnica de Catalunya Campus Nord, Jordi Girona Salgado 1-3, 08034, Barcelona, Spain

    José L. Balcázar

  2. Department of Bioengineering (MC063), University Illinois at Chicago, 851 S. Morgan Str, Chicago, IL, 60607-7052, USA

    Yang Dai

  3. Department of Mathematical and Computing Sciences, Tokyo Institute of Technology, Meguro-ku Ookayama, Tokyo, 152-8552, Japan

    Junichi Tanaka & Osamu Watanabe

Authors
  1. José L. Balcázar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yang Dai
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Junichi Tanaka
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Osamu Watanabe
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Osamu Watanabe.

Additional information

The first and the fourth authors started this research while visiting the Centre de Recerca Matemàtica of the Institute of Catalan Studies in Barcelona.

The first author was supported by IST Programme of the EU under contract number IST-1999-14186 (ALCOM-FT), Spanish Government TIC2004-07925-C03-02, and CIRIT 2001SGR-00252.

The second author conducted this research while she was with Department of Mathematical and Computing Sciences, Tokyo Institue of Technology, and was supported by a Grant-in-Aid (C-13650444) from Japanese Goverment.

The fourth author was supported in part by a Grant-in-Aid for Scientific Research on Priority Areas “Discovery Science” 1998–2000 from Japanese Goverment.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Balcázar, J.L., Dai, Y., Tanaka, J. et al. Provably Fast Training Algorithms for Support Vector Machines. Theory Comput Syst 42, 568–595 (2008). https://doi.org/10.1007/s00224-007-9094-6

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00224-007-9094-6

Keywords

Search

Navigation