Skip to main content
SpringerLink
Log in

Meta-heuristics, Machine Learning, and Deep Learning Methods

  • Chapter
  • First Online:
Evolutionary Approach to Machine Learning and Deep Neural Networks

Abstract

This chapter introduces several meta-heuristics and learning methods, which will be employed in later chapters. These methods will be employed to extend evolutionary computation frameworks in later chapters. Readers familiar with these methods may skip this chapter.

I remember the first time I met Edsger Dijkstra. ... He asked me what I was working on. Perhaps just to provoke a memorable exchange I said, “AI.” To that he immediately responded, “Why don’t you work on I?”

(Leslie Valiant, Probably Approximately Correct: Nature’s Algorithms for Learning and Prospering in a Complex, Basic Books 2014)

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 129.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 169.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 169.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Given a set of vectors \(\mathbf {v}_1, \ldots , \mathbf {v}_n\), its Gram matrix G is an \(n\times n\) matrix, whose element is an inner product of two vectors, i.e., \(G_{ij} = \mathbf {v}_i \cdot \mathbf {v}_j\).

  2. 2.

    The point P is in the range which is outside a circle (the center is a mid-point of centers of circle \(O_{1}\) and circle \(O_{2}\), and the radius is \(\frac{\mid r_{1} - r_{2}\mid }{2}\)), and which is inside a circle (the center is the same, and the radius is\(\frac{(r_{1} +r_{2})}{2}\)).

  3. 3.

    A degree measure of the similarity between two documents A and B, i.e., \(\frac{\mid A \cap B\mid }{\mid A \cup B \mid }\).

  4. 4.

    The used parameters are as follows: 300 sample points, cluster centers \(=\) [[2, 2], [−2, −2], [2, −2]], and cluster std. \(=\) 0.5.

  5. 5.

    The first, second, and fifth of these layers are linked to max pooling.

  6. 6.

    The softmax activation function is defined as \(f_i(x_1,\ldots ,x_d) = \frac{\exp (x_i)}{\sum _j \exp (x_j)}\;(i=1,\ldots ,d)\), where d is the unit number of the layer.

  7. 7.

    Principal component analysis of the RGB values in the data set is performed and then random changes are made, centered on these axes, thereby preserving the essential features of the original natural images.

  8. 8.

    ILSVRC2012 dataset is a subset of the ImageNet database [40]. It contains 1,000 object categories. The training set, validation set, and testing set contain 1.3 M, 50 K, and 150 K images, respectively. The input images are of \(224\times 224\times 3\) pixels.

References

  1. Aha, D.W., Kibler, D., Albert, M.: Instance-based learning algorithms. Mach. Learn. 6, 37–66 (1991)

    Google Scholar 

  2. Angeline, P.J.: Evolutionary optimization versus particle swarm optimization: philosophy and performance differences. In: Porto, V.W., Saravanan, N., Waagen, D., Eiben, A.E. (eds.) Evolutionary Programming VII, pp. 601–610. Springer, Berlin (1998)

    Google Scholar 

  3. Bäck, T., Fogel, D.B., Michalewicz, Z. (eds.): Evolutionary Computation 1: Basic Algorithms and Operators. Institute of Physics Publishing, Bristol (2000)

    Google Scholar 

  4. Bishop, C.M.: Pattern Recognition and Machine Learning. Springer, Berlin (2006)

    Google Scholar 

  5. Buchberger, B.: A criterion for detecting unnecessary reductions in the construction of Gröbner-bases. In: Proceedings of the International Symposium on Symbolic and Algebraic Computation (EUROSAM’79), pp. 3–21 (1979)

    Chapter  Google Scholar 

  6. Buchberger, B.: A note on the complexity of constructing Gröbner-bases. In: van Hulzen, J.A. (ed.) Computer Algebra, EUROCAL 1983. Lecture Notes in Computer Science, vol. 162. Springer, Berlin (1983)

    Google Scholar 

  7. Buchberger, B.: A critical pair completion algorithm for finitely generated ideals in rings. In: Borger, E., Hasenjaeger, G., Rodding, D. (eds.) Logic and Machines: Decision Problems and Complexity. Lecture Notes in Computer Science, vol. 171. Springer, Berlin (1984)

    Chapter  Google Scholar 

  8. Buchberger, B.: Gröbner bases: an algorithmic method in polynomial ideal theory. In: Bose, N.K. (ed.) Multidimensional Systems Theory. D. Reidel Publishing Company, Dordrecht (1985)

    Chapter  Google Scholar 

  9. Buchberger, B.: Applications of Gröbner bases in non-linear computational geometry. In: Jansen, R. (ed.) Trends in Computer Algebra. Lecture Notes in Computer Science, vol. 296. Springer, Berlin (1987)

    Google Scholar 

  10. Buchberger, B., Winkler, F.: Gröbner Bases and Applications, vol. 251. Cambridge University Press, Cambridge (1998)

    Google Scholar 

  11. Caviness, B.F., Johnson, J.R. (eds.): Quantifier Elimination and Cylindrical Algebraic Decomposition. Springer, Berlin (2013)

    Google Scholar 

  12. Chakraborti, N., Misra, K., Bhatt, P., Barman, N., Prasad, R.: Tight-binding calculations of Si-H clusters using genetic algorithms and related techniques: studies using differential evolution. J. Phase Equilibria 22(5), 525–530 (2001)

    Article  Google Scholar 

  13. Chang, C.-C., Lin, C.-J.: LIBSVM: a library for support vector machines. ACM Trans. Intell. Syst. Technol. 2(3), 1–27 (2011)

    Article  Google Scholar 

  14. Chou, S., Schelter, W.F.: Proving geometry theorems with rewrite rules. J. Autom. Reason. 2, 253–273 (1986)

    Article  MATH  Google Scholar 

  15. Cortes, C., Vapnik, V.: Support-vector networks. Mach. Learn. 20(3), 273–297 (1995)

    MATH  Google Scholar 

  16. Dasarathy, B.: Nearest Neighbor (NN) Norms: NN Pattern Classification Techniques. IEEE Computer Society Press, Los Alamitos (1991)

    Google Scholar 

  17. Drucker, H.: Improving Regression Using Boosting Techniques. In: Proceedings of International Conference on Machine Learning (ICML97) (1997)

    Google Scholar 

  18. Dueck, D., Frey, B.J.: Non-metric affinity propagation for unsupervised image categorization. In: 2007 IEEE 11th International Conference on Computer Vision, pp. 1–8. IEEE (2007)

    Google Scholar 

  19. Eberhart, R.C., Shi, Y.: Comparison between genetic algorithms and particle swarm optimization. In: Proceedings of the Seventh Annual Conference on Evolutionary Programming, pp. 611–619 (1998)

    Google Scholar 

  20. Fan, R.-E., Chang, L.-W., Hsieh, C.-J., Wang, X.-R., Lin, C.-J.: LIBLINEAR: a library for large linear classification. J. Mach. Learn. Res. 9, 1871–1874 (2008)

    MATH  Google Scholar 

  21. Felzenszwalb, P.F., Huttenlocher, D.P.: Efficient belief propagation for early vision. Int. J. Comput. Vis. 70, 41–54 (2006)

    Article  Google Scholar 

  22. Freund, Y., Schapire, R.E.: Experiments with a new boosting algorithm. In: Proceedings of International Conference on Machine Learning (ICML96) (1996)

    Google Scholar 

  23. Frey, B.J., Dueck, D.: Clustering by passing messages between data points. Science 315, 972–976 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  24. Fukushima, K.: Neocognitron: a self organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biol. Cybern. 36(4), 193–202 (1980)

    Article  MATH  Google Scholar 

  25. Gämperle, R., Müller, S.D., Koumoutsakos, P.: A parameter study for differential evolution. In: Proceedings of International Conference on Advances in Intelligent Systems, Fuzzy Systems, Evolutionary Computation, pp. 293–298 (2002)

    Google Scholar 

  26. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Bengio, Y.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)

    Google Scholar 

  27. Higashi, N., Iba, H.: Particle swarm optimization with Gaussian mutation. In: Proceedings of IEEE Swarm Intelligence Symposium (SIS03), pp. 72–79 (2003)

    Google Scholar 

  28. Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science 313, 504–507 (2006)

    Article  MathSciNet  MATH  Google Scholar 

  29. Holland, J.H.: Adaptation in Natural and Artificial Systems. University of Michigan Press (1975)

    Google Scholar 

  30. Hu, R., Namee, B.M., Delany, S.J.: Off to a good start: using clustering to select the initial training set in active learning. In: Proceedings of the Florida Artificial Intelligence Research Society Conference (FLAIRS) (2010)

    Google Scholar 

  31. Huang, C.-H., Wang, C.-J.: A GA-based feature selection and parameters optimization for support vector machines. Expert Syst. Appl. 31, 231–240 (2006)

    Article  Google Scholar 

  32. Iba, H., Hirochika Inoue, H.: Reasoning of geometric concepts based on algebraic constraint-directed method. In: Proceedings of the IJCAI, pp. 143–151 (1991)

    Google Scholar 

  33. Iba, H., Noman, N.: New Frontiers in Evolutionary Algorithms: Theory and Applications. World Scientific Publishing Company, Singapore (2011)

    Google Scholar 

  34. Jia, Y., Wang, J., Zhang, C., Hua, X.-S.: Finding image exemplars using fast sparse affinity propagation. In: Proceedings of the 16th ACM International Conference on Multimedia, pp. 639–642. ACM (2008)

    Google Scholar 

  35. Kapur, D.: Geometric reasoning and artificial intelligence: introduction to the special volume. Artif. Intell. 37 (1988)

    Article  Google Scholar 

  36. Kennedy, J., Eberhart, R.C.: Particle swarm optimization. In: Proceedings of IEEE the International Conference on Neural Networks, pp. 1942–1948 (1995)

    Google Scholar 

  37. Kennedy, J., Eberhart, R.C.: Swarm Intelligence. Morgan Kaufmann Publishers, San Francisco (2001)

    Google Scholar 

  38. Kennedy, J., Spears, W.M.: Matching algorithms to problems: an experimental test of the particle swarm and some genetic algorithms on the multimodal problem generator. In: Proceedings of the IEEE Congress on Evolutionary Computation (CEC), pp. 78–83 (1998)

    Google Scholar 

  39. Krink, T., Filipič, B., Fogel, G., Thomsen, R.: Noisy optimization problems – a particular challenge for differential evolution? In: Proceedings of Congress on Evolutionary Computation, pp. 332–339 (2004)

    Google Scholar 

  40. Krizhevsky, A., Sutskerver, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems 25 (NIPS), pp. 1097–1105 (2012)

    Google Scholar 

  41. Larsen, A.B.L., S\(\phi \)nderby, S.K., Winther, O.: Autoencoding beyond pixels using a learned similarity metric (2015). arXiv:1512.09300

  42. Laubenbacher, R., Stigler, B.: A computational algebra approach to the reverse engineering of gene regulatory networks. J. Theor. Biol. 229, 523–537 (2004)

    Article  MathSciNet  Google Scholar 

  43. Le, Q., Ranzato, M., Monga, R., Devin, M., Chen, K., Corrado, G., Dean, J., Ng, A.: Building high-level features using large scale unsupervised learning. In: Proceedings of the 29th International Conference on Machine Learning (2012)

    Google Scholar 

  44. LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)

    Article  Google Scholar 

  45. Limbeck, J.: Computation of approximate border bases and applications, Ph.D. thesis, Passau, Universität Passau, Dissertation (2014)

    Google Scholar 

  46. Liu, Z., Li, P., Zheng, Y., Sun, M.: Clustering to find exemplar terms for key phrase extraction. In: Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, vol. 1, pp. 257–266. Association for Computational Linguistics (2009)

    Google Scholar 

  47. Loos, R.: Introduction. In: Buchberger, B., et al. (eds.) Computer Algebra Symbolic and Algebraic Computation. Springer, Berlin (1982)

    Google Scholar 

  48. Maclin, R., Opitz, D.: An empirical evaluation of bagging and boosting. In: Proceedings of National Conference on Artificial Intelligence (AAAI97) (1997)

    Google Scholar 

  49. Möller, H.M., Buchberger, B.: The construction of multivariate polynomials with preassigned zeros. In: Proceedings of the European Computer Algebra Conference on Computer Algebra, pp. 24–31 (1982)

    Google Scholar 

  50. Mourrain, B.: A new criterion for normal form algorithms. Applied Algebra, Algebraic Algorithms and Error-correcting Codes, pp. 430–442. Springer, Berlin (1999)

    Chapter  MATH  Google Scholar 

  51. Nguyen, A., Yosinski, J., Clune, J.: Deep neural networks are easily fooled: high confidence predictions for unrecognizable images. In: Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 427–436 (2015)

    Google Scholar 

  52. Noman, N., Iba, H.: Enhancing differential evolution performance with local search for high dimensional function optimization. In: Proceedings of Genetic and Evolutionary Computation Conference (GECCO2005), pp. 967–974 (2005)

    Google Scholar 

  53. Noman, N., Iba, H.: A new generation alternation model for differential evolution. In: Proceedings of Genetic and Evolutionary Computation Conference (GECCO 2006), pp. 1265–1272 (2006)

    Google Scholar 

  54. Noman, N., Iba, H.: Differential evolution for economic load dispatch problems. Elsevier Electric Power Syst. Res. 78(8), 1322–1331 (2008)

    Article  Google Scholar 

  55. Noman, N., Iba, H.: Accelerating differential evolution using an adaptive local search. IEEE Trans. Evol. Comput. 12(1), 107–125 (2008)

    Article  Google Scholar 

  56. Pan, S.J., Yang, Q.: A survey on transfer learning. IEEE Trans. Knowl. Data Eng. 22(10), 1345–1359 (2010)

    Article  Google Scholar 

  57. Paul, T.K., Ueno, K., Iwata, K., Hayashi, T., Honda, N.: Genetic algorithm based methods for identification of health risk factors aimed at preventing metabolic syndrome. In: Proceedings of the 7th International Conference on Simulated Evolution And Learning (SEAL’08). LNCS, vol. 5361, pp. 210–219. Springer, Berlin (2008)

    Google Scholar 

  58. Pearl, J.: Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufmann, San Mateo (1988)

    Google Scholar 

  59. Price, K.V., Storn, R.M., Lampinen, J.A.: Differential Evolution: A Practical Approach to Global Optimization. Springer, Berlin (2005)

    MATH  Google Scholar 

  60. Quinlan, J.R.: Bagging, Boosting, and C4.5. In: Proceedings of National Conference on Artificial Intelligence (AAAI96) (1996)

    Google Scholar 

  61. Radford, A., Metz, L., Chintala, S.: Unsupervised representation learning with deep convolutional generative adversarial networks (2015). arXiv:1511.06434

  62. Sato, H., Hasegawa, Y., Bollegala, D., Iba, H.: Improved sampling using loopy belief propagation for probabilistic model building genetic programming. Swarm Evol. Comput. 23, 1–10 (2015)

    Article  Google Scholar 

  63. Stifter, S.: Algebraic methods for computing inverse kinematics. J. Intell. Robot. Syst. 11(1–2), 79–89 (1994)

    Article  MATH  Google Scholar 

  64. Storn, R.: System design by constraint adaptation and differential evolution. IEEE Trans. Evol. Comput. 3(1), 22–34 (1999)

    Article  Google Scholar 

  65. Storn, R., Price, K.V.: Differential evolution – a simple and efficient adaptive scheme for global optimization over continuous spaces. Technical report TR-95-012, ICSI (1995)

    Google Scholar 

  66. Storn, R., Price, K.V.: Differential evolution -a simple and efficient heuristic for global optimization over continuous spaces. J. Global Optim. 11(4), 341–359 (1997)

    Article  MathSciNet  MATH  Google Scholar 

  67. Swan, J., Neumann, G.K., Krawiec, K.: Analysis of semantic building blocks via Grobner bases. In: Johnson, C., Krawiec, K., O’Neill, M., Moraglio, A. (eds.) Semantic Methods in Genetic Programming (SMGP) at Parallel Problem Solving from Nature (PPSN XIV), Ljubljana, Slovenia (2014)

    Google Scholar 

  68. Tipping, M.E.: The relevance vector machine. Advances in Neural Information Processing Systems, pp. 652–658. MIT Press, Cambridge (2000)

    Google Scholar 

  69. Vapnik, V.: Statistical Learning Theory. Wiley, New York (1998)

    Google Scholar 

  70. Yang, Y., Iba, H.: Fooling voice based on evolutionary computation. In: Proceedings of Evolutionary Computation Symposium, Dec. 9–10, Hokkaido, Japan (2017)

    Google Scholar 

  71. Zaharie, D.: Critical values for the control parameters of differential evolution algorithms. In: Proceedings of MENDEL 2002, 8th International Conference on Soft Computing, pp. 62–67 (2002)

    Google Scholar 

  72. Zha, Z.-J., Yang, L., Mei, T., Wang, M., Wang, Z.: Visual query suggestion. In: Proceedings of the 17th ACM International Conference on Multimedia, pp. 15–24. ACM (2009)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. The University of Tokyo, Tokyo, Japan

    Hitoshi Iba

Authors
  1. Hitoshi Iba
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hitoshi Iba .

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer Nature Singapore Pte Ltd.

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Iba, H. (2018). Meta-heuristics, Machine Learning, and Deep Learning Methods. In: Evolutionary Approach to Machine Learning and Deep Neural Networks. Springer, Singapore. https://doi.org/10.1007/978-981-13-0200-8_2

Download citation

  • DOI: https://doi.org/10.1007/978-981-13-0200-8_2

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-13-0199-5

  • Online ISBN: 978-981-13-0200-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation