Skip to main content
SpringerLink
Log in

Explorations of the Semantic Learning Machine Neuroevolution Algorithm: Dynamic Training Data Use, Ensemble Construction Methods, and Deep Learning Perspectives

  • Chapter
  • First Online:
Genetic Programming Theory and Practice XVII

Part of the book series: Genetic and Evolutionary Computation ((GEVO))

  • 711 Accesses

Abstract

The recently proposed Semantic Learning Machine (SLM) neuroevolution algorithm is able to construct Neural Networks (NNs) over unimodal error landscapes in any supervised learning problem where the error is measured as a distance to the known targets. This chapter studies how different methods of dynamically using the training data affect the resulting generalization of the SLM algorithm. Across four real-world binary classification datasets, SLM is shown to outperform the Multi-layer Perceptron, with statistical significance, after parameter tuning is performed in both algorithms. Furthermore, this chapter also studies how different ensemble constructions methods influence the resulting generalization. The results show that the stochastic nature of SLM already confers enough diversity to the ensembles such that Bagging and Boosting cannot improve upon a simple averaging ensemble construction method. Finally, some initial results with SLM and Convolutional NNs are presented and future Deep Learning perspectives are discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 139.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 179.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 179.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Abiodun, O.I., Jantan, A., Omolara, A.E., Dada, K.V., Mohamed, N.A., Arshad, H.: State-of-the-art in artificial neural network applications: A survey. Heliyon 4(11), e00,938 (2018)

    Article  Google Scholar 

  2. Agostinelli, F., Hoffman, M., Sadowski, P., Baldi, P.: Learning activation functions to improve deep neural networks. arXiv preprint arXiv:1412.6830 (2014)

    Google Scholar 

  3. Alba, E., Aldana, J., Troya, J.M.: Full automatic ANN design: A genetic approach. In: International Workshop on Artificial Neural Networks, pp. 399–404. Springer (1993)

    Google Scholar 

  4. Angeline, P.J., Saunders, G.M., Pollack, J.B.: An evolutionary algorithm that constructs recurrent neural networks. IEEE Transactions on Neural Networks 5(1), 54–65 (1994)

    Article  Google Scholar 

  5. Bornholdt, S., Graudenz, D.: General asymmetric neural networks and structure design by genetic algorithms. Neural Networks 5(2), 327–334 (1992)

    Article  Google Scholar 

  6. Breiman, L.: Bagging predictors. Machine Learning 24(2), 123–140 (1996).

    MATH  Google Scholar 

  7. Chauvin, Y., Rumelhart, D.E.: Backpropagation: Theory, architectures, and applications. Psychology Press (2013)

    Google Scholar 

  8. Cun, Y.L., Denker, J.S., Solla, S.A.: Advances in neural information processing systems 2. chap. Optimal Brain Damage, pp. 598–605. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA (1990)

    Google Scholar 

  9. DasGupta, B., Schnitger, G.: Efficient approximation with neural networks: A comparison of gate functions. Pennsylvania State University, Department of Computer Science (1992)

    Google Scholar 

  10. Dill, F.A., Deer, B.C.: An exploration of genetic algorithms for the selection of connection weights in dynamical neural networks. In: Proceedings of the IEEE 1991 National Aerospace and Electronics Conference NAECON 1991, vol. 3, pp. 1111–1115 (1991)

    Google Scholar 

  11. Drucker, H.: Improving regressors using boosting techniques. In: Proceedings of the Fourteenth International Conference on Machine Learning, ICML ’97, pp. 107–115. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA (1997).

    Google Scholar 

  12. Epitropakis, M.G., Plagianakos, V.P., Vrahatis, M.N.: Evolutionary Algorithm Training of Higher-Order Neural Networks. IGI Global (2009)

    Google Scholar 

  13. Fahlman, S.E., Lebiere, C.: Advances in neural information processing systems 2. chap. The Cascade-correlation Learning Architecture, pp. 524–532. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA (1990)

    Google Scholar 

  14. Floreano, D., Dürr, P., Mattiussi, C.: Neuroevolution: From architectures to learning. Evolutionary Intelligence 1(1), 47–62 (2008)

    Article  Google Scholar 

  15. Frean, M.: The upstart algorithm: A method for constructing and training feedforward neural networks. Neural Computation 2(2), 198–209 (1990)

    Article  Google Scholar 

  16. Galván-López, E., Vázquez-Mendoza, L., Schoenauer, M., Trujillo, L.: On the use of dynamic GP fitness cases in static and dynamic optimisation problems. In: International Conference on Artificial Evolution (Evolution Artificielle), pp. 72–87. Springer (2017)

    Google Scholar 

  17. Garro, B.A., Vázquez, R.A.: Designing artificial neural networks using particle swarm optimization algorithms. Computational Intelligence and Neuroscience 2015, 61 (2015)

    Article  Google Scholar 

  18. Gonçalves, I., Silva, S.: Experiments on controlling overfitting in genetic programming. In: Local proceedings of the 15th Portuguese Conference on Artificial Intelligence, EPIA 2011 (2011)

    Google Scholar 

  19. Gonçalves, I., Silva, S., Melo, J.B., Carreiras, J.M.B.: Random sampling technique for overfitting control in genetic programming. In: Genetic Programming, pp. 218–229. Springer (2012)

    Google Scholar 

  20. Gonçalves, I., Silva, S.: Balancing learning and overfitting in genetic programming with interleaved sampling of training data. In: Genetic Programming, pp. 73–84. Springer (2013)

    Google Scholar 

  21. Gonçalves, I., Silva, S., Fonseca, C.M.: On the generalization ability of geometric semantic genetic programming. In: Genetic Programming, pp. 41–52. Springer (2015)

    Google Scholar 

  22. Gonçalves, I., Silva, S., Fonseca, C.M.: Semantic learning machine: A feedforward neural network construction algorithm inspired by geometric semantic genetic programming. In: Progress in Artificial Intelligence, Lecture Notes in Computer Science, vol. 9273, pp. 280–285. Springer (2015)

    Google Scholar 

  23. Gonçalves, I., Silva, S., Fonseca, C.M., Castelli, M.: Arbitrarily close alignments in the error space: A geometric semantic genetic programming approach. In: Proceedings of the 2016 on Genetic and Evolutionary Computation Conference Companion, pp. 99–100. ACM (2016)

    Google Scholar 

  24. Gonçalves, I.: An exploration of generalization and overfitting in genetic programming: Standard and geometric semantic approaches. Ph.D. thesis, Department of Informatics Engineering, University of Coimbra, Portugal (2017)

    Google Scholar 

  25. Gonçalves, I., Silva, S., Fonseca, C.M., Castelli, M.: Unsure when to stop? ask your semantic neighbors. In: Proceedings of the Genetic and Evolutionary Computation Conference, GECCO’17, pp. 929–936. ACM, New York, NY, USA (2017).

    Google Scholar 

  26. Greenwood, G.W.: Training partially recurrent neural networks using evolutionary strategies. IEEE Transactions on Speech and Audio Processing 5(2), 192–194 (1997)

    Article  Google Scholar 

  27. He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1026–1034 (2015)

    Google Scholar 

  28. Hornik, K., Stinchcombe, M., White, H.: Multilayer feedforward networks are universal approximators. Neural Networks 2(5), 359–366 (1989)

    Article  MATH  Google Scholar 

  29. Hornik, K.: Approximation capabilities of multilayer feedforward networks. Neural Networks 4(2), 251–257 (1991)

    Article  MathSciNet  Google Scholar 

  30. Hush, D.R., Horne, B.G.: Progress in supervised neural networks. IEEE Signal Processing Magazine 10(1), 8–39 (1993)

    Article  Google Scholar 

  31. Irie, B., Miyake, S.: Capabilities of three-layered perceptrons. In: IEEE International Conference on Neural Networks, vol. 1, p. 218 (1988)

    Google Scholar 

  32. Jagusch, J.B., Gonçalves, I., Castelli, M.: Neuroevolution under unimodal error landscapes: An exploration of the semantic learning machine algorithm. In: Proceedings of the Genetic and Evolutionary Computation Conference Companion, pp. 159–160. ACM (2018)

    Google Scholar 

  33. Jarrett, K., Kavukcuoglu, K., LeCun, Y., et al.: What is the best multi-stage architecture for object recognition? In: 2009 IEEE 12th International Conference on Computer Vision, pp. 2146–2153. IEEE (2009)

    Google Scholar 

  34. Jian, F., Yugeng, X.: Neural network design based on evolutionary programming. Artificial Intelligence in Engineering 11(2), 155–161 (1997)

    Article  Google Scholar 

  35. Kiefer, J., Wolfowitz, J.: Stochastic estimation of the maximum of a regression function. Ann. Math. Statist. 23(3), 462–466 (1952).

    Article  MathSciNet  MATH  Google Scholar 

  36. Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. CoRR abs/1412.6980 (2014).

    Google Scholar 

  37. Kiranyaz, S., Ince, T., Yildirim, A., Gabbouj, M.: Evolutionary artificial neural networks by multi-dimensional particle swarm optimization. Neural Networks 22(10), 1448–1462 (2009)

    Article  Google Scholar 

  38. Konda, K., Memisevic, R., Krueger, D.: Zero-bias autoencoders and the benefits of co-adapting features. arXiv preprint arXiv:1402.3337 (2014)

    Google Scholar 

  39. Koza, J.R., Rice, J.P.: Genetic generation of both the weights and architecture for a neural network. In: IJCNN-91-Seattle International Joint Conference on Neural Networks, vol. 2, pp. 397–404. IEEE (1991)

    Google Scholar 

  40. Lapa, P., Gonçalves, I., Rundo, L., Castelli, M.: Enhancing classification performance of convolutional neural networks for prostate cancer detection on magnetic resonance images: A study with the semantic learning machine. In: Proceedings of the Genetic and Evolutionary Computation Conference Companion, GECCO ’19. ACM, New York, NY, USA (2019).

    Google Scholar 

  41. Lapa, P., Gonçalves, I., Rundo, L., Castelli, M.: Semantic learning machine improves the cnn-based detection of prostate cancer in non-contrast-enhanced mri. In: Proceedings of the Genetic and Evolutionary Computation Conference Companion, GECCO ’19. ACM, New York, NY, USA (2019).

    Google Scholar 

  42. Lei, J., He, G., Jiang, J.P.: The state estimation of the cstr system based on a recurrent neural network trained by HGAs. In: Proceedings of International Conference on Neural Networks (ICNN’97), vol. 2, pp. 779–782 (1997)

    Google Scholar 

  43. Lichman, M.: UCI Machine Learning Repository (2013).

    Google Scholar 

  44. Litjens, G., Debats, O., Barentsz, J., Karssemeijer, N., Huisman, H.: “PROSTATEx Challenge data”, The Cancer Imaging Archive. https://wiki.cancerimagingarchive.net/display/Public/SPIE-AAPM-NCI+PROSTATEx+Challenges (2017). Online; Accessed on January 25, 2019

  45. Liu, S., Zheng, H., Feng, Y., Li, W.: Prostate cancer diagnosis using deep learning with 3D multiparametric MRI. In: Medical Imaging 2017: Computer-Aided Diagnosis, Proceedings SPIE, vol. 10134, p. 1013428. International Society for Optics and Photonics (2017).

    Google Scholar 

  46. Maas, A.L., Hannun, A.Y., Ng, A.Y.: Rectifier nonlinearities improve neural network acoustic models. In: Proc. ICML, vol. 30, p. 3 (2013)

    Google Scholar 

  47. Manessi, F., Rozza, A.: Learning combinations of activation functions. In: 2018 24th International Conference on Pattern Recognition (ICPR), pp. 61–66. IEEE (2018)

    Google Scholar 

  48. Mani, G.: Learning by gradient descent in function space. In: 1990 IEEE International Conference on Systems, Man, and Cybernetics Conference Proceedings, pp. 242–247 (1990)

    Google Scholar 

  49. Martinez, Y., Trujillo, L., Naredo, E., Legrand, P.: A comparison of fitness-case sampling methods for symbolic regression with genetic programming. In: EVOLVE-A Bridge between Probability, Set Oriented Numerics, and Evolutionary Computation V, pp. 201–212. Springer (2014)

    Google Scholar 

  50. Martínez, Y., Naredo, E., Trujillo, L., Legrand, P., López, U.: A comparison of fitness-case sampling methods for genetic programming. Journal of Experimental & Theoretical Artificial Intelligence 29(6), 1203–1224 (2017)

    Article  Google Scholar 

  51. Miikkulainen, R., Liang, J., Meyerson, E., Rawal, A., Fink, D., Francon, O., Raju, B., Shahrzad, H., Navruzyan, A., Duffy, N., Hodjat, B.: Evolving deep neural networks. In: R. Kozma, C. Alippi, Y. Choe, F.C. Morabito (eds.) Artificial Intelligence in the Age of Neural Networks and Brain Computing. Amsterdam: Elsevier (2018).

    Google Scholar 

  52. Miller, G.F., Todd, P.M., Hegde, S.U.: Designing neural networks using genetic algorithms. In: Proceedings of the Third International Conference on Genetic Algorithms, pp. 379–384. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA (1989)

    Google Scholar 

  53. Montana, D.J., Davis, L.: Training feedforward neural networks using genetic algorithms. In: IJCAI, vol. 89, pp. 762–767 (1989)

    MATH  Google Scholar 

  54. Moraglio, A., Krawiec, K., Johnson, C.G.: Geometric semantic genetic programming. In: Parallel Problem Solving from Nature-PPSN XII, pp. 21–31. Springer (2012)

    Google Scholar 

  55. Moraglio, A., Mambrini, A.: Runtime analysis of mutation-based geometric semantic genetic programming for basis functions regression. In: Proceedings of the 15th annual conference on Genetic and Evolutionary Computation, pp. 989–996. ACM (2013)

    Google Scholar 

  56. Mozer, M.C., Smolensky, P.: Advances in neural information processing systems 1. chap. Skeletonization: A Technique for Trimming the Fat from a Network via Relevance Assessment, pp. 107–115. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA (1989)

    Google Scholar 

  57. Nikolopoulos, C., Fellrath, P.: A hybrid expert system for investment advising. Expert Systems 11(4), 245–250 (1994)

    Article  Google Scholar 

  58. Oliker, S., Furst, M., Maimon, O.: Design architectures and training of neural networks with a distributed genetic algorithm. In: IEEE International Conference on Neural Networks, pp. 199–202. IEEE (1993)

    Google Scholar 

  59. Robbins, H., Monro, S.: A stochastic approximation method. Ann. Math. Statist. 22(3), 400–407 (1951).

    Article  MathSciNet  MATH  Google Scholar 

  60. Rumelhart, D.E., Hinton, G.E., Williams, R.J.: Learning representations by back-propagating errors. Nature 323(6088), 533 (1986)

    Article  MATH  Google Scholar 

  61. S. Ding H. Li, C.S.J.Y.F.J.: Evolutionary artificial neural networks: A review. Artificial Intelligence Review 39, 251–260 (2013).

    Google Scholar 

  62. Samarasinghe, S.: Neural networks for applied sciences and engineering: From fundamentals to complex pattern recognition. Auerbach Publications (2016)

    Google Scholar 

  63. Schaffer, J.D., Caruana, R.A., Eshelman, L.J.: Using genetic search to exploit the emergent behavior of neural networks. Physica D: Nonlinear Phenomena 42(1–3), 244–248 (1990)

    Article  Google Scholar 

  64. Schiffmann, W., Joost, M., Werner, R.: Synthesis and performance analysis of multilayer neural network architectures (1992)

    Google Scholar 

  65. Schoenauer, M., Ronald, E.: Genetic extensions of neural net learning: Transfer functions and renormalisation coefficients

    Google Scholar 

  66. Sietsma, J., Dow, R.J.: Creating artificial neural networks that generalize. Neural Networks 4(1), 67–79 (1991)

    Article  Google Scholar 

  67. Silva, S., Ingalalli, V., Vinga, S., Carreiras, J.M., Melo, J.B., Castelli, M., Vanneschi, L., Gonçalves, I., Caldas, J.: Prediction of forest aboveground biomass: An exercise on avoiding overfitting. In: European Conference on the Applications of Evolutionary Computation, pp. 407–417. Springer (2013)

    Google Scholar 

  68. Srinivas, M., Patnaik, L.M.: Learning neural network weights using genetic algorithms-improving performance by search-space reduction. In: [Proceedings] 1991 IEEE International Joint Conference on Neural Networks, vol. 3, pp. 2331–2336 (1991)

    Google Scholar 

  69. Stanley, K.O., Miikkulainen, R.: Evolving neural networks through augmenting topologies. Evolutionary Computation 10(2), 99–127 (2002)

    Article  Google Scholar 

  70. Stanley, K.O., Clune, J., Lehman, J., Miikkulainen, R.: Designing neural networks through neuroevolution. Nature Machine Intelligence 1(1), 24–35 (2019)

    Article  Google Scholar 

  71. Sutton, R.S.: Two problems with backpropagation and other steepest-descent learning procedures for networks. In: Proceedings of the Eighth Annual Conference of the Cognitive Science Society. Hillsdale, NJ: Erlbaum (1986)

    Google Scholar 

  72. White, D., Ligomenides, P.: Gannet: A genetic algorithm for optimizing topology and weights in neural network design. In: International Workshop on Artificial Neural Networks, pp. 322–327. Springer (1993)

    Google Scholar 

  73. Whitley, D., Starkweather, T., Bogart, C.: Genetic algorithms and neural networks: Optimizing connections and connectivity. Parallel Computing 14(3), 347–361 (1990)

    Article  Google Scholar 

  74. Widrow, B., Lehr, M.A.: 30 years of adaptive neural networks: Perceptron, madaline, and backpropagation. Proceedings of the IEEE 78(9), 1415–1442 (1990)

    Article  Google Scholar 

  75. Wilson, S.W.: Perception redux: Emergence of structure. Physica D: Nonlinear Phenomena 42(1–3), 249–256 (1990)

    Article  Google Scholar 

  76. Yao, X.: Evolving artificial neural networks. Proceedings of the IEEE 87(9), 1423–1447 (1999)

    Article  Google Scholar 

  77. Yao, X., Liu, Y.: A new evolutionary system for evolving artificial neural networks. IEEE Transactions on Neural Networks 8(3), 694–713 (1997)

    Article  Google Scholar 

  78. Zhang, C., Shao, H., Li, Y.: Particle swarm optimisation for evolving artificial neural network. In: Systems, Man, and Cybernetics, 2000 IEEE International Conference on, vol. 4, pp. 2487–2490. IEEE (2000)

    Google Scholar 

  79. Zhang, G., Patuwo, B.E., Hu, M.Y.: Forecasting with artificial neural networks:: The state of the art. International Journal of Forecasting 14(1), 35–62 (1998)

    Article  Google Scholar 

Download references

Acknowledgements

This work was partially supported by projects UID/MULTI/00308/2019 and by the European Regional Development Fund through the COMPETE 2020 Programme, FCT—Portuguese Foundation for Science and Technology and Regional Operational Program of the Center Region (CENTRO2020) within project MAnAGER (POCI-01-0145-FEDER-028040). This work was also partially supported by national funds through FCT (Fundação para a Ciência e a Tecnologia) under project DSAIPA/DS/0022/2018 (GADgET).

Author information

Authors and Affiliations

  1. INESC Coimbra, DEEC, University of Coimbra, Coimbra, Portugal

    Ivo Gonçalves

  2. NOVA IMS, Universidade Nova de Lisboa, Lisboa, Portugal

    Marta Seca & Mauro Castelli

Authors
  1. Ivo Gonçalves
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Marta Seca
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mauro Castelli
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Computer Science and Engineering, John R. Koza Chair, Michigan State University, East Lansing, MI, USA

    Wolfgang Banzhaf

  2. BEACON Center, Michigan State University, East Lansing, MI, USA

    Erik Goodman

  3. Department of Computer Science and Engineering, Michigan State University, Okemos, MI, USA

    Leigh Sheneman

  4. Depto Ingenieria en Electronic Electrica Tecnológico Nacional de México/ IT, Tijuana, Baja California, Mexico

    Leonardo Trujillo

  5. Evolution Enterprises, Ann Arbor, MI, USA

    Bill Worzel

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Gonçalves, I., Seca, M., Castelli, M. (2020). Explorations of the Semantic Learning Machine Neuroevolution Algorithm: Dynamic Training Data Use, Ensemble Construction Methods, and Deep Learning Perspectives. In: Banzhaf, W., Goodman, E., Sheneman, L., Trujillo, L., Worzel, B. (eds) Genetic Programming Theory and Practice XVII. Genetic and Evolutionary Computation. Springer, Cham. https://doi.org/10.1007/978-3-030-39958-0_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-39958-0_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-39957-3

  • Online ISBN: 978-3-030-39958-0

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation