Abstract
Applications of Artificial Neural Networks are used with success in many fields such as e.g. medicine, economy, science and entertainment. But most of those models processes all input signals without attention directing mechanisms. In this paper contextual neural networks are considered which are using multi-step conditional aggregation functions to direct attention of neurons during data processing. It is verified if aggregation function with constant field of attention (CFA) can help to build classification models of higher accuracy and of lower activity of hidden connections than aggregation function with dynamic field of attention (SIF). Experiments are performed with use of the H2O machine learning framework implementing Generalized Backpropagation Algorithm for selected benchmark problems from UCI ML repository and real-life ALL-AML leukemia gene expression data. Presented analysis of results gives important clues on the most promising directions of research on contextual neural networks and indicates possible improvements of the H2O framework.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Tsai, Y.-C., et al.: FineNet: a joint convolutional and recurrent neural network model to forecast and recommend anomalous financial items. In: Proceedings of the 13th ACM Conference on Recommender Systems RecSys 2019, pp. 536–537. ACM, New York (2019)
Chen, S., Zhang, S., Shang, J., Chen, B., Zheng, N.: Brain-inspired cognitive model with attention for self-driving cars. IEEE Trans. Cogn. Dev. Syst. 11(1), 13–25 (2019)
Suleymanova, I., et al.: A deep convolutional neural network approach for astrocyte detection. Sci. Rep. 8(12878), 1–7 (2018)
Guest, D., Cranmer, K., Whiteson, D.: Deep learning and its application to LHC physics. Annu. Rev. Nucl. Part. Sci. 68, 1–22 (2018)
Liu, L., et al.: Automatic skin binding for production characters with deep graph networks. ACM Trans. Graph. (SIGGRAPH) 38(4), 1–12 (2019). Article 114
Gao, D., Li, X., Dong, Y., Peers, P., Xu, K., Tong, X.: Deep inverse rendering for high-resolution SVBRDF estimation from an arbitrary number of images. ACM Trans. Graph. (SIGGRAPH) 38(4), 1–15 (2019). Article 134
Gong, K., et al.: Iterative PET image reconstruction using convolutional neural network representation. IEEE Trans. Med. Imaging 38(3), 675–685 (2019)
Higgins, I., et al.: Beta-VAE: learning basic visual concepts with a constrained variational framework. In: International Conference on Learning Representations, ICLR 2017, vol. 2, no. 5, pp. 1–22 (2017)
Karras, T., Aila, T., Laine, S., Lehtinen, J.: Progressive growing of GANs for improved quality, stability, and variation. In: International Conference on Learning Representations, ICLR 2018, pp. 1–26 (2018)
Huang, X., Tan, H., Lin, G., Tian, Y.: A LSTM-based bidirectional translation model for optimizing rare words and terminologies. In: 2018 IEEE International Conference on Artificial Intelligence and Big Data (ICAIBD), pp. 5077–5086. IEEE (2018)
Athiwaratkun, B., Stokes, J.W.: Malware classification with LSTM and GRU language models and a character-level CNN. In: Proceedings 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2482–2486. IEEE (2017)
Amato, F., Mazzocca, N., Moscato, F., Vivenzio, E.: Multilayer perceptron: an intelligent model for classification and intrusion detection. In: 31st International Conference on Advanced Information Networking and Applications Workshops (WAINA), pp. 686–691. IEEE, Taipei (2017)
Dozono, H., Niina, G., Araki, S.: Convolutional self organizing map. In: 2016 IEEE International Conference on Computational Science and Computational Intelligence (CSCI), pp. 767–771. IEEE (2016)
Mel, B.W.: The Clusteron: toward a simple abstraction for a complex neuron. In: Advances in Neural Information Processing Systems, vol. 4, pp. 35–42. Morgan Kaufmann (1992)
Spratling, M.W., Hayes, G.: Learning synaptic clusters for nonlinear dendritic processing. Neural Process. Lett. 11, 17–27 (2000)
Gupta, M.: Correlative type higher-order neural units with applications. In: IEEE International Conference on Automation and Logistics, ICAL 2008, pp. 715–718 (2008). Springer Computer Science
Vanrullen, R., Koch, C.: Visual selective behavior can be triggered by a feed-forward process. J. Cogn. Neurosci. 15, 209–217 (2003)
Huk, M.: Learning distributed selective attention strategies with the Sigma-if neural network. In: Akbar, M., Hussain, D. (eds.) Advances in Computer Science and IT, pp. 209–232. InTech, Vukovar (2009)
Huk, M.: Notes on the generalized backpropagation algorithm for contextual neural networks with conditional aggregation functions. J. Intell. Fuzzy Syst. 32, 1365–1376 (2017)
Huk, M.: Backpropagation generalized delta rule for the selective attention sigma-if artificial neural network. Int. J. App. Math. Comp. Sci. 22, 449–459 (2012)
Huk, M., Pietraszko, J.: Contextual neural network based spectrum prediction for cognitive radio. In: 4th International Conference on Future Generation Communication Technology (FGCT 2015), pp. 1–5. IEEE Computer Society, London (2015)
Huk, M.: Non-uniform initialization of inputs groupings in contextual neural networks. In: Nguyen, N.T., Gaol, F.L., Hong, T.-P., Trawiński, B. (eds.) ACIIDS 2019. LNCS (LNAI), vol. 11432, pp. 420–428. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-14802-7_36
Huk, M.: Training contextual neural networks with rectifier activation functions: role and adoption of sorting methods. J. Intell. Fuzzy Syst. 38, 1–10 (2019)
Huk, M.: Weights ordering during training of contextual neural networks with generalized error backpropagation: importance and selection of sorting algorithms. In: Nguyen, N.T., Hoang, D.H., Hong, T.-P., Pham, H., Trawiński, B. (eds.) ACIIDS 2018. LNCS (LNAI), vol. 10752, pp. 200–211. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-75420-8_19
Szczepanik, M., Jóźwiak, I.: Fingerprint recognition based on minutes groups using directing attention algorithms. In: Rutkowski, L., Korytkowski, M., Scherer, R., Tadeusiewicz, R., Zadeh, L.A., Zurada, J.M. (eds.) ICAISC 2012. LNCS (LNAI), vol. 7268, pp. 347–354. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-29350-4_42
Szczepanik, M., Jóźwiak, I.: Data management for fingerprint recognition algorithm based on characteristic points’ groups. In: Pechenizkiy, M., Wojciechowski, M. (eds.) New Trends in Databases and Information Systems. Foundations of Computing and Decision Sciences, 38(2), pp. 123–130. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-32518-2_40
Huk, M.: Context-related data processing with artificial neural networks for higher reliability of telerehabilitation systems. In: 17th International Conference on E-Health Networking, Application & Services (HealthCom), pp. 217–221. IEEE Computer Society, Boston (2015)
Huk, M., Kwiatkowski, J., Konieczny, D., Kędziora, M., Mizera-Pietraszko, J.: Context-sensitive text mining with fitness leveling genetic algorithm. In: 2015 IEEE 2nd International Conference on Cybernetics (CYBCONF), Gdynia, Poland, Electronic Publication, pp. 1–6 (2015). ISBN: 978-1-4799-8321-6
Privitera, C.M., Azzariti, M., Stark, L.W.: Locating regions-of-interest for the Mars Rover expedition. Int. J. Remote Sens. 21, 3327–3347 (2000)
Dua, D., Graff, C.: UCI Machine Learning Repository. University of California, School of Information and Computer Science, Irvine, CA (2019). http://archive.ics.uci.edu/ml
Golub, T.R., et al.: Molecular classification of cancer: class discovery and class prediction by gene expression monitoring. Science 286, 531–537 (1999)
Huk, M., Kwaśnicka, H.: The concept and properties of Sigma-if neural network. In: Ribeiro, B., Albrecht, R.F., Dobnikar, A., Pearson, D.W., Steele, N.C. (eds.) Adaptive and Natural Computing Algorithms, ICANNGA 2005, pp. 13–17. Springer Computer Science, Vienna (2005). https://doi.org/10.1007/3-211-27389-1_4
Huk, M.: Manifestation of selective attention in Sigma-if neural network. In: 2nd International Symposium Advances in Artificial Intelligence and Applications, International Multiconference on Computer Science and Information Technology, IMCSIT/AAIA 2007, vol. 2, pp. 225–236 (2007)
Huk, M.: Sigma-if neural network as the use of selective attention technique in classification and knowledge discovery problems solving. Ann. UMCS Sectio AI Informatica 4(2), 121–131 (2006)
Raczkowski, D., Canning, A., Wang, L.: Thomas Fermi charge mixing for obtaining self-consistency in density functional calculations. Phys. Rev. B 64(12), 121101–121105 (2001)
Glosser, C., Piermarocchi, C., Shanker, B.: Analysis of dense quantum dot systems using a self-consistent Maxwell-Bloch framework. In: Proceedings of 2016 IEEE International Symposium on Antennas and Propagation (USNC-URSI), Puerto Rico, pp. 1323–1324. IEEE (2016)
H2O.ai: H2O Version 3.24.0.4, Fast Scalable Machine Learning API For Smarter Applications (2019). http://h2o-release.s3.amazonaws.com/h2o/rel-yates/4/index.html
Janusz, B.J., Wołk, K.: Implementing contextual neural networks in distributed machine learning framework. In: Nguyen, N.T., Hoang, D.H., Hong, T.-P., Pham, H., Trawiński, B. (eds.) ACIIDS 2018. LNCS (LNAI), vol. 10752, pp. 212–223. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-75420-8_20
Wołk, K., Burnell, E.: Implementation and analysis of contextual neural networks in H2O framework. In: Nguyen, N.T., Gaol, F.L., Hong, T.-P., Trawiński, B. (eds.) ACIIDS 2019. LNCS (LNAI), vol. 11432, pp. 429–440. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-14802-7_37
Bouckaert, R.R., Frank, E.: Evaluating the replicability of significance tests for comparing learning algorithms. In: Dai, H., Srikant, R., Zhang, C. (eds.) PAKDD 2004. LNCS (LNAI), vol. 3056, pp. 3–12. Springer, Heidelberg (2004). https://doi.org/10.1007/978-3-540-24775-3_3
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Burnell, E.D., Wołk, K., Waliczek, K., Kern, R. (2020). The Impact of Constant Field of Attention on Properties of Contextual Neural Networks. In: Nguyen, N., Jearanaitanakij, K., Selamat, A., Trawiński, B., Chittayasothorn, S. (eds) Intelligent Information and Database Systems. ACIIDS 2020. Lecture Notes in Computer Science(), vol 12034. Springer, Cham. https://doi.org/10.1007/978-3-030-42058-1_31
Download citation
DOI: https://doi.org/10.1007/978-3-030-42058-1_31
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-42057-4
Online ISBN: 978-3-030-42058-1
eBook Packages: Computer ScienceComputer Science (R0)