Abstract
One of the long-standing difficulties in machine learning involves distortions present in data – different input feature vectors may represent the same entity. This observation has led to the introduction of invariant machine learning methods, for example techniques that ignore shifts, rotations, or light and pose changes in images. These approaches typically utilize pre-defined invariant features or invariant kernels, and require the designer to analyze what type of distortions are to be expected. While specifying possible sources of variance is straightforward for images, it is more difficult in other domains. Here, we focus on learning an invariant representation from data, without any information of what the distortions present in the data, only based on information whether any two samples are distorted variants of the same entity, or not. In principle, standard neural network architectures should be able to learn the invariance from data, given sufficient numbers of examples of it. We report that, somewhat surprisingly, learning to approximate even a simple types of invariant representation is difficult. We then propose a new type of layer, with a richer output representation, one that is better suited for learning invariances from data.
Supported by NSF grant IIS-1453658.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Arodź, T.: Invariant object recognition using radon-based transform. Comput. Inform. 24(2), 183–199 (2012)
Benton, G., Finzi, M., Izmailov, P., Wilson, A.G.: Learning invariances in neural networks. In: Advances in Neural Information Processing Systems, vol. 33, pp. 17605–17616 (2020)
Cheng, G., Han, J., Zhou, P., Xu, D.: Learning rotation-invariant and fisher discriminative convolutional neural networks for object detection. IEEE Trans. Image Process. 28(1), 265–278 (2018)
Cybenko, G.: Approximation by superpositions of a sigmoidal function. Math. Control Signals Syst. 2(4), 303–314 (1989). https://doi.org/10.1007/BF02551274
Gama, J., Žliobaitė, I., Bifet, A., Pechenizkiy, M., Bouchachia, A.: A survey on concept drift adaptation. ACM Comput. Surv. 46(4), 1–37 (2014)
Haasdonk, B., Burkhardt, H.: Invariant kernel functions for pattern analysis and machine learning. Mach. Learn. 68(1), 35–61 (2007)
Hornik, K.: Approximation capabilities of multilayer feedforward networks. Neural Netw. 4(2), 251–257 (1991)
Khotanzad, A., Hong, Y.H.: Invariant image recognition by Zernike moments. IEEE Trans. Pattern Anal. Mach. Intell. 12(5), 489–497 (1990)
Krawczyk, B.: Learning from imbalanced data: open challenges and future directions. Prog. Artif. Intell. 5(4), 221–232 (2016). https://doi.org/10.1007/s13748-016-0094-0
Lai, J.H., Yuen, P.C., Feng, G.C.: Face recognition using holistic Fourier invariant features. Pattern Recogn. 34(1), 95–109 (2001)
Laptev, D., Savinov, N., Buhmann, J.M., Pollefeys, M.: Ti-pooling: transformation-invariant pooling for feature learning in convolutional neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 289–297 (2016)
LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proc. IEEE 86(11), 2278–2324 (1998)
Lenc, K., Vedaldi, A.: Understanding image representations by measuring their equivariance and equivalence. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 991–999 (2015)
Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Advances in Neural Information Processing Systems, pp. 3111–3119 (2013)
Panahi, A., Saeedi, S., Arodz, T.: word2ket: space-efficient word embeddings inspired by quantum entanglement. In: International Conference on Learning Representations (2019)
Pennington, J., Socher, R., Manning, C.: GloVe: global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pp. 1532–1543 (2014)
Qi, G.J., Zhang, L., Lin, F., Wang, X.: Learning generalized transformation equivariant representations via autoencoding transformations. IEEE Trans. Pattern Anal. Mach. Intell. (2020, in press). https://www.computer.org/csdl/journal/tp/5555/01/09219238/1nMMelzChbO
Shen, X., Tian, X., He, A., Sun, S., Tao, D.: Transform-invariant convolutional neural networks for image classification and search. In: Proceedings of the 24th ACM International Conference on Multimedia, pp. 1345–1354 (2016)
Wood, J.: Invariant pattern recognition: a review. Pattern Recogn. 29(1), 1–17 (1996)
Zhao, H., Des Combes, R.T., Zhang, K., Gordon, G.: On learning invariant representations for domain adaptation. In: International Conference on Machine Learning, pp. 7523–7532. PMLR (2019)
Acknowledgements
T.A. is supported by NSF grant IIS-1453658.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Zhang, H., Arodz, T. (2021). Learning Invariance in Deep Neural Networks. In: Paszynski, M., Kranzlmüller, D., Krzhizhanovskaya, V.V., Dongarra, J.J., Sloot, P.M. (eds) Computational Science – ICCS 2021. ICCS 2021. Lecture Notes in Computer Science(), vol 12744. Springer, Cham. https://doi.org/10.1007/978-3-030-77967-2_6
Download citation
DOI: https://doi.org/10.1007/978-3-030-77967-2_6
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-77966-5
Online ISBN: 978-3-030-77967-2
eBook Packages: Computer ScienceComputer Science (R0)