Abstract
With their capability to recognise complex patterns in data, deep learning models are rapidly becoming the most prominent set of tools for a broad range of data science tasks from image classification to natural language processing. This trend is supplemented by the availability of deep learning software platforms and modern hardware environments. We propose a declarative benchmarking framework to evaluate the performance of different software and hardware systems. We further use our framework to analyse the performance of three different software frameworks on different hardware setups for a representative set of deep learning workloads and corresponding neural network architectures (Our framework is publicly available at https://github.com/vdeuschle/rysia.).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
As of Version 1.0 Pytorch features a just-in-time compiler that will enable user to precompile static models before runtime without the need of symbolic operators.
- 2.
Under ideal conditions, our approach should indeed result in exactly the same accuracy curves between platforms. In reality however, even ensuring that the same operations are performed on exactly the same data in each step, does not result in perfectly aligning accuracy rates.
References
Amazon Web Services. https://aws.amazon.com/. Accessed 28 May 2019
Apache Incubator. https://incubator.apache.org/. Accessed 10 June 2019
CIFAR-10 dataset. https://www.cs.toronto.edu/~kriz/cifar.html. Accessed 10 June 2019
Comparing deep learning frameworks: a rosetta stone approach. https://blogs.technet.microsoft.com/machinelearning/2018/03/14/comparing-deep-learning-frameworks-a-rosetta-stone-approach/. Accessed 10 June 2019
Docker Platform. https://www.docker.com/. Accessed 10 June 2019
EC2 Instances. https://aws.amazon.com/ec2/instance-types/. Accessed 10 June 2019
Eigen Library. http://eigen.tuxfamily.org/index.php?title=Main_Page. Accessed 10 June 2019
Facebook AI Research. https://research.fb.com/category/facebook-ai-research/. Accessed 10 June 2019
Google Brain. https://ai.google/research/teams/brain. Accessed 10 June 2019
Keras Framework. https://keras.io/. Accessed 10 June 2019
MLPerf benchmark suite. https://mlperf.org/. Accessed 10 June 2019
MNIST database of handwritten digits. http://yann.lecun.com/exdb/mnist/. Accessed 10 June 2019
NVIDIA Cuda. https://developer.nvidia.com/cuda-toolkit. Accessed 10 June 2019
OpenBlas Library. https://www.openblas.net/. Accessed 10 June 2019
Perform sentiment analysis with LSTMs, using TensorFlow. https://www.oreilly.com/learning/perform-sentiment-analysis-with-lstms-using-tensorflow. Accessed 10 June 2019
Abadi, M., et al.: Tensorflow: a system for large-scale machine learning. In: OSDI, vol. 16, pp. 265–283 (2016)
Baydin, A.G., Pearlmutter, B.A., Radul, A.A., Siskind, J.M.: Automatic differentiation in machine learning: a survey. J. Mach. Learn. Res. 18(153), 1–43 (2018). http://jmlr.org/papers/v18/17-468.html
Bourrasset, C., et al.: Requirements for an enterprise AI benchmark. In: Nambiar, R., Poess, M. (eds.) TPCTC 2018. LNCS, vol. 11135, pp. 71–81. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-11404-6_6
Chen, T., et al.: MXNet: a flexible and efficient machine learning library for heterogeneous distributed systems. arXiv preprint arXiv:1512.01274 (2015)
Coleman, C., et al.: DAWNBench: an end-to-end deep learning benchmark and competition. Training 100(101), 102 (2017)
Collobert, R., Kavukcuoglu, K., Farabet, C.: Torch7: A matlab-like environment for machine learning. In: BigLearn, NIPS Workshop. No. EPFL-CONF-192376 (2011)
Hecht-Nielsen, R.: Theory of the backpropagation neural network. In: Neural Networks for Perception, pp. 65–93. Elsevier (1992)
Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)
Maas, A.L., Daly, R.E., Pham, P.T., Huang, D., Ng, A.Y., Potts, C.: Learning word vectors for sentiment analysis. In: Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142–150. Association for Computational Linguistics, Portland, June 2011. http://www.aclweb.org/anthology/P11-1015
Pennington, J., Socher, R., Manning, C.D.: GloVe: global vectors for word representation. In: Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543 (2014). http://www.aclweb.org/anthology/D14-1162
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
Svozil, D., Kvasnicka, V., Pospichal, J.: Introduction to multi-layer feed-forward neural networks. Chemometr. Intell. Lab. Syst. 39(1), 43–62 (1997)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Deuschle, V., Alexandrov, A., Januschowski, T., Markl, V. (2020). End-to-End Benchmarking of Deep Learning Platforms. In: Nambiar, R., Poess, M. (eds) Performance Evaluation and Benchmarking for the Era of Cloud(s). TPCTC 2019. Lecture Notes in Computer Science(), vol 12257. Springer, Cham. https://doi.org/10.1007/978-3-030-55024-0_8
Download citation
DOI: https://doi.org/10.1007/978-3-030-55024-0_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-55023-3
Online ISBN: 978-3-030-55024-0
eBook Packages: Computer ScienceComputer Science (R0)