Abstract
Ever growing interest and usage of deep learning rises a question on the performance of various infrastructures suitable for training of neural networks. We present here our approach and first results of tests performed with TensorFlow Benchmarks which use best practices for multi-GPU and distributed training. We pack the Benchmarks in Docker containers and execute them by means of uDocker and Singularity container tools on a single machine and in the HPC environment. The Benchmarks comprise a number of convolutional neural network models run across synthetic data and e.g. the ImageNet dataset. For the same Nvidia K80 GPU card we achieve the same performance in terms of processed images per second and similar scalability between 1-2-4 GPUs as presented by the TensorFlow developers. We therefore do not obtain statistically significant overhead due to the usage of containers in the multi-GPU case, and the approach of using TF Benchmarks in a Docker container can be applied across various systems.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Computational resource ForHLR-II available at Karlsruhe Institute of Technology. https://wiki.scc.kit.edu/hpc/index.php/Category:ForHLR. Accessed 12 Apr 2019
DEEP as a Service (DEEPaaS) API. https://deepaas.readthedocs.io. Accessed 12 Apr 2019
Docker Hub website. https://hub.docker.com. Accessed 12 Apr 2019
ImageNet Object Localization Challenge at Kaggle. https://www.kaggle.com/c/imagenet-object-localization-challenge. Accessed 12 Apr 2019
Nvidia Geforce GTX980 Ti specifications. https://www.geforce.com/hardware/desktop-gpus/geforce-gtx-980-ti/specifications. Accessed 12 Apr 2019
Nvidia Tesla K80 GPU Accelerator, Board Specification (2015). https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/tesla-product-literature/Tesla-K80-BoardSpec-07317-001-v05.pdf. Accessed 12 Apr 2019
Results of TF Benchmarks runs, GitHub repository. https://github.com/adriangrupp/tf_cnn_benchmarks_evaluation. Accessed 12 Apr 2019
Scripts for downloading and converting ImageNet data to TFRecord format. https://github.com/tensorflow/models/tree/master/research/inception/inception/data. Accessed 12 Apr 2019
TensorFlow Benchmarks, GitHub repository. https://github.com/tensorflow/benchmarks/. Accessed 8 Apr 2019
TensorFlow Benchmarks webpage. https://www.tensorflow.org/guide/performance/benchmarks. Accessed 8 Apr 2019
TF Benchmarks fork for TF 1.10.0, KIT Gitlab. https://git.scc.kit.edu/tf-benchmarks/tf_cnn_tf_benchmarks. Accessed 12 Apr 2019
udocker: A basic user tool to execute simple docker containers in batch or interactive systems without root privileges. GiHub repository. https://github.com/indigo-dc/udocker. Accessed 8 Apr 2019
Baker, M.: Is there a reproducibility crisis? Nature 533, 452–454 (2016)
Gomes, J., et al.: Enabling rootless linux containers in multi-user environments: the udocker tool. Comput. Phys. Commun. 232, 84–97 (2018). https://doi.org/10.1016/j.cpc.2018.05.021
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2015)
Jacobsen, D.M., Canon, R.S.: Contain this, unleashing Docker for HPC. Cray User Group 2015 (2015)
Krizhevsky, A.: Learning multiple layers of features from tiny images. Technical report (2009). https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf
Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Pereira, F., Burges, C.J.C., Bottou, L., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems, vol. 25, pp. 1097–1105. Curran Associates, Inc. (2012). http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf
Kurtzer, G.M., Sochat, V., Bauer, M.W.: Singularity: scientific containers for mobility of compute. PLoS One (2017). https://doi.org/10.1371/journal.pone.0177459
Lin, T., et al.: Microsoft COCO: common objects in context. CoRR abs/1405.0312 (2014). http://arxiv.org/abs/1405.0312
Merkel, D.: Docker: lightweight Linux containers for consistent development and deployment (2014). https://doi.org/10.1097/01.NND.0000320699.47006.a3
Panayotov, V., Chen, G., Povey, D., Khudanpur, S.: Librispeech: an ASR corpus based on public domain audio books, pp. 5206–5210, April 2015. https://doi.org/10.1109/ICASSP.2015.7178964
Priedhorsky, R., Randles, T.C., Randles, T.: Charliecloud: unprivileged containers for user-defined software stacks in HPC. In: SC17: International Conference for High Performance Computing, Networking, Storage and Analysis (2017). https://doi.org/10.1145/3126908.3126925
Russakovsky, O., et al.: ImageNet large scale visual recognition challenge. Int. J. Comput. Vis. (IJCV) 115(3), 211–252 (2015). https://doi.org/10.1007/s11263-015-0816-y
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. CoRR abs/1409.1556 (2014)
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: InceptionV3. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2818–2826 (2016)
Acknowledgments
uDocker is being developed within the DEEP HybridDataCloud project, which receives funding from the European Union’s Horizon 2020 research and innovation program under agreement RIA 777435.
A part of this work was performed on the computational resource ForHLR-II funded by the Ministry of Science, Research and the Arts Baden-Wuerttemberg and DFG (“Deutsche Forschungsgemeinschaft”).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2019 Springer Nature Switzerland AG
About this paper
Cite this paper
Grupp, A., Kozlov, V., Campos, I., David, M., Gomes, J., López García, Á. (2019). Benchmarking Deep Learning Infrastructures by Means of TensorFlow and Containers. In: Weiland, M., Juckeland, G., Alam, S., Jagode, H. (eds) High Performance Computing. ISC High Performance 2019. Lecture Notes in Computer Science(), vol 11887. Springer, Cham. https://doi.org/10.1007/978-3-030-34356-9_36
Download citation
DOI: https://doi.org/10.1007/978-3-030-34356-9_36
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-34355-2
Online ISBN: 978-3-030-34356-9
eBook Packages: Computer ScienceComputer Science (R0)