Abstract
Secure Multi-Party Computation (SMPC) is usually treated as a special encryption way. Unlike most encryption methods using a private or public key to encrypt data, it splits a value into different shares, and each share works like a private key. Only get all these shares, we can get the original data correctly. In this paper, we utilize SMPC to protect the privacy of gradient updates in distributed learning, where each client computes an update and shares their updates by encrypting them so that no information about the clients’ data can be leaked through the whole computing process. However, encryption brings a sharp increase in communication cost. To improve the training efficiency, we apply gradient sparsification to compress the gradient by sending only the important gradients. In order to improve the accuracy and efficiency of the model, we also make some improvements to the original sparsification algorithm. Extensive experiments show that the amount of data that needs to be transferred is reduced while the model still achieves 99.6% accuracy on the MNIST dataset.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Xing, E.P., et al.: Petuum: a new platform for distributed machine learning on big data. IEEE Trans. Big Data 1(2), 49–67 2015
Wang, Z., Song, M., Zhang, Z., et al.: Beyond inferring class representatives: user-level privacy leakage from federated learning. In: 2019-IEEE Conference on Computer Communications, pp. 2512-2520. IEEE (2019)
Melis, L., et al.: Exploiting unintended feature leakage in collaborative learning. In: 2019 IEEE Symposium on Security and Privacy (SP). IEEE (2019)
Ryffel, T., et al.: A generic framework for privacy preserving deep learning. arXiv preprint arXiv:1811.04017 (2018)
Bogdanov, D., et al.: High-performance secure multi-party computation for data mining applications. Int. J. Inf. Secur. 11(6), 403–418 (2012)
Ma, X., et al.: Privacy preserving multi-party computation delegation for deep learning in cloud computing. Inf. Sci. 459, 103–116 (2018)
Chase, M., et al.: Private collaborative neural network learning. IACR Cryptology ePrint Archive 2017, p. 762 (2017)
Speedtest.net: Speedtest market report, March 2020. https://www.speedtest.net/global-index#mobile
Han, S., Mao, H., Dally, W.J.: Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149 (2015)
Vogels, T., Karimireddy, S.P., Jaggi, M.: PowerSGD: practical low-rank gradient compression for distributed optimization. In: Advances in Neural Information Processing Systems (2019)
Konečný, J., et al.: Federated learning: strategies for improving communication efficiency. arXiv preprint arXiv:1610.05492 (2016)
Alistarh, D., et al.: QSGD: communication-efficient SGD via gradient quantization and encoding. In: Advances in Neural Information Processing Systems (2017)
Lin, Y., et al.: Deep gradient compression: reducing the communication bandwidth for distributed training. In: International Conference on Learning Representations (ICLR) (2018)
Ryffel, T., et al.: A generic framework for privacy preserving deep learning. arXiv preprint arXiv:1811.04017 (2018)
Qian, Ning: On the momentum term in gradient descent learning algorithms. Neural Netw. 12(1), 145–151 (1999)
Goyal, P., et al.: Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677 (2017)
Shokri, R., Shmatikov, V.: Privacy-preserving deep learning. In: Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security (2015)
Zhao, B., Mopuri, K.R., Bilen, H.: iDLG: improved deep leakage from gradients. arXiv preprint arXiv:2001.02610 (2020)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Zhang, M., Zhou, Q., Liang, S., Yang, H. (2020). Improving Communication Efficiency for Encrypted Distributed Training. In: Yu, S., Mueller, P., Qian, J. (eds) Security and Privacy in Digital Economy. SPDE 2020. Communications in Computer and Information Science, vol 1268. Springer, Singapore. https://doi.org/10.1007/978-981-15-9129-7_40
Download citation
DOI: https://doi.org/10.1007/978-981-15-9129-7_40
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-15-9128-0
Online ISBN: 978-981-15-9129-7
eBook Packages: Computer ScienceComputer Science (R0)