Abstract
Deep neural networks (DNNs) are notorious for two key drawbacks: the vulnerability against adversarial attacks and the prohibitive cost of storage and computation, which greatly hinders DNNs’ deployment on safety-critical yet resource-limited platforms. Although researchers have proposed adversary-aware pruning methods where adversarial training and network pruning are studied jointly to improve the robustness of pruned networks, they failed to attain a double-win, i.e., the achieved robustness is still limited and cannot surpass that of dense networks. In this work, pursuing a win-win in robustness and efficiency, we demonstrate that the robustness of pruned networks can be easily boosted by leveraging the stochastic policy. More specifically, we propose a Random Mask Selection (RMS) strategy where pruning masks are randomly sampled during inference to confuse attackers. Furthermore, a necessary hardware-aware algorithm optimization is introduced to eliminate the potential hardware overhead of RMS, and thus ensures a convenient implementation of RMS on existing hardware accelerators without sacrificing processing speed or power efficiency. Extensive experiments show that our approach achieves a double-win in robustness and compactness compared to dense models and outperforms the SOTA adversary-aware pruning method in terms of robustness.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Chen, C., Seff, A., Kornhauser, A., Xiao, J.: DeepDriving: learning affordance for direct perception in autonomous driving. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2722–2730 (2015)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
Feinman, R., Curtin, R.R., Shintre, S., Gardner, A.B.: Detecting adversarial samples from artifacts. arXiv preprint arXiv:1703.00410 (2017)
Figurnov, M., et al.: Spatially adaptive computation time for residual networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1039–1048 (2017)
Fu, Y., et al.: Drawing robust scratch tickets: subnetworks with inborn robustness are found within randomly initialized networks. Adv. Neural Inf. Process. Syst. 34, 20 (2021)
Gondimalla, A., Chesnut, N., Thottethodi, M., Vijaykumar, T.: SparTen: a sparse tensor accelerator for convolutional neural networks. In: Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture, pp. 151–165 (2019)
Goodfellow, I.J., Shlens, J., Szegedy, C.: Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 (2014)
Guo, Y., Zhang, C., Zhang, C., Chen, Y.: Sparse DNNs with improved adversarial robustness. Adv. Neural Inf. Process. Syst. 31, 10 (2018)
Han, S., Mao, H., Dally, W.J.: Deep compression: compressing deep neural networks with pruning, trained quantization and Huffman coding. arXiv preprint arXiv:1510.00149 (2015)
Jacob, B., et al.: Quantization and training of neural networks for efficient integer-arithmetic-only inference. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2704–2713 (2018)
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., Vladu, A.: Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083 (2017)
Osborne, M.J., Rubinstein, A.: A Course in Game Theory. MIT Press, Cambridge (1994)
Rakin, A.S., He, Z., Yang, L., Wang, Y., Wang, L., Fan, D.: Robust sparse regularization: simultaneously optimizing neural network robustness and compactness. arXiv preprint arXiv:1905.13074 (2019)
Sehwag, V., Wang, S., Mittal, P., Jana, S.: HYDRA: pruning adversarially robust neural networks. Adv. Neural. Inf. Process. Syst. 33, 19655–19666 (2020)
Szegedy, C., et al.: Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 (2013)
Wang, L., Liu, T., Wang, G., Chan, K.L., Yang, Q.: Video tracking using learned hierarchical features. IEEE Trans. Image Process. 24(4), 1424–1435 (2015)
Wu, B., et al.: FBNet: hardware-aware efficient convnet design via differentiable neural architecture search. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10734–10742 (2019)
Ye, S., et al.: Adversarial robustness vs. model compression, or both? In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 111–120 (2019)
Zhao, Y., Li, C., Wang, Y., Xu, P., Zhang, Y., Lin, Y.: DNN-chip predictor: an analytical performance predictor for DNN accelerators with various dataflows and hardware architectures. In: ICASSP 2020–2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1593–1597. IEEE (2020)
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Xue, R., Wang, M., Wang, Z. (2022). Boosting Both Robustness and Hardware Efficiency via Random Pruning Mask Selection. In: Pimenidis, E., Angelov, P., Jayne, C., Papaleonidas, A., Aydin, M. (eds) Artificial Neural Networks and Machine Learning – ICANN 2022. ICANN 2022. Lecture Notes in Computer Science, vol 13529. Springer, Cham. https://doi.org/10.1007/978-3-031-15919-0_5
Download citation
DOI: https://doi.org/10.1007/978-3-031-15919-0_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-15918-3
Online ISBN: 978-3-031-15919-0
eBook Packages: Computer ScienceComputer Science (R0)