Abstract
Pre-trained models have shown their significant values on a number of natural language processing (NLP) tasks. However, there is still a lack of corresponding work in the field of service computing to effectively utilize the rich knowledge accumulated in the Web service ecosystem. In this paper, we propose ServiceBERT, which learns domain knowledge of Web service ecosystem aiming to support service intelligence tasks, such as Web API tagging and Mashup-oriented API recommendation. The ServiceBERT is developed with the Transformer-based neural architecture. In addition to using the objective of masked language modeling (MLM), we also introduce the replaced token detection (RTD) objective for efficiently learning pre-trained model. Finally, we also implement the contrastive learning to learn noise-invariant representations at the sentence level in pre-training stage. Comprehensive experiments on two service-related tasks successfully demonstrate the better performance of ServiceBERT through the comparison with a variety of representative methods.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Barros, A.P., Dumas, M.: The rise of web service ecosystems. Science and Engineering Faculty (2006)
Cao, Y., Liu, J., Cao, B., Shi, M., Wen, Y., Peng, Z.: Web services classification with topical attention based BI-LSTM. In: International Conference on Collaborative Computing: Networking, Applications and Worksharing, pp. 394–407 (2019)
Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: ICML 2020: 37th International Conference on Machine Learning, vol. 1, pp. 1597–1607 (2020)
Clark, K., Luong, M.T., Le, Q.V., Manning, C.D.: ELECTRA: pre-training text encoders as discriminators rather than generators. In: ICLR 2020 : Eighth International Conference on Learning Representations (2020)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.N.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171–4186 (2018)
Devries, T., Taylor, G.W.: Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv:1708.04552 (2017)
Fang, H., Xie, P.: CERT: contrastive self-supervised learning for language understanding. arXiv preprint arXiv:2005.12766 (2020)
Fletcher, K.K.: An attention model for mashup tag recommendation. In: International Conference on Services Computing, pp. 50–64 (2020)
Gidaris, S., Singh, P., Komodakis, N.: Unsupervised representation learning by predicting image rotations. In: International Conference on Learning Representations (2018)
Liu, Y., et al.: RoBERTa: a robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692 (2019)
Peters, M.E., et al.: Deep contextualized word representations. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), vol. 1, pp. 2227–2237 (2018)
Radford, A., Narasimhan, K., Salimans, T., Sutskever, I.: Improving language understanding by generative pre-training (2018)
Saied, M.A., Raelijohn, E., Batot, E., Famelis, M., Sahraoui, H.A.: Towards assisting developers in API usage by automated recovery of complex temporal patterns. Inf. Softw. Technol. 119(119), 106213 (2020)
Shi, M., Tang, Y., Liu, J.: Functional and contextual attention-based LSTM for service recommendation in mashup creation. IEEE Trans. Parallel Distrib. Syst. 30(5), 1077–1090 (2019)
Shi, M., Tang, Y., Liu, J.: TA-BLSTM: tag attention-based bidirectional long short-term memory for service recommendation in mashup creation. In: 2019 International Joint Conference on Neural Networks (IJCNN), pp. 1–8 (2019)
Szegedy, C., et al.: Going deeper with convolutions. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1–9 (2015)
Uddin, G., Khomh, F., Roy, C.K.: Mining API usage scenarios from stack overflow. Inf. Softw. Technol. 122(122), 106277 (2020)
Vaswani, A., et al.: Attention is all you need. In: Proceedings of the 31st International Conference on Neural Information Processing Systems, vol. 30, pp. 5998–6008 (2017)
Wang, X., Liu, J., Liu, X., Cui, X., Wu, H.: A novel dual-graph convolutional network based web service classification framework. In: 2020 IEEE International Conference on Web Services (ICWS), pp. 281–288 (2020)
Wang, X., Liu, J., Liu, X., Cui, X., Wu, H.: A spatial and sequential combined method for web service classification. In: APWeb-WAIM 2020 : Proceedings of the 4th Asia-Pacific and Web-Age Information Management International Joint Conference on Web and Big Data, pp. 764–778 (2020)
Wang, X., Liu, X., Liu, J., Chen, X., Wu, H.: A novel knowledge graph embedding based API recommendation method for mashup development. World Wide Web 24(3), 869–894 (2021)
Wang, X., et al.: SYNCOBERT: syntax-guided multi-modal contrastive pre-training for code representation. arXiv: Computation and Language (2021)
Wei, J.W., Zou, K.: Eda: easy data augmentation techniques for boosting performance on text classification tasks. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 6381–6387 (2019)
Wu, Z., Wang, S., Gu, J., Khabsa, M., Sun, F., Ma, H.: CLEAR: contrastive learning for sentence representation. arXiv preprint arXiv:2012.15466 (2020)
Xiong, R., Wang, J., Zhang, N., Ma, Y.: Deep hybrid collaborative filtering for web service recommendation. Expert Syst. Appl. 110, 191–205 (2018)
Yang, Y., et al.: ServeNet: a deep neural network for web services classification. In: 2020 IEEE International Conference on Web Services (ICWS), pp. 168–175 (2020)
Yao, L., Wang, X., Sheng, Q.Z., Benatallah, B., Huang, C.: Mashup recommendation by regularizing matrix factorization with API co-invocations. IEEE Trans. Serv. Comput. 14, 502–515 (2021)
Ye, H., Cao, B., Chen, J., Liu, J., Wen, Y., Chen, J.: A web services classification method based on GCN. In: 2019 IEEE International Conference on Parallel and Distributed Processing with Applications, Big Data and Cloud Computing, Sustainable Computing and Communications, Social Computing and Networking (ISPA/BDCloud/SocialCom/SustainCom), pp. 1107–1114 (2019)
Yuan, W., Nguyen, H.H., Jiang, L., Chen, Y., Zhao, J., Yu, H.: API recommendation for event-driven android application development. Inf. Softw. Technol. 107, 30–47 (2019)
Zhong, Y., Fan, Y., Tan, W., Zhang, J.: Web service recommendation with reconstructed profile from mashup descriptions. IEEE Trans. Autom. Sci. Eng. 15(2), 468–478 (2018)
Acknowledgment
This work is supported by the National Natural Science Foundation of China under Grants 61972290 and the National Key R&D Program of China under Grant 2018YFC1604000. This work is also supported by the National Natural Science Foundation of China (61962061) and partially supported by the Yunnan Provincial Foundation for Leaders of Disciplines in Science and Technology (202005AC160005).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Wang, X., Zhou, P., Wang, Y., Liu, X., Liu, J., Wu, H. (2021). ServiceBERT: A Pre-trained Model for Web Service Tagging and Recommendation. In: Hacid, H., Kao, O., Mecella, M., Moha, N., Paik, Hy. (eds) Service-Oriented Computing. ICSOC 2021. Lecture Notes in Computer Science(), vol 13121. Springer, Cham. https://doi.org/10.1007/978-3-030-91431-8_29
Download citation
DOI: https://doi.org/10.1007/978-3-030-91431-8_29
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-91430-1
Online ISBN: 978-3-030-91431-8
eBook Packages: Computer ScienceComputer Science (R0)