Abstract
Answer selection (AS), as one of the hottest topics in the field of natural language processing, has developed rapidly with outstanding performances reported, especially with the emergency of pretrained model (e.g., BERT). However, the current BERT based AS methods applied BERT only by fine-tuning or stacking other modules such as CNN and RNN, but ignored to exploit the discrimination embedded inside the BERT. In this paper, we proposed a novel method LMPF-IE, i.e., Lightweight Multiple Perspective Fusion with Information Enriching. The method can mine and fuse the multi-layer discrimination inside different layers of BERT and can use Question Category and Name Entity Recognition to enrich the information which can help BERT better understand the relationship between questions and answers. We test the proposed BERT layer-wised attention model in 5 benchmark datasets of answer selection task. The experimental results clearly verify better performances than the baseline models can be achieved by our method.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of NAACL: Human Language Technologies. Association for Computational Linguistics, June 2019
Li, D., Yu, Y., Chen, Q., Li, X.: BERTSel: answer selection with pre-trained models. CoRR abs/1905.07588 (2019)
Loshchilov, I., Hutter, F.: Fixing weight decay regularization in adam. CoRR abs/1711.05101 (2017)
Madabushi, H.T., Lee, M., Barnden, J.: Integrating question classification and deep learning for improved answer selection. In: Bender, E.M., Derczynski, L., Isabelle, P. (eds.) Proceedings of the 27th International Conference on Computational Linguistics, COLING 2018, Santa Fe, New Mexico, USA, 20–26 August 2018, pp. 3283–3294. Association for Computational Linguistics (2018)
Mozafari, J., Fatemi, A., Nematbakhsh, M.A.: BAS: an answer selection method using BERT language model. CoRR abs/1911.01528 (2019)
Nakov, P., et al.: Semeval-2017 task 3: community question answering. In: Proceedings of the 11th International Workshop on Semantic Evaluation, SemEval@ACL 2017 (2017)
Nakov, P., et al.: Semeval-2016 task 3: community question answering. In: Proceedings of the 10th International Workshop on Semantic Evaluation, SemEval@NAACL-HLT 2016 (2016)
Peters, M., et al.: Deep contextualized word representations. In: Proceedings of the 2018 Conference of NAACL: Human Language Technologies. Association for Computational Linguistics, June 2018
Peters, M.E., Ruder, S., Smith, N.A.: To tune or not to tune? Adapting pretrained representations to diverse tasks (2019)
Qi, P., Zhang, Y., Zhang, Y., Bolton, J., Manning, C.D.: Stanza: a Python natural language processing toolkit for many human languages. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations (2020)
Qiao, Y., Xiong, C., Liu, Z., Liu, Z.: Understanding the behaviors of BERT in ranking (2019)
Qin, S., Rong, W., Shi, L., Yang, J., Yang, H., Xiong, Z.: Syntax tree aware adversarial question rewriting for answer selection. In: 2019 IJCNN, July 2019
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I.: Language models are unsupervised multitask learners (2019)
Rossetto, F., Gravina, A., Severini, S., Attardi, G.: A comparative study of models for answer sentence selection. In: Proceedings of the 6th CLiC-it (2019)
dos Santos, C., Tan, M., Xiang, B., Zhou, B.: Attentive pooling networks (2016)
Severyn, A., Moschitti, A.: Learning to rank short text pairs with convolutional deep neural networks. In: Proceedings of the 38th ACM SIGIR Conference on Research and Development in Information Retrieval (2015)
Sha, L., Zhang, X., Qian, F., Chang, B., Sui, Z.: A multi-view fusion neural network for answer selection (2018)
Tang, D., Rong, W., Qin, S., Yang, J., Xiong, Z.: A n-gated recurrent unit with review for answer selection. Neurocomputing 371, 158–165 (2020)
Tay, Y., Phan, M.C., Luu, A.T., Hui, S.C.: Learning to rank question answer pairs with holographic dual LSTM architecture. In: Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval (2017)
Tay, Y., Tuan, L.A., Hui, S.C.: Hyperbolic representation learning for fast and efficient neural question answering. In: Proceedings of the 11th WSDM, WSDM 2018, Association for Computing Machinery (2018)
Tran, N.K., Niederée, C.: Multihop attention networks for question answer matching. In: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018 (2018)
Wang, B., Liu, K., Zhao, J.: Inner attention based recurrent neural networks for answer selection. In: Proceedings of the 54th ACL. Association for Computational Linguistics, August 2016
Wang, C., Jiang, F., Yang, H.: A hybrid framework for text modeling with convolutional RNN. In: the 23rd ACM SIGKDD Conference (2017)
Wang, D., Nyberg, E.: A long short-term memory model for answer sentence selection in question answering. In: Proceedings of the 53rd ACL and the 7th IJCNLP. Association for Computational Linguistics, July 2015
Wang, M., Manning, C.: Probabilistic tree-edit models with structured latent variables for textual entailment and question answering. In: Proceedings of the 23rd ICCL (Coling 2010). Coling 2010 Organizing Committee, August 2010
Wang, M., Smith, N.A., Mitamura, T.: What is the Jeopardy model? A quasi-synchronous grammar for QA. In: Proceedings of the 2007 Joint Conference on EMNLP and CoNLL. Association for Computational Linguistics, June 2007
Wang, Z., Hamza, W., Florian, R.: Bilateral multi-perspective matching for natural language sentences. In: 26th IJCAI (2017)
Yang, Y., Yih, W.T., Meek, C.: WikiQA: A challenge dataset for open-domain question answering. In: Proceedings of the 2015 Conference on EMNLP. Association for Computational Linguistics, September 2015
Yih, W.T., Chang, M.W., Meek, C., Pastusiak, A.: Question answering using enhanced lexical semantic models. In: Proceedings of the 51st ACL. Association for Computational Linguistics, August 2013
Yu, L., Hermann, K.M., Blunsom, P., Pulman, S.: Deep learning for answer sentence selection. CoRR abs/1412.1632 (2014)
Acknowledgement
This work is partially supported by National Natural Science Foundation of China (Grants no. 61772568), Guangdong Basic and Applied Basic Research Foundation (Grant no. 2019A1515012029), and Guangdong Special Support Program.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Gu, Y., Yang, M., Lin, P. (2020). Lightweight Multiple Perspective Fusion with Information Enriching for BERT-Based Answer Selection. In: Zhu, X., Zhang, M., Hong, Y., He, R. (eds) Natural Language Processing and Chinese Computing. NLPCC 2020. Lecture Notes in Computer Science(), vol 12430. Springer, Cham. https://doi.org/10.1007/978-3-030-60450-9_43
Download citation
DOI: https://doi.org/10.1007/978-3-030-60450-9_43
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-60449-3
Online ISBN: 978-3-030-60450-9
eBook Packages: Computer ScienceComputer Science (R0)
