Skip to main content
SpringerLink
Log in

Incorporating Phrase-Level Agreement into Neural Machine Translation

  • Conference paper
  • First Online:
Natural Language Processing and Chinese Computing (NLPCC 2020)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 12430))

Abstract

Phrase information has been successfully integrated into current state-of-the-art neural machine translation (NMT) models. However, the natural property of the source and target phrase alignment has not been explored. In this paper, we propose a novel phrase-level agreement method to deal with this problem. First, we explore n-gram models over minimal translation units (MTUs) to explicitly capture aligned bilingual phrases from the parallel corpora. Then, we propose a phrase-level agreement loss that directly reduces the difference between the representations of the source-side and target-side phrase. Finally, we integrate the phrase-level agreement loss into the NMT models, to improve the translation performance. Empirical results on the NIST Chinese-to-English and the WMT English-to-German translation tasks demonstrate that the proposed phrase-level agreement method achieves significant improvements over state-of-the-art baselines, demonstrating the effectiveness and necessity of exploiting phrase-level agreement for NMT.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    The corpora include LDC2002E18, LDC2003E07, LDC2003E14, Hansards portion of LDC2004T07, LDC2004T08 and LDC2005T06.

  2. 2.

    https://github.com/facebookresearch/SentEval/tree/master/data/probing.

References

  1. Bahdanau, D., Cho, K., Bengio, Y.: Neural machine translation by jointly learning to align and translate. In: Proceedings of the 3rd ICLR (2015)

    Google Scholar 

  2. Chen, K., Wang, R., Utiyama, M., Sumita, E., Zhao, T.: Syntax-directed attention for neural machine translation. In: Proceedings of the 32nd AAAI (2018)

    Google Scholar 

  3. Cho, K., et al.: Learning phrase representations using RNN encoder-decoder for statistical machine translation. In: Proceedings of EMNLP 2014 (2014)

    Google Scholar 

  4. Conneau, A., Kruszewski, G., Lample, G., Barrault, L., Baroni, M.: What you can cram into a single \$&!#* vector: probing sentence embeddings for linguistic properties. In: Proceedings of the 56th ACL (2018)

    Google Scholar 

  5. Durrani, N., Schmid, H., Fraser, A.: A joint sequence translation model with integrated reordering. In: Proceedings of the 49th ACL (2011)

    Google Scholar 

  6. Hao, J., Wang, X., Shi, S., Zhang, J., Tu, Z.: Multi-granularity self-attention for neural machine translation. In: Proceedings of EMNLP-IJCNLP 2019 (2019)

    Google Scholar 

  7. Hao, J., Wang, X., Yang, B., Wang, L., Zhang, J., Tu, Z.: Modeling recurrence for transformer. In: NAACL 2019 (2019)

    Google Scholar 

  8. Hu, Y., Auli, M., Gao, Q., Gao, J.: Minimum translation modeling with recurrent neural networks. In: Proceedings of the 14th EACL (2014)

    Google Scholar 

  9. Huang, P., Wang, C., Huang, S., Zhou, D., Deng, L.: Towards neural phrase-based machine translation. In: Proceedings of the 6th ICLR (2018)

    Google Scholar 

  10. Kalchbrenner, N., Blunsom, P.: Recurrent continuous translation models. In: Proceedings of EMNLP 2013 (2013)

    Google Scholar 

  11. Kalchbrenner, N., Grefenstette, E., Blunsom, P.: A convolutional neural network for modelling sentences. In: Proceedings of the 52nd ACL (2014)

    Google Scholar 

  12. Koehn, P.: Statistical Machine Translation, 1st edn. Cambridge University Press, Cambridge (2010)

    MATH  Google Scholar 

  13. Koehn, P., Och, F.J., Marcu, D.: Statistical phrase-based translation. In: Proceedings of NAACL 2003 (2003)

    Google Scholar 

  14. Kuang, S., Li, J., Branco, A., Luo, W., Xiong, D.: Attention focusing for neural machine translation by bridging source and target embeddings. In: Proceedings of the 56th ACL (2018)

    Google Scholar 

  15. Le, Q.V., Mikolov, T.: Distributed representations of sentences and documents. In: Proceedings of ICML 2014 (2014)

    Google Scholar 

  16. Luong, T., Pham, H., Manning, C.D.: Effective approaches to attention-based neural machine translation. In: Proceedings of EMNLP 2015 (2015)

    Google Scholar 

  17. Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. CoRR abs/1301.3781 (2013)

    Google Scholar 

  18. Mitchell, J., Lapata, M.: Composition in distributional models of semantics. Cogn. Sci. 38(4), 1388–1429 (2010)

    Article  Google Scholar 

  19. Nguyen, P.X., Joty, S.: Phrase-based attentions. CoRR abs/1810.03444 (2018)

    Google Scholar 

  20. Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: BLEU: a method for automatic evaluation of machine translation. In: Proceedings of 40th ACL (2002)

    Google Scholar 

  21. Park, C.Y., Tsvetkov, Y.: Learning to generate word- and phrase-embeddings for efficient phrase-based neural machine translation. In: Proceedings of the 3rd Workshop on Neural Generation and Translation (2019)

    Google Scholar 

  22. Post, M.: A call for clarity in reporting BLEU scores. In: Proceedings of the Third Conference on Machine Translation: Research Papers (2018)

    Google Scholar 

  23. Quirk, C., Menezes, A.: Do we need phrases?: challenging the conventional wisdom in statistical machine translation. In: Proceedings of NAACL 2006 (2006)

    Google Scholar 

  24. Sennrich, R., Haddow, B., Birch, A.: Edinburgh neural machine translation systems for WMT 16. In: Proceedings of the First Conference on Machine Translation (2016)

    Google Scholar 

  25. Sennrich, R., Haddow, B., Birch, A.: Neural machine translation of rare words with subword units. In: Proceedings of the 54th ACL (2016)

    Google Scholar 

  26. Vaswani, A., et al.: Attention is all you need. In: Proceedings of Advances in Neural Information Processing Systems, vol. 30 (2017)

    Google Scholar 

  27. Wang, X., Lu, Z., Tu, Z., Li, H., Xiong, D., Zhang, M.: Neural machine translation advised by statistical machine translation. In: Proceedings of the 31th AAAI (2017)

    Google Scholar 

  28. Wang, X., Tu, Z., Wang, L., Shi, S.: Exploiting sentential context for neural machine translation. In: Proceedings of the 57th ACL (2019)

    Google Scholar 

  29. Wang, X., Tu, Z., Wang, L., Shi, S.: Self-attention with structural position representations. In: Proceedings of EMNLP-IJCNLP (2019)

    Google Scholar 

  30. Wang, X., Tu, Z., Xiong, D., Zhang, M.: Translating phrases in neural machine translation. In: Proceedings of EMNLP 2017 (2017)

    Google Scholar 

  31. Wuebker, J., Green, S., DeNero, J., Hasan, S., Luong, M.T.: Models and inference for prefix-constrained machine translation. In: Proceedings of the 54th ACL (2016)

    Google Scholar 

  32. Yang, B., Tu, Z., Wong, D.F., Meng, F., Chao, L.S., Zhang, T.: Modeling localness for self-attention networks. In: Proceedings of EMNLP 2018 (2018)

    Google Scholar 

  33. Yang, M., et al.: Sentence-level agreement for neural machine translation. In: Proceedings of the 57th ACL (2019)

    Google Scholar 

  34. Yang, M., Zhang, M., Chen, K., Wang, R., Zhao, T.: Neural machine translation with target-attention model. IEICE Trans. Inf. Syst. 103(3), 684–694 (2020)

    Article  Google Scholar 

  35. Zhang, H., Toutanova, K., Quirk, C., Gao, J.: Beyond left-to-right: multiple decomposition structures for SMT. In: Proceedings of NAACL 2013 (2013)

    Google Scholar 

  36. Zhang, M., Wu, Y., Li, W., Li, W.: Learning universal sentence representations with mean-max attention autoencoder. In: Proceedings of EMNLP 2018 (2018)

    Google Scholar 

  37. Zhao, Y., Wang, Y., Zhang, J., Zong, C.: Phrase table as recommendation memory for neural machine translation. In: Proceedings of the 27th IJCAI (2018)

    Google Scholar 

  38. Zhou, L., Hu, W., Zhang, J., Zong, C.: Neural system combination for machine translation. In: Proceedings of the 55th ACL (2017)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Harbin Institute of Technology, Harbin, China

    Mingming Yang & Tiejun Zhao

  2. Tencent AI Lab, Shenzhen, China

    Xing Wang

  3. School of Computer Science and Technology, Soochow University, Suzhou, China

    Min Zhang

Authors
  1. Mingming Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xing Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Min Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Tiejun Zhao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mingming Yang .

Editor information

Editors and Affiliations

  1. ECE & Ingenuity Labs Research Institute, Queen’s University, Kingston, ON, Canada

    Xiaodan Zhu

  2. Department of Computer Science and Technology, Tsinghua University, Beijing, China

    Min Zhang

  3. School of Computer Science and Technology, Soochow University, Suzhou, China

    Yu Hong

  4. College of Intelligence and Computing, Tianjin University, Tianjin, China

    Ruifang He

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Yang, M., Wang, X., Zhang, M., Zhao, T. (2020). Incorporating Phrase-Level Agreement into Neural Machine Translation. In: Zhu, X., Zhang, M., Hong, Y., He, R. (eds) Natural Language Processing and Chinese Computing. NLPCC 2020. Lecture Notes in Computer Science(), vol 12430. Springer, Cham. https://doi.org/10.1007/978-3-030-60450-9_33

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-60450-9_33

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-60449-3

  • Online ISBN: 978-3-030-60450-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation