Skip to main content
SpringerLink
Log in

OdiEnCorp: Odia–English and Odia-Only Corpus for Machine Translation

  • Conference paper
  • First Online:
Smart Intelligent Computing and Applications

Part of the book series: Smart Innovation, Systems and Technologies ((SIST,volume 159))

Abstract

A multilingual country like India needs language corpora for low-resource languages not only to provide its citizens with technologies of natural language processing (NLP) readily available in other countries, but also to support its people in their education and cultural needs. In this work, we focus on one of the low-resource languages, Odia, and build an Odia–English parallel (OdiEnCorp) and an Odia monolingual (OdiMonoCorp) corpus. The parallel corpus is based on Odia–English parallel texts extracted from online resources and formally corrected by volunteers. We also preprocess the parallel corpus for machine translation research or training. The monolingual corpus comes from a diverse set of online resources and we organize it into a collection of segments and paragraphs, easy to handle by NLP tools. OdiEnCorp parallel corpus contains 29,346 sentence pairs and 756K English and 648K Odia tokens. OdiMonoCorp contains 2.6 million tokens in 221K sentences in 71K paragraphs. Despite their small size, OdiEnCorp and OdiMonoCorp are still the largest Odia language resources, freely available for noncommercial educational or research purposes.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 219.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://en.wikipedia.org/wiki/Odia_language.

  2. 2.

    https://www.britannica.com/topic/Oriya-language.

  3. 3.

    https://www.wordproject.org/bibles/parallel/oriya/.

  4. 4.

    http://www.odisha.gov.in/.

  5. 5.

    http://www.homeodisha.gov.in/.

  6. 6.

    http://odiabibhaba.in/.

  7. 7.

    http://ova.odisha.gov.in/en/.

  8. 8.

    https://en.wikipedia.org/wiki/Odia_Wikipedia.

  9. 9.

    http://magazines.odisha.gov.in/Orissareview/2014/Jun/engpdf/158-160.pdf.

  10. 10.

    http://www.ameodia.com/.

  11. 11.

    http://www.aahwaan.com/.

  12. 12.

    http://www.odiasahitya.com/.

  13. 13.

    http://odiagapa.com/.

  14. 14.

    https://anoopkunchukuttan.github.io/indic_nlp_library/.

  15. 15.

    http://mokk.bme.hu/en/resources/hunalign/.

  16. 16.

    https://nvidia.github.io/OpenSeq2Seq/html/api-docs/optimizers.html.

  17. 17.

    https://github.com/mjpost/sacreBLEU.

  18. 18.

    SacreBleu Signatures: BLEU+case.mixed+numrefs.1+smooth.exp+tok.13a+version.1.2.11.

  19. 19.

    https://creativecommons.org/licenses/by-nc-sa/4.0/.

References

  1. Balabantaray, R., Sahoo, D.: An experiment to create parallel corpora for Odia. IJCA 67(19) (2013)

    Article  Google Scholar 

  2. Behera, P., Singh, R., Jha, G.N.: Evaluation of Anuvadaksh (EILMT) English-Odia machine-assisted translation tool. In: WILDRE: LREC (2016)

    Google Scholar 

  3. Dash, N.S., Chaudhuri, B.B.: Why do we need to develop corpora in indian languages. In: International Conference on SCALLA, Bangalore (2001)

    Google Scholar 

  4. Gaurav Mohanty, P.M., Mamidi, R.: Kabithaa: an annotated corpus of Odia poems with sentiment polarity information. In: Proceedings of LREC. ELRA (May 2018)

    Google Scholar 

  5. Godase, A., Govilkar, S.: Machine translation development for indian languages and its approaches. Int. J. Natl. Lang. Comput. (IJNLC) 4(2), 55–74 (2015)

    Article  Google Scholar 

  6. Johnson, M., Schuster, M., Le, Q.V., Krikun, M., Wu, Y., Chen, Z., Thorat, N., Viégas, F., Wattenberg, M., Corrado, G., Hughes, M., Dean, J.: Google’s multilingual neural machine translation system: enabling zero-shot translation. TACL 5, 339–351 http://aclweb.org/anthology/Q17-1024 (2017)

  7. Kalyani, A., Sajja, P.S.: A review of machine translation systems in india and different translation evaluation methodologies. IJCA 121(23) (2015)

    Article  Google Scholar 

  8. Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B., Shen, W., Moran, C., Zens, R., Dyer, C., Bojar, O., Constantin, A., Herbst, E.: Moses: open source toolkit for statistical machine translation. In: Proceedings of ACL demo and poster sessions, pp. 177–180. ACL (June 2007)

    Google Scholar 

  9. Koehn, P., Knowles, R.: Six challenges for neural machine translation. In: Proceedings of WNMT. ACL (August 2017)

    Google Scholar 

  10. Margaretha, E., Lüngen, H.: Building linguistic corpora from Wikipedia articles and discussions. JLCL 29(2), 59–82 (2014)

    Google Scholar 

  11. Mohanty, G., Kannan, A., Mamidi, R.: Building a sentiwordnet for Odia. In: Proceedings of Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pp. 143–148 (2017)

    Google Scholar 

  12. Naskar, S., Bandyopadhyay, S.: Use of machine translation in India: current status. AAMT J. 25–31 (2005)

    Google Scholar 

  13. Popel, M., Bojar, O.: Training tips for the transformer model. Prague Bull. Math. Linguist. 110(1), 43–70 (2018)

    Article  Google Scholar 

  14. Rautaray, J., Hota, A., Gochhayat, S.S.: A shallow parser-based Hindi to Odia machine translation system. In: Computational Intelligence in Data Mining, pp. 51–62. Springer, Berlin (2019)

    Google Scholar 

  15. Varga, D., Halácsy, P., Kornai, A., Nagy, V., Németh, L., Trón, V.: Parallel corpora for medium density languages. In: Amsterdam Studies in the Theory and History of Linguistic Science Series 4, vol. 292, p. 247 (2007)

    Google Scholar 

  16. Vaswani, A., Bengio, S., Brevdo, E., Chollet, F., Gomez, A., Gouws, S., Jones, L., Kaiser, L., Kalchbrenner, N., Parmar, N., Sepassi, R., Shazeer, N., Uszkoreit, J.: Tensor2tensor for neural machine translation. In: Proceedings of AMTA Research Papers, pp. 193–199. AMTA. http://aclweb.org/anthology/W18-1819 (2018)

Download references

Acknowledgements

This study was supported by the grant 18-24210S of the Czech Science Foundation. This work has been using language resources and tools stored and distributed by the LINDAT/CLARIN project of the Ministry of Education, Youth and Sports of the Czech Republic (projects LM2015071 and OP VVV VI CZ.02.1.01/0.0/0.0/16 013/0001781). The work was carried out during Shantipriya Parida’s post-doc fully funded by Charles University.

Author information

Authors and Affiliations

  1. Charles University, Faculty of Mathematics and Physics, Institute of Formal and Applied Linguistics, Prague, Czech Republic

    Shantipriya Parida & Ondřej Bojar

  2. School of Computer Applications, Kalinga Institute of Industrial Technology (KIIT), Deemed to be University, Bhubaneswar, 751024, Odisha, India

    Satya Ranjan Dash

Authors
  1. Shantipriya Parida
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ondřej Bojar
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Satya Ranjan Dash
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ondřej Bojar .

Editor information

Editors and Affiliations

  1. School of Computer Engineering, KIIT Deemed to be University, Bhubaneswar, Odisha, India

    Suresh Chandra Satapathy

  2. Department of Electronics and Communication Engineering, Shri Ramswaroop Memorial Group of Professional Colleges, Lucknow, Uttar Pradesh, India

    Vikrant Bhateja

  3. School of Computer Applications, KIIT University, Bhubaneswar, Odisha, India

    J. R. Mohanty

  4. School of Computer and Information Science, University of Hyderabad, Hyderabad, Telangana, India

    Siba K. Udgata

Rights and permissions

Reprints and permissions

Copyright information

© 2020 Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Parida, S., Bojar, O., Dash, S.R. (2020). OdiEnCorp: Odia–English and Odia-Only Corpus for Machine Translation. In: Satapathy, S., Bhateja, V., Mohanty, J., Udgata, S. (eds) Smart Intelligent Computing and Applications . Smart Innovation, Systems and Technologies, vol 159. Springer, Singapore. https://doi.org/10.1007/978-981-13-9282-5_47

Download citation

Publish with us

Policies and ethics

Search

Navigation