Skip to main content
SpringerLink
Log in

A New Tool for Benchmarking and Assessing Arabic Syntactic Parsers

  • Conference paper
  • First Online:
Arabic Language Processing: From Theory to Practice (ICALP 2017)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 782))

Included in the following conference series:

  • 928 Accesses

Abstract

This work aims to develop a Natural Language Processing (NLP) tool for benchmarking and assessing Arabic syntactic parsers. This tool is integrated within the Software Architecture For Arabic language pRocessing (SAFAR). Indeed, SAFAR contains several ANLP tools from simple preprocessing up to the semantic level. The benchmarking tool will take advantage of the available basic tools in addition to the flexibility and reusability of SAFAR. The benchmark process takes as input an evaluation corpus and one/several syntactic parsers implementations. As a result, it outputs the most common metrics used for evaluation namely: precision, recall, accuracy and F-measure. We introduced also a new metric called Gp-score which takes into account the execution time besides the accuracy. The execution time is very crucial for some tasks such as real-time automatic translators or in the context of processing huge data. This benchmarking solution will help researchers in comparing their parsers against each other; it will help as well other researchers in selecting the appropriate parser to use within their high level projects. Two Arabic syntactic parsers are evaluated to give a concrete example of this tool: The Stanford parser and the ATKS parser.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    http://tree-edit-distance.dbresearch.uni-salzburg.at/.

  2. 2.

    http://www.tsarfaty.com/unipar/download.html.

  3. 3.

    https://github.com/jkkummerfeld/berkeley-parser-analyser.

  4. 4.

    http://en.wikipedia.org/wiki/Precision_and_recall.

  5. 5.

    http://arabic.emi.ac.ma/safar/javadoc.

  6. 6.

    https://www.iso.org/standard/37243.html.

References

  1. Miniwatts Marketing Group (2001). http://www.internetworldstats.com

  2. Hattab, M., Haddad, B., Yaseen, M., Duraidi, A., Shmais, A.A.: Addaall Arabic search engine: improving search based on combination of morphological analysis and generation considering semantic patterns. In: The 2nd International Conference on Arabic Language Resources & Tools (2009)

    Google Scholar 

  3. Ittycheriah, A., Roukos, S.: A maximum entropy word aligner for Arabic-English machine translation. In: Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing (2005)

    Google Scholar 

  4. Jaafar, Y., Namly, D., Bouzoubaa, K., Yousfi, A.: Enhancing Arabic stemming process using resources and benchmarking tools. J. King Saud Univ. Comput. Inf. Sci. 29(2), 164–170 (2016)

    Google Scholar 

  5. Jaafar, Y., Bouzoubaa, K.: Benchmark of Arabic morphological analyzers: challenges and solutions. In: 9th International Conference on Intelligent Systems: Theories and Applications (SITA 2014), Rabat, Morocco (2014)

    Google Scholar 

  6. Jaafar, Y., Bouzoubaa, K., Yousfi, A., Tajmout, R., Khamar, H.: Improving Arabic morphological analyzers benchmark. Int. J. Speech Technol. 19(2), 259–267 (2016)

    Article  Google Scholar 

  7. Green, S., Manning, C.D.: Better Arabic parsing: baselines, evaluations, and analysis. In: The 23rd International Conference on Computational Linguistics (COLING 2010), Beijing (2010)

    Google Scholar 

  8. Microsoft: Arabic Toolkit Service (ATKS). https://www.microsoft.com/en-us/research/project/arabic-toolkit-service-atks/. Accessed 01 Mar 2017

  9. Pawlik, M., Augsten, N.: Tree edit distance. Inf. Syst. 56(C), 157–173 (2016)

    Article  Google Scholar 

  10. Pawlik, M., Augsten, N.: RTED: a robust algorithm for the tree edit distance. In: Proceedings of the VLDB Endowment (2011)

    Google Scholar 

  11. Atwell, E.: Comparative evaluation of grammatical annotation models. Ind. Parsing Software Manuals 17, 25–46 (1996)

    Google Scholar 

  12. Tsarfaty, R., Nivre, J., Andersson, E.: Cross-framework evaluation for statistical parsing. In: Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics (2012)

    Google Scholar 

  13. Black, E., Abney, S., Flickenger, D., Gdaniec, C., Grishman, R., Harrison, P., Hindle, D., Ingria, R., Jelinek, F., Klavans, J., Liberman, M., Marcus, M., Roukos, S., Santorini, B., Strzalkowski, T.: A procedure for quantitatively comparing the syntactic coverage of English grammars. In: Proceedings of the Fourth DARPA Speech and Natural Language Workshop, Pacific Grove (1991)

    Google Scholar 

  14. Harrison, P., Abney, S., Black, E., Flickinger, D., Gdaniec, C., Grishman, R., Hindle, D., Ingria, R., Marcus, M., Santorini, B., Strzalkowski, T.: Evaluating syntax performance of parser/grammars. In: Proceedings of the Natural Language Processing Systems Evaluation Workshop, Berekely (1991)

    Google Scholar 

  15. Lin, D.: A dependency-based method for evaluating broad-coverage parsers. Nat. Lang. Eng. 4(02), 97–114 (1998)

    Article  Google Scholar 

  16. Kummerfeld, J.K., Hall, D., Curran, J.R., Klein, D.: Parser showdown at the wall street corral: an empirical investigation of error types in parser output. In: Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, Jeju Island (2012)

    Google Scholar 

  17. Seddah, D., Candito, M., Crabbé, B.: Cross parser evaluation and tagset variation: a French Treebank study. In: Proceedings of the 11th International Conference on Parsing Technologies (2009)

    Google Scholar 

  18. Hall, D., Berg-Kirkpatrick, T., Klein, D.: Sparser, Better, Faster GPU Parsing. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Baltimore (2014)

    Google Scholar 

  19. Bikel, D.: Design of a multi-lingual, parallel-processing statistical parsing engine. In: Proceedings of the Second International Conference on Human Language Technology Research (2002)

    Google Scholar 

  20. Charniak, E.: A maximum-entropy-inspired parser. In: Proceedings of the 1st North American Chapter of the Association for Computational Linguistics Conference (2000)

    Google Scholar 

  21. Collins, M.J.: A new statistical parser based on bigram lexical dependencies. In: Proceedings of the 34th Annual Meeting on Association for Computational Linguistics (1996)

    Google Scholar 

  22. Maamouri, M., Bies, A., Buckwalter, T., Mekki, W.: The Penn Arabic Treebank: building a large-scale annotated Arabic corpus. In: NEMLAR Conference on Arabic Language Resources and Tools, vol. 27, pp. 466–467 (2004)

    Google Scholar 

  23. Smrž, O., Bielický, V., Jakub, I.K.: Prague Arabic dependency treebank: a word on the million words. In: Proceedings of the Workshop on Arabic and Local Languages (LREC 2008), Marrakech (2008)

    Google Scholar 

  24. Habash, N., Roth, R.M.: CATiB: The Columbia Arabic Treebank. In: Proceedings of the ACL-IJCNLP 2009 Conference Short Papers (2009)

    Google Scholar 

  25. Dukes, K., Buckwalter, T.: A dependency treebank of the Quran using traditional Arabic grammar. In: 2010 The 7th International Conference on Informatics and Systems (INFOS), Cairo (2010)

    Google Scholar 

  26. Pradhan, S.S., Hovy, E., Marcus, M., Palmer, M., Ramshaw, L., Weischedel, R.: Ontonotes: a unified relational semantic representation. Int. J. Semant. Comput. 1(04), 405–419 (2007)

    Article  Google Scholar 

  27. Bies, A.: catalog.ldc.upenn.edu, 28.1.2003. https://catalog.ldc.upenn.edu/docs/LDC2003T06/arabic-POStags-collapse-to-PennPOStags.txt. Accessed 6 June 2017

  28. Jaafar, Y., Bouzoubaa, K.: SAFAR: Software Architecture For Arabic language pRocessing. http://arabic.emi.ac.ma/safar/. Accessed 6 June 2017

  29. Souteh, Y., Bouzoubaa, K.: SAFAR platform and its morphological layer. In: Eleventh Conference on Language Engineering, ESOLEC 2011, Cairo (2011)

    Google Scholar 

  30. Jaafar, Y., Bouzoubaa, K.: Arabic natural language processing from software engineering to complex pipeline. In: First International Conference on Arabic Computational Linguistics (ACLing), Egypt, Cairo (2015)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Mohammadia School of Engineers, Mohammed Vth University, Rabat, Morocco

    Younes Jaafar & Karim Bouzoubaa

Authors
  1. Younes Jaafar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Karim Bouzoubaa
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Younes Jaafar .

Editor information

Editors and Affiliations

  1. Ex ENSA-USMBA, Fez, Morocco

    Abdelmonaime Lachkar

  2. EMI, UM5, Rabat, Morocco

    Karim Bouzoubaa

  3. FS, UMP, Oujda, Morocco

    Azzedine Mazroui

  4. IERA, UM5, Rabat, Morocco

    Abdelfettah Hamdani

  5. FS, UMP, Oujda, Morocco

    Abdelhak Lekhouaja

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer International Publishing AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Jaafar, Y., Bouzoubaa, K. (2018). A New Tool for Benchmarking and Assessing Arabic Syntactic Parsers. In: Lachkar, A., Bouzoubaa, K., Mazroui, A., Hamdani, A., Lekhouaja, A. (eds) Arabic Language Processing: From Theory to Practice. ICALP 2017. Communications in Computer and Information Science, vol 782. Springer, Cham. https://doi.org/10.1007/978-3-319-73500-9_17

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-73500-9_17

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-73499-6

  • Online ISBN: 978-3-319-73500-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation