Skip to main content
SpringerLink
Log in

A Layered Approach to Automatic Essay Evaluation Using Word-Embedding

  • Conference paper
  • First Online:
Computer Supported Education (CSEDU 2018)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1022))

Included in the following conference series:

Abstract

Automated Essay Evaluation (AEE) use a set of features to evaluate and score students essay solutions. Most of the features like lexical similarity, syntax, vocabulary and shallow content were addressed but evaluating students essays using the semantics and context of the essay are not addressed well. To address the issue which are related to the semantics and context, we propose a layered approach to AEE which uses neural word embedding in order to evaluate student answers semantically and the similarity will be computed by using Word Mover’s Distance. We also implemented a plagiarism detection algorithms to protect the students from submitting someone else solution as their own using k-shingles and local sensitive hashing. We also implemented an algorithm that penalize students who are trying to fool the system by submitting only content bearing works. The performance of the proposed AEE was evaluated and compared to other state-of-the-art methods qualitatively and quantitatively. The experimental results show that the proposed AEE approach using neural word embedding achieve higher level of accuracy as compared to others baselines and are promising in evaluating students essay solutions semantically.

Tomáš Horváth is also associated with the Institute of Computer Science of the Faculty of Science at the Pavol Jozef Šafárik University in Košice, Slovakia.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://www.kaggle.com/c/asap-sas.

  2. 2.

    https://code.google.com/archive/p/word2vec/.

  3. 3.

    http://scikit-learn.org/.

  4. 4.

    http://www.numpy.org/.

References

  1. Miller, M.D., Linn, R.L., Gronlund, N.E.: Measurement and Assessment in Teaching, 11th edn. Pearson, London (2013)

    Google Scholar 

  2. Page, E.B.: Grading essays by computer: progress report. In: Invitational Conference on Testing Problems (1966)

    Google Scholar 

  3. Deerwester, S., Dumais, S.T., Furnas, G.W., Landauer, T.K., Harshman, R.: Indexing by latent semantic analysis. J. Am. Soc. Inf. Sci. 41, 391–407 (1999)

    Article  Google Scholar 

  4. Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent dirichlet allocation. J. Mach. Learn. Res. 3, 993–1022 (2003)

    MATH  Google Scholar 

  5. Attali, Y.: A differential word use measure for content analysis in automated essay scoring. ETS Res. Rep. Ser. 36, i–19 (2011)

    Google Scholar 

  6. Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Advances in Neural Information Processing Systems, vol. 26, pp. 3111–3119 (2013)

    Google Scholar 

  7. Kusner, M.J., Sun, Y., Kolkin, N.I., Weinberger, K.Q.: From word embeddings to document distances. In: International Conference on Machine Learning, vol. 37, pp. 957–966 (2015)

    Google Scholar 

  8. Bengio, Y., Ducharme, R., Vincent, P., Janvin, C.: A neural probabilistic language model. J. Mach. Learn. Res. 3, 1137–1155 (2003)

    MATH  Google Scholar 

  9. Li, Y., Xu, L., Tian, F., Jiang, L., Zhong, X., Chen, E.: Word embedding revisited: a new representation learning and explicit matrix factorization perspective. In: IJCAI International Joint Conference on Artificial Intelligence, pp. 3650–3656 (2015)

    Google Scholar 

  10. Tashu, T.M., Horváth, T.: Pair-wise: automatic essay evaluation using word mover’s distance. In: Proceedings of the 10th International Conference on Computer Supported Education, CSEDU, INSTICC, vol. 2, pp. 59–66. SciTePress (2018)

    Google Scholar 

  11. Shermis, M.D., Koch, C.M., Page, E.B., Keith, T.Z., Harrington, S.: Trait ratings for automated essay grading. Educ. Psychol. Measur. 62, 5–18 (2002)

    Article  MathSciNet  Google Scholar 

  12. Wang, X.B.: J. Educ. Behav. Stat. 30 (2005)

    Google Scholar 

  13. Zhang, L.: Review of handbook of automated essay evaluation: Current applications and new directions. Lang. Learn. Technol. 18, 65–69 (2014)

    Google Scholar 

  14. Ben-Simon, A., Bennett, R.E.: Toward more substantively meaningful automated essay scoring. J. Technol. Learn. Assess. 6(1) (2007)

    Google Scholar 

  15. Attali, Y., Burstein, J.: Automated essay scoring with e-rater® V.2. J. Technol. Learn. Assess. 4 (2006)

    Google Scholar 

  16. Cutrone, L., Chang, M.: Kinshuk: auto-assessor: computerized assessment system for marking student’s short-answers automatically. In: Proceedings of the IEEE International Conference on Technology for Education, pp. 81–88 (2011). https://doi.org/10.1109/T4E.2011.21

  17. Foltz, P.W., Laham, D., Landauer, T.K.: Automated essay scoring: applications to educational technology. In: World Conference on Educational Multimedia, Hypermedia and Telecommunications (ED-MEDIA) (1999)

    Google Scholar 

  18. Islam, M., Hoque, A.S.M.L.: Automated essay scoring using generalized latent semantic analysis. In: IEEE 13th International Conference on Computer and Information Technology, vol. 7, pp. 616–626 (2012)

    Google Scholar 

  19. Shermis, M.D., Burstein, J.: Automated essay scoring a cross-disciplinary perspective. Br. J. Math. Stat. Psychol. (2003)

    Google Scholar 

  20. Jin, C., He, B.: Utilizing latent semantic word representations for automated essay scoring. In: 12th International Conference on Ubiquitous Intelligence and Computing and IEEE 12th International Conference on Autonomic and Trusted Computing and IEEE 15th International Conference on Scalable Computing and Communications and Its Associated Workshops (UIC-ATC-ScalCom) (2015)

    Google Scholar 

  21. Alikaniotis, D., Yannakoudakis, H., Rei, M.: Automatic text scoring using neural networks. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pp. 715–725 (2016)

    Google Scholar 

  22. Taghipour, K., Ng, H.T.: A neural approach to automated essay scoring. In: Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1882–1891. Association for Computational Linguistics (2016)

    Google Scholar 

  23. Jin, C., He, B., Xu, J.: A study of distributed semantic representations for automated essay scoring. In: Li, G., Ge, Y., Zhang, Z., Jin, Z., Blumenstein, M. (eds.) KSEM 2017. LNCS (LNAI), vol. 10412, pp. 16–28. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-63558-3_2

    Chapter  Google Scholar 

  24. Thanawala, P., Pareek, J., Shah, M.: OntoBAeval: ontology-based automatic evaluation of free-text response. In: 2014 IEEE Sixth International Conference on Technology for Education (2014)

    Google Scholar 

  25. Fauzi, M.A., Utomo, D.C., Setiawan, B.D., Pramukantoro, E.S.: Automatic essay scoring system using N-gram and cosine similarity for gamification based E-learning. In: Proceedings of the International Conference on Advances in Image Processing, ICAIP 2017, pp. 151–155. ACM, New York (2017)

    Google Scholar 

  26. Zupanc, K., Bosnifć, Z.: Automated essay evaluation with semantic analysis. Knowl.-Based Syst. 120, 118–132 (2017)

    Article  Google Scholar 

  27. Yamamoto, M., Umemura, N., Kawano, H.: Automated essay scoring system based on rubric. In: Lee, R. (ed.) ACIT 2017. SCI, vol. 727, pp. 177–190. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-64051-8_11

    Chapter  Google Scholar 

  28. Dumais, T.K., Landauer, S.: Latent semantic analysis. Scholarpedia 3(11), 4356 (2008)

    Article  Google Scholar 

  29. Salton, G.: Automatic Text Processing: The Transformation, Analysis, and Retrieval of Information by Computer. Addison-Wesley Longman Publishing Co., Inc., Boston (1989)

    Google Scholar 

  30. Porter, M.: The Porter Stemming Algorithm (1980)

    Google Scholar 

  31. Rajaraman, A., Ullman, J.D.: Mining of Massive Datasets. Cambridge University Press, New York (2011)

    Book  Google Scholar 

  32. Islam, M., Latiful Hoque, A.S.M.: Automated essay scoring using generalized latent semantic analysis. In: International Conference on Computer and Information Technology (2010)

    Google Scholar 

  33. Atoum, I., Otoom, A.: Efficient hybrid semantic text similarity using wordnet and a corpus. Int. J. Adv. Comput. Sci. Appl. (IJACSA) 7, 124–130 (2016)

    Google Scholar 

  34. Wan, S., Angryk, R.A.: Measuring semantic similarity using WordNet-based context vectors. In: IEEE International Conference on Systems, Man and Cybernetics (2007)

    Google Scholar 

  35. Zhuge, W., Hua, J.: WordNet-based way to identify Chinglish in automated essay scoring systems. In: International Symposium on Knowledge Acquisition and Modeling (2009)

    Google Scholar 

  36. Ewees, A.A., Eisa, M., Refaat, M.M.: Comparison of cosine similarity and k-NN automated essays scoring. Int. J. Adv. Res. Comput. Commun. Eng. 3 (2014)

    Google Scholar 

  37. Xia, P., Zhang, L., Li, F.: Learning similarity with cosine similarity ensemble. Inf. Sci. 307, 39–52 (2015)

    Article  MathSciNet  Google Scholar 

  38. Williamson, D.: A framework for implementing automated scoring. In: The Annual Meeting of the American Educational Research Association (AERA) and the National Council on Measurement in Education (NCME) (2009)

    Google Scholar 

  39. Clough, P., Stevenson, M.: Developing a corpus of plagiarised short answers. Lang. Resour. Eval. 45, 5–24 (2011)

    Article  Google Scholar 

  40. Powers, D.M.W.: Evaluation: from precision, recall and F-measure to ROC, informedness, markedness & correlation. J. Mach. Learn. Technol. 2, 37–63 (2011)

    Google Scholar 

Download references

Acknowledgements

The research has been supported by the European Union, co- financed by the European Social Fund (EFOP-3.6.2-16-2017-00013).

Supported by Telekom Innovation Laboratories (T-Labs), the Research and Development unit of Deutsche Telekom.

Author information

Authors and Affiliations

  1. Faculty of Informatics, Department of Data Science and Engineering (Telekom Innovation Laboratories), ELTE – Eötvös Loránd University, Pázmány Péter Sétány 1/C, Budapest, 1117, Hungary

    Tsegaye Misikir Tashu & Tomáš Horváth

Authors
  1. Tsegaye Misikir Tashu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tomáš Horváth
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tsegaye Misikir Tashu .

Editor information

Editors and Affiliations

  1. Carnegie Mellon University, Pittsburgh, PA, USA

    Bruce M. McLaren

  2. MIT, Cambridge, MA, USA

    Rob Reilly

  3. University of Denver, Denver, CO, USA

    Susan Zvacek

  4. University of Ulster, Newtownabbey, UK

    James Uhomoibhi

Rights and permissions

Reprints and permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Tashu, T.M., Horváth, T. (2019). A Layered Approach to Automatic Essay Evaluation Using Word-Embedding. In: McLaren, B., Reilly, R., Zvacek, S., Uhomoibhi, J. (eds) Computer Supported Education. CSEDU 2018. Communications in Computer and Information Science, vol 1022. Springer, Cham. https://doi.org/10.1007/978-3-030-21151-6_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-21151-6_5

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-21150-9

  • Online ISBN: 978-3-030-21151-6

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation