Evaluation of Approaches for Automatic E-Assessment Item Annotation with Levels of Bloom’s Taxonomy

Abstract

The classification of e-assessment items with levels of Bloom’s taxonomy is an important aspect of effective e-assessment. Such annotations enable the automatic generation of parallel tests with the same competence profile as well as a competence-oriented analysis of the students’ exam results. Unfortunately, manual annotation by item creators is rarely done, either because the used e-learning systems do not provide the functionality or because teachers shy away from the manual workload. In this paper we present an approach for the automatic classification of items according to Bloom’s taxonomy and the results of their evaluation. We use natural language processing techniques for pre-processing from four different NLP libraries, calculate 19 item features with and without stemming and stop word removal, employ six classification algorithms and evaluate the results of all these factors by using two real world data sets. Our results show that 1) the selection of the classification algorithm and item features are most impactful on the F1 scores, 2) automatic classification can achieve F1 scores of up to 90% and is thus well suited for a recommender system supporting item creators, and 3) some algorithms and features are worth using and should be considered in future studies.

This work was supported by the German Federal Ministry of Education and Research for the tech4comp project under grant No 16DHB2102.