Skip to main content

Advertisement

SpringerLink
Log in

Towards detection of learner misconceptions in a medical learning environment: a subgroup discovery approach

  • Development Article
  • Published:
Educational Technology Research and Development Aims and scope Submit manuscript

Abstract

Ill-structured problems, by definition, have multiple paths to a solution and are multifaceted making automated assessment and feedback a difficult challenge. Diagnostic reasoning about medical cases meet the criteria of ill-structured problem solving since there are multiple solution paths. The goal of this study was to develop an adaptive feedback mechanism that is capable of identifying and responding to novice physician misconceptions by mining the log trace data of user interactions in BioWorld, a computer-based learning environment designed to support medical students in regulating their own diagnostic reasoning. We applied a series of text pre-processing algorithms to extract the linguistic features that characterized symptoms identified by 30 medical students solving three endocrinology cases: diabetes mellitus (type 1), Pheochromocytoma, and Hyperthyroidism. A subgroup discovery mining algorithm was applied in two steps. First, rules were induced to predict an incorrect diagnosis submitted as the final solution for a case on the basis of symptoms highlighted by medical students as being pertinent, that were in fact not pertinent. Second, rules were induced to predict a distractor hypothesis (an incorrect hypothesis listed as the most probable) during the differential diagnosis process while solving the case. The rule set discovered through the subgroup discovery task was shown to predict both incorrect and distractor hypotheses set by novice physicians while solving the case. We discuss the implications in terms of developing an adaptive feedback mechanism that can detect physicians’ misconceptions and errors during problem-solving as a means to deliver prompts and feedback that promote the acquisition of metacognitive monitoring and control skills.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1

Similar content being viewed by others

References

  • Aleven, V., Roll, I., McLaren, B. M., & Koedinger, K. R. (2016). Help helps, but only so much: Research on help seeking with intelligent tutoring systems. International Journal of Artificial Intelligence in Education, 26(1), 205–223.

    Article  Google Scholar 

  • Avella, J. T., Kebritchi, M., Nunn, S. G., & Kanai, T. (2016). Learning analytics methods, benefits, and challenges in higher education: A systematic literature review. Online Learning. https://doi.org/10.24059/olj.v20i2.790.

    Google Scholar 

  • Azevedo, R., & Aleven, V. (Eds.). (2013). International handbook of metacognition and learning technologies. Amsterdam: Springer.

    Google Scholar 

  • Azevedo, R., Harley, J., Trevors, G., Feyzi-Behnagh, R., Duffy, M., Bouchet, F., et al. (2013). Using trace data to examine the complex roles of cognitive, metacognitive, andemotional self-regulatory processes during learning with multi-agent systems. In R. Azevedo & V. Aleven (Eds.), International handbook of metacognition and learning technologies (pp. 427–449). Amsterdam: Springer.

    Chapter  Google Scholar 

  • Baker, R. S. (2014). Educational data mining: An advance for intelligent systems in education. IEEE Intelligent Systems, 29(3), 78–82.

    Article  Google Scholar 

  • Baker, R., Walonoski, J., Heffernan, N., Roll, I., Corbett, A., & Koedinger, K. (2008). Why students engage in “gaming the system” behavior in interactive learning environments. Journal of Interactive Learning Research, 19(2), 185–224.

    Google Scholar 

  • Baker, R. S. J. D., & Yacef, K. (2009). The state of educational data mining in 2009: A review and future visions. Journal of Educational Data Mining, 1(1), 3–17.

    Google Scholar 

  • Basu, S., Biswas, G., & Kinnebrew, J. S. (2017). Learner modeling for adaptive scaffolding in a computational thinking-based science learning environment. User Modeling and User-Adapted Interaction. https://doi.org/10.1007/s11257-017-9187-0.

    Google Scholar 

  • Berland, M., Baker, R., & Blikstein, P. (2014). Educational data mining and learning analytics: Applications to constructionist research. Technology, Knowledge and Learning, 19(1–2), 205–220.

    Article  Google Scholar 

  • Berner, E., & Graber, M. (2008). Overconfidence as a cause of diagnostic error in medicine. The American Journal of Medicine, 121(5), S2–S23. https://doi.org/10.1016/j.amjmed.2008.01.001.

    Article  Google Scholar 

  • Boekaerts, M. (1995). Self-regulated learning: Bridging the gap between metacognitive and metamotivation theories. Educational Psychologist, 30(4), 195–200. https://doi.org/10.1207/s15326985ep3004_4.

    Article  Google Scholar 

  • Bydžovská, H. & Popelínský, L. (2014). Course Recommendation from Social Data. In Proceedings of the 6th international conference on computer supported education, (pp. 268–275). Barcelona, Spain.

  • Cleary, T. J. (2011). Emergence of self-regulated learning microanalysis: Historical overview, essential features, and implications for research and practice. In B. J. Zimmerman & D. H. Schunk (Eds.), Handbook of self-regulation of learning and performance (pp. 329–345). New York: Routledge.

    Google Scholar 

  • Collins, A. (2006). Cognitive apprenticeship. In K. Sawyer (Ed.), Cambridge handbook of the learning sciences (pp. 47–60). NY: Cambridge University Press.

    Google Scholar 

  • Curtis, D. A., Lind, S. L., Dellinges, M., & Schroeder, K. (2012). Identifying student misconceptions in biomedical course assessments in dental education. Journal of Dental Education, 76(9), 1183–1194.

    Google Scholar 

  • Delany, C., & Golding, C. (2014). Teaching clinical reasoning by making thinking visible: An action research project with allied health clinical educators. BMC Medical Education, 14(1), 20. https://doi.org/10.1186/1472-6920-14-20.

    Article  Google Scholar 

  • Doleck, T., Basnet, R. B., Poitras, E. G., & Lajoie, S. P. (2015). Mining learner-system interaction data: Implications for modeling learner behaviors and improving overlay models. Journal of Computers in Education, 2(4), 421–447. https://doi.org/10.1007/s40692-015-0040-3.

    Article  Google Scholar 

  • Doleck, T., Jarrell, A., Poitras, E. G., Chaouachi, M., & Lajoie, S. (2016). A tale of three cases: Examining accuracy, efficiency, and process differences in diagnosing virtual patient cases. Australasian Journal of Educational Technology, 36(5), 61–76. https://doi.org/10.14742/ajet.2759.

    Google Scholar 

  • Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data. Cambridge, MA: MIT Press.

    Google Scholar 

  • Eva, K. (2005). What every teacher needs to know about clinical reasoning. Medical Education, 39(1), 98–106. https://doi.org/10.1111/j.1365-2929.2004.01972.x.

    Article  Google Scholar 

  • Friedman, C., Gatti, G., Franz, T., Murphy, G., Wolf, F., Heckerling, P., et al. (2005). Do physicians know when their diagnoses are correct? Journal of General Internal Medicine, 20(4), 334–339. https://doi.org/10.1111/j.1525-1497.2005.30145.x.

    Article  Google Scholar 

  • García, Á. M., Charte, F., González, P., Carmona, C. J., & del Jesus, M. J. (2016). Subgroup discovery with evolutionary fuzzy systems in R: the SDEFSR package. The R Journal, 8(2), 307–323.

    Google Scholar 

  • Graber, M., Franklin, N., & Gordon, R. (2005). Diagnostic error in internal medicine. Archives of Internal Medicine, 165(13), 1493. https://doi.org/10.1001/archinte.165.13.1493.

    Article  Google Scholar 

  • Graesser, A. C. (2016). Conversations with AutoTutor help students learn. International Journal of Artificial Intelligence in Education, 26, 124–132.

    Article  Google Scholar 

  • Greene, J., Bolick, C., Jackson, W., Caprino, A., Oswald, C., & McVea, M. (2015). Domain-specificity of self-regulated learning processing in science and history. Contemporary Educational Psychology, 42, 111–128. https://doi.org/10.1016/j.cedpsych.2015.06.001.

    Article  Google Scholar 

  • Herrera, F., Carmona, C. J., González, P., & del Jesus, M. J. (2011). An overview on subgroup discovery: Foundations and applications. Knowledge and Information Systems, 29(3), 495–525. https://doi.org/10.1007/s10115-010-0356-2.

    Article  Google Scholar 

  • Holzinger, A., Kickmeier-Rust, M. D., Wassertheurer, S., & Hessinger, M. (2009). Learning performance with interactive simulations in medical education: Lessons learned from results of learning complex physiological models with the HAEMOdynamics SIMulator. Computers & Education, 52(2), 292–301.

    Article  Google Scholar 

  • Holzinger, A., Valdez, A. C., & Ziefle, M. (2016). Towards Interactive Recommender Systems with the Doctor-in-the-Loop. Mensch und Computer 2016–Workshopband.

  • Klösgen, W. (2002). Types and forms of knowledge (patterns): Subgroup patterns. In W. Klösgen & J. Zytkow (Eds.), Handbook of data mining and knowledge discovery (pp. 47–51). New York: Oxford University Press.

    Google Scholar 

  • Koedinger, K. R., D’Mello, S., McLaughlin, E. A., Pardos, Z. A., & Rosé, C. P. (2015). Data mining and education. WIREs Cognitive Science, 6, 333–353. https://doi.org/10.1002/wcs.1350.

    Article  Google Scholar 

  • Lajoie, S. P. (2003). Transitions and trajectories for studies of expertise. Educational Researcher, 32(8), 21–25.

    Article  Google Scholar 

  • Lajoie, S. P. (2005). Cognitive tools for the mind: The Promises of technology—Cognitive amplifiers or bionic prosthetics? In R. J. Sternberg & D. Preiss (Eds.), Intelligence and technology: Impact of tools on the nature and development of human skills (pp. 87–102). Mahwah, NJ: Erlbaum.

    Google Scholar 

  • Lajoie, S. P. (2009). Developing professional expertise with a cognitive apprenticeship model: Examples from avionics and medicine. In K. A. Ericsson (Ed.), Development of professional expertise: Toward measurement of expert performance and design of optimal learning environments (pp. 61–83). Cambridge: Cambridge University Press.

    Chapter  Google Scholar 

  • Lajoie, S. P., Gauthier, G., & Lu, J. (2009). Convergence of data sources in the analysis of complex learning environments. Research and Practice in Technology Enhanced Learning, 4(3), 195–219.

    Article  Google Scholar 

  • Lajoie, S. P., Naismith, L., Poitras, E., Hong, Y.-J., Cruz-Panesso, I., Ranellucci, J., et al. (2013). Technology-rich tools to support self-regulated learning and performance in medicine. In R. Azevedo & V. Aleven (Eds.), International handbook of metacognition and learning technologies (pp. 229–242). New York: Springer.

    Chapter  Google Scholar 

  • Lajoie, S. P., & Poitras, E. G. (2017). Crossing disciplinary boundaries to improve technology-rich learning environments. Teachers College Record, 19(3), 1–30.

    Google Scholar 

  • Lemmerich, F., Ifland, M., & Puppe, F. (2011). Identifying influence factors on students success by subgroup discovery. In Proceedings of the 4th international conference on educational data mining, (pp. 345–346). International Educational Data Mining Society, Eindhoven.

  • Luz, M., de Oliveira, G., de Sousa, C., & Da Poian, A. (2008). Glucose as the sole metabolic fuel: The possible influence of formal teaching on the establishment of a misconception about energy-yielding metabolism among students from Rio de Janeiro, Brazil. Biochemistry And Molecular Biology Education, 36(6), 407–416. https://doi.org/10.1002/bmb.20235.

    Article  Google Scholar 

  • Midgley, C., Maehr, M. L., Hruda, L. Z., Anderman, E., Anderman, L., Freeman, K. E., et al. (2000). Manual for the patterns of adaptive learning scales (PALS). Ann Arbor, MI: University of Michigan.

    Google Scholar 

  • Mierswa, I., Wurst, M., Klinkenberg, R., Scholz, M., & Euler, T. (2006). YALE: Rapid prototyping for complex data mining tasks. In Proceedings of the 12th ACM SIGKDD international conference on knowledge discovery and data mining (pp. 935–940). New York: ACM Press.

  • Morton, J., Doran, D., & MacLaren, D. (2008). Common student misconceptions in exercise physiology and biochemistry. AJP Advances In Physiology Education, 32(2), 142–146. https://doi.org/10.1152/advan.00095.2007.

    Article  Google Scholar 

  • Naismith, L. (2013). Examining motivational and emotional influences on medical students' attention to feedback in a technology-rich environment for learning clinical reasoning (Doctoral Thesis). Retrieved from Digitool Library at McGill University Dissertations and Theses database.

  • Norman, G., & Eva, K. (2010). Diagnostic error and clinical reasoning. Medical Education, 44(1), 94–100. https://doi.org/10.1111/j.1365-2923.2009.03507.x.

    Article  Google Scholar 

  • Ocumpaugh, J., San Pedro, M. O., Lai, H., Baker, R. S., & Borgen, F. (2016). Middle school engagement with mathematics software and later interest and self-efficacy for STEM careers. Journal of Science Education and Technology, 25(6), 877–887. https://doi.org/10.1007/s10956-016-9637-1.

    Article  Google Scholar 

  • Olde Bekkink, M., Donders, A., Kooloos, J., de Waal, R., & Ruiter, D. (2016). Uncovering students’ misconceptions by assessment of their written questions. BMC Medical Education. https://doi.org/10.1186/s12909-016-0739-5.

    Google Scholar 

  • Panadero, E., & Järvelä, S. (2017). Models of self-regulated learning: Zimmerman, Boekaerts, Winne, Pintrich, Efklides, and Hadwin & Järvelä. A review in 2017. Frontiers. https://doi.org/10.3389/fpsyg.2017.00422.

    Google Scholar 

  • Papamitsiou, Z., & Economides, A. (2014). learning analytics and educational data mining in practice: A systematic literature review of empirical evidence. Educational Technology & Society, 17(4), 49–64.

    Google Scholar 

  • Pekrun, R., Goetz, T., Frenzel, A., Barchfeld, P., & Perry, R. (2011). Measuring emotions in students’ learning and performance: The Achievement Emotions Questionnaire (AEQ). Contemporary Educational Psychology, 36(1), 36–48. https://doi.org/10.1016/j.cedpsych.2010.10.002.

    Article  Google Scholar 

  • Pelaez, N. J., Boyd, D. D., Rojas, J. B., & Hoover, M. A. (2005). Prevalence of blood circulation misconceptions among prospective elementary teachers. AJP Advances In Physiology Education, 29(3), 172–181. https://doi.org/10.1152/advan.00022.2004.

    Article  Google Scholar 

  • Peña-Ayala, A. (2014). Educational data mining: A survey and a data mining-based analysis of recent works. Expert Systems with Applications, 41(4), 1432–1462. https://doi.org/10.1016/j.eswa.2013.08.042.

    Article  Google Scholar 

  • Perry, N., & Winne, P. (2006). Learning from learning kits: gStudy traces of students’ self-regulated engagements with computerized content. Educational Psychology Review, 18(3), 211–228. https://doi.org/10.1007/s10648-006-9014-3.

    Article  Google Scholar 

  • Pintrich, P. (2004). A conceptual framework for assessing motivation and self-regulated learning in college students. Educational Psychology Review, 16(4), 385–407. https://doi.org/10.1007/s10648-004-0006-x.

    Article  Google Scholar 

  • Poitras, E., & Lajoie, S. (2013). A domain-specific account of self-regulated learning: The cognitive and metacognitive activities involved in learning through historical inquiry. Metacognition And Learning, 8(3), 213–234. https://doi.org/10.1007/s11409-013-9104-9.

    Article  Google Scholar 

  • Poitras, E. G., Lajoie, S. P., Doleck, T., & Jarrell, A. (2016a). Subgroup discovery with user interaction data: An empirically guided approach to improving intelligent tutoring systems. Educational Technology & Society, 19(2), 204–214.

    Google Scholar 

  • Poitras, E. G., Lajoie, S. P., Jarrell, A., Doleck, T., & Naismith, L. (2016b). Intelligent tutoring systems in the medical domain: Fostering self-regulatory skills in problem-solving. In R. K. Atkinson (Ed.), Intelligent tutoring systems: Structure, applications, and challenges. Hauppauge, NY: Nova Publishing.

    Google Scholar 

  • Poitras, E. G., Naismith, L. M., Doleck, T., & Lajoie, S. P. (2016c). Using learning analytics to identify medical student misconceptions in an online virtual patient environment. Online Learning Journal, 20(2), 183–194.

    Google Scholar 

  • Puustinen, M., & Pulkkinen, L. (2001). Models of self-regulated learning: A review.Scandinavian. Journal of Educational Research, 45(3), 269–286. https://doi.org/10.1080/00313830120074206.

    Google Scholar 

  • Richardson, D., & Speck, D. (2004). Addressing students’ misconceptions of renal clearance. AJP Advances In Physiology Education, 28(4), 210–212. https://doi.org/10.1152/advan.00021.2004.

    Article  Google Scholar 

  • Romero, C., González, P., Ventura, S., del Jesus, M., & Herrera, F. (2009). Evolutionary algorithms for subgroup discovery in E-learning: A practical application using moodle data. Expert Systems with Applications, 36(2), 1632–1644.

    Article  Google Scholar 

  • Romero, C., & Ventura, S. (2010). Educational data mining: A review of the state of the art. IEEE Transactions on Systems, Man, and Cybernetics, Part C C (Applications and Reviews), 40(6), 601–618.

    Article  Google Scholar 

  • Romero, C., & Ventura, S. (2012). Data mining in education. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 3(1), 12–27. https://doi.org/10.1002/widm.1075.

    Google Scholar 

  • Scott, I. (2009). Errors in clinical reasoning: Causes and remedial strategies. BMJ, 338, 1860. https://doi.org/10.1136/bmj.b1860.

    Article  Google Scholar 

  • Sitzmann, T., & Ely, K. (2011). A meta-analysis of self-regulated learning in work-related training and educational attainment: What we know and where we need to go. Psychological Bulletin, 137(3), 421–442. https://doi.org/10.1037/a0022777.

    Article  Google Scholar 

  • Slater, S., Joksimovic, S., Kovanovic, V., Baker, R. S., & Gasevic, D. (2016). Tools for educational data mining: A review. Journal of Educational and Behavioral Statistics. https://doi.org/10.3102/1076998616666808.

    Google Scholar 

  • Taub, M., Azevedo, R., Bouchet, F., & Khosravifar, B. (2014). Can the use of cognitive and metacognitive self-regulated learning strategies be predicted by learners’ levels of prior knowledge in hypermedia-learning environments? Computers in Human Behavior, 39, 356–367. https://doi.org/10.1016/j.chb.2014.07.018.

    Article  Google Scholar 

  • Winne, P. (2005). A perspective on state-of-the-art research on self-regulated learning. Instructional Science, 33(5–6), 559–565. https://doi.org/10.1007/s11251-005-1280-9.

    Article  Google Scholar 

  • Winne, P. H. (2010). Improving measurements of self-regulated learning. Educational Psychologist, 45(4), 267–276. https://doi.org/10.1080/00461520.2010.517150.

    Article  Google Scholar 

  • Winne, P. H., & Baker, R. S. J. D. (2013). The potentials of educational data mining for researching metacognition, motivation, and self-regulated learning. Journal of Educational Data Mining, 5(1), 1–8.

    Google Scholar 

  • Wrobel, S. (1997). An Algorithm for multi-relational discovery of subgroups. In Proceedings of the first European symposium on principles of data mining and knowledge discovery (pp. 78–87). New York: Springer.

  • Zimmerman, B. (1990). Self-regulated learning and academic achievement: An overview. Educational Psychologist, 25(1), 3–17. https://doi.org/10.1207/s15326985ep2501_2.

    Article  Google Scholar 

  • Zimmerman, B. (2002). Becoming a self-regulated learner: An overview. Theory Into Practice, 41(2), 64–70. https://doi.org/10.1207/s15430421tip4102_2.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Utah, Salt Lake City, UT, USA

    Eric G. Poitras

  2. McGill University, Montreal, Canada

    Tenzin Doleck & Susanne P. Lajoie

Authors
  1. Eric G. Poitras
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tenzin Doleck
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Susanne P. Lajoie
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Eric G. Poitras.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

Ethics approved by the McGill Research Ethics Board and informed consent was obtained from all participants included in the study.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Poitras, E.G., Doleck, T. & Lajoie, S.P. Towards detection of learner misconceptions in a medical learning environment: a subgroup discovery approach. Education Tech Research Dev 66, 129–145 (2018). https://doi.org/10.1007/s11423-017-9555-9

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11423-017-9555-9

Keywords

Search

Navigation