Skip to main content
SpringerLink
Log in

Video-based learners’ observed attention estimates for lecture learning gain evaluation

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

A significant problem in the field of higher education is maintaining learners’ attention during lectures, which is known to significantly affect their learning outcomes. Attention management is commonly associated with the individual ability of a lecturer to track and respond to the common behaviour of an auditorium; it lacks a detailed estimation of the intra-variability of individual learners’ attention during the course of the lecture. This paper suggests an objective and non-intrusive evaluation of learners’ attention against learning outcomes by introducing an observed attention estimate (OAE). The procedure uses human annotations based on visual cues with a supporting video recording/playback system and a web-based annotation system. This proposed procedure enables us to estimate the attention level of individual learners as observed by human annotators for given time intervals associated with specific concepts covered by the lecture. As part of the procedure, we use an inventory-based lecture gain evaluation and representation based on a novel learning gain matrix. This procedure allows for a detailed analysis of the lecture time flow with regard to learners’ attention. We have verified the applicability of the procedure on a small-scale case study.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Angelo, T. A., & Cross, K. P. (1993). Classroom Assessment Techniques: A Handbook for College Teachers. ERIC. doi:https://doi.org/10.2307/2943957

  2. Asteriadis S, Tzouveli P, Karpouzis K, Kollias S (2009) Estimation of behavioral user state based on eye gaze and head pose---application in an e-learning environment. Multimed Tools Appl 41:469–493. https://doi.org/10.1007/s11042-008-0240-1

    Article  Google Scholar 

  3. Bao L (2006) Theoretical comparisons of average normalized gain calculations. Am J Phys 74:917–922. https://doi.org/10.1119/1.2213632

    Article  Google Scholar 

  4. Bligh, D. A. (1998). What's the Use of Lectures? Intellect (UK) doi:https://doi.org/10.1080/03098268508708932

  5. Cain J, Black EP, Rohr J (2009) An Audience Response System Strategy to Improve Student Motivation, Attention, and Feedback. Am J Pharm Educ 73:21. https://doi.org/10.5688/aj730221

    Article  Google Scholar 

  6. Chen C-M, Wang J-Y, Yu C-M (2015) Assessing the Attention Levels of Students by Using a Novel Attention Aware System based on Brainwave Signals. (T. Matsuo, K. Hashimoto, T. Mine, & S. Hirokawa, Eds.) Br J Educ Technol:379–384. https://doi.org/10.1109/IIAI-AAI.2015.224

  7. D'Mello SK (2016) On the influence of an iterative affect annotation approach on interobserver and self-observer reliability. IEEE Trans Affect Comput 7(2):136–149

    Article  Google Scholar 

  8. von Eye A, Mun EY (2004) Analyzing rater agreement: Manifest variable methods. Lawrence Erlbaum Associates Publishers. doi:https://doi.org/10.4324/9781410611024

  9. Giannopoulos I, Schöning J, Krüger A, Raubal M (2016) Attention as an input modality for Post-WIMP interfaces using the viGaze eye tracking framework. Multimed Tools Appl 75:2913–2929. https://doi.org/10.1007/s11042-014-2412-5

    Article  Google Scholar 

  10. Gibbs G (2011) Student attention over an hour. https://epigeum.com/downloads/uct_accessible/uk/01_lecturing1/html/course_files/2_30.html. Accessed 27 February 2017

  11. Graesser A, McDaniel B, Chipman P, Witherspoon A, D'Mello SK, Gholson B (2006) Detection of emotions during learning with AutoTutor. In: Sun R, Miyake N (eds) Proceedings of the 28th Annual Conference of the Cognitive Science Society. Cognitive Science Society, Austin, pp 285–290

    Google Scholar 

  12. Hake RR (1998) Interactive-engagement versus traditional methods: A six-thousand-student survey of mechanics test data for introductory physics courses. Am J Phys 66:64–74

    Article  Google Scholar 

  13. Hallgren KA (2012) Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial. Tutor Quant Methods Psychol 8:23–34. 10.20982/tqmp.08.1.p023

    Article  Google Scholar 

  14. Halloun IA, Hestenes D (1985) Common sense concepts about motion. Am J Phys 53:1056–1065. https://doi.org/10.1119/1.14031

    Article  Google Scholar 

  15. Hussey T, Smith P (2003) The Uses of Learning Outcomes. Teach High Educ 8:357–368. https://doi.org/10.1080/13562510309399

    Article  Google Scholar 

  16. Johnstone AH, Percival F (1976) Attention Breaks in Lectures. Educ Chem 13:49–50

    Google Scholar 

  17. Kondermann, D. (2013). Ground Truth Design Principles: An Overview. Proceedings of the International Workshop on Video and Image Ground Truth in Computer Vision Applications (pp. 5:1--5:4). New York: Association for Computing Machinery (ACM). doi:https://doi.org/10.1145/2501105.2501114

  18. Krippendorff K (2004) Content Analysis: An Introduction to Its Methodology. 2nd Ed. SAGE Publications, London pp 413

  19. Malle BF, Pearce GE (2001) Attention to behavioral events during interaction: Two actor-observer gaps and three attempts to close them. J Pers Soc Psychol 81(2):278–294

    Article  Google Scholar 

  20. Martinez HP, Yannakakis GN, Hallam J (2014) Don't Classify Ratings of Affect; Rank Them. IEEE Trans Affect Comput 5:314–326. https://doi.org/10.1109/taffc.2014.2352268

    Article  Google Scholar 

  21. Marx JD, Cummings K (2007) Normalized change. Am J Phys 75:87–91. https://doi.org/10.1119/1.2372468

    Article  Google Scholar 

  22. Matheson C (2008) The educational value and effectiveness of lectures. Clin Teach 5:218–221. https://doi.org/10.1111/j.1743-498X.2008.00238.x

    Article  Google Scholar 

  23. Wilcox RR (2003) Applying Contemporary Statistical Techniques. Academic Press, London, pp 608

  24. Perrenet JC, Bouhuijs PA, Smits JG (2000) The Suitability of Problem-based Learning for Engineering Education: Theory and practice. Teach High Educ 5:345–358. https://doi.org/10.1080/713699144

    Article  Google Scholar 

  25. Porta, M., Ricotti, S., & Perez, C. J. (2012). Emotional e-learning through eye tracking. Proceedings of the 2012 I.E. Global Engineering Education Conference (EDUCON). Institute of Electrical and Electronics Engineers (IEEE). doi:https://doi.org/10.1109/educon.2012.6201145

  26. Raykar VC, Yu S, Zhao LH, Valadez GH, Florin C, Bogoni L, Moy L (2010) Learning From Crowds. J Mach Learn Res 11:1297–1322

    MathSciNet  Google Scholar 

  27. Ried LD (2011) A Model for Curricular Quality Assessment and Improvement. Am J Pharm Educ 75:196. https://doi.org/10.5688/ajpe7510196

    Article  Google Scholar 

  28. Risko EF, Anderson N, Sarwal A, Engelhardt M, Kingstone A (2012) Everyday Attention: Variation in Mind Wandering and Memory in a Lecture. Appl Cogn Psychol 26:234–242. https://doi.org/10.1002/acp.1814

    Article  Google Scholar 

  29. Ruhl KL, Hughes CA, Schloss PJ (1987) Using the pause procedure to enhance lecture recall. TESE: J TED CEC 10:14–18. https://doi.org/10.1177/088840648701000103

    Google Scholar 

  30. Russell BC, Torralba A, Murphy KP, Freeman WT (2007) LabelMe: A Database and Web-Based Tool for Image Annotation. Int J Comput Vis 77:157–173. https://doi.org/10.1007/s11263-007-0090-8

    Article  Google Scholar 

  31. Tkalčič M, Odić A, Košir A (2013) The impact of weak ground truth and facial expressiveness on affect detection accuracy from time-continuous videos of facial expressions. Inf Sci 249:13–23. https://doi.org/10.1016/j.ins.2013.06.006

    Article  Google Scholar 

  32. Wage KE, Buck JR, Wright CH, Welch TB (2005) The Signals and Systems Concept Inventory. IEEE Trans Educ 48:448–461. https://doi.org/10.1109/te.2005.849746

    Article  Google Scholar 

  33. Wilson K, Korn JH (2007) Attention During Lectures: Beyond Ten Minutes. Teach Psychol 34:85–89. https://doi.org/10.1080/00986280701291291

    Article  Google Scholar 

  34. Yannakakis, G. N., & Martinez, H. P. (2015). Grounding truth via ordinal annotation. 2015 International Conference on Affective Computing and Intelligent Interaction (ACII) (pp. 574-580). Institute of Electrical and Electronics Engineers (IEEE). doi:https://doi.org/10.1109/acii.2015.7344627

  35. Young MS, Robinson S, Alberts P (2009) Students pay attention!: Combating the vigilance decrement to improve learning during lectures. Act Learn High Educ 10:41–55. https://doi.org/10.1177/1469787408100194

    Article  Google Scholar 

Download references

Acknowledgments

The authors would like to thank the students in the digital signal processing course at the University of Ljubljana for taking part in the study. Special thanks goes to John R. Buck and Kathleen E. Wage for giving us access to the SSCI used in this study.

Funding

This study was partially funded by Slovenian Research Agency (P2–0246 B).

Author information

Authors and Affiliations

  1. Faculty of Electrical Engineering, University of Ljubljana, Tržaška 25, 1000, Ljubljana, Slovenia

    Urban Burnik, Janez Zaletelj & Andrej Košir

Authors
  1. Urban Burnik
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Janez Zaletelj
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Andrej Košir
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Urban Burnik.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Burnik, U., Zaletelj, J. & Košir, A. Video-based learners’ observed attention estimates for lecture learning gain evaluation. Multimed Tools Appl 77, 16903–16926 (2018). https://doi.org/10.1007/s11042-017-5259-8

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-017-5259-8

Keywords

Search

Navigation