Skip to main content
SpringerLink
Log in

Evaluation Strategies for Cybersecurity Training Methods: A Literature Review

  • Conference paper
  • First Online:
Human Aspects of Information Security and Assurance (HAISA 2021)

Part of the book series: IFIP Advances in Information and Communication Technology ((IFIPAICT,volume 613))

Abstract

The human aspect of cybersecurity continues to present challenges to researchers and practitioners worldwide. While measures are being taken to improve the situation, a vast majority of security incidents can be attributed to user behavior. Security and Awareness Training (SAT) has been available for several decades and is commonly given as a suggestion for improving the cybersecurity behavior of end-users. However, attackers continue to exploit the human factor suggesting that current SAT methods are not enough. Researchers argue that providing knowledge alone is not enough, and some researchers suggest that many currently used SAT methods are, in fact, not empirically evaluated. This paper aims to examine how SAT has been evaluated in recent research using a structured literature review. The result is an overview of evaluation methods which describes what results that can be obtained using them. The study further suggests that SAT methods should be evaluated using a variety of methods since different methods will inevitably provide different results. The presented results can be used as a guide for future research projects seeking to develop or evaluate methods for SAT.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 109.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Al-Daeef, M.M., Basir, N., Saudi, M.M.: Security awareness training: a review. Proc. World Congress Eng. 1, 5–7 (2017)

    Google Scholar 

  2. Alshaikh, M., Maynard, S.B., Ahmad, A., Chang, S.: An exploratory study of current information security training and awareness practices in organizations. In: Proceedings of the 51st Hawaii International Conference on System Sciences (2018)

    Google Scholar 

  3. Ayyagari, R., Figueroa, N.: Is seeing believing? training users on information security: evidence from java applets. J. Inf. Syst. Educ. 28(2), 115–120 (2017)

    Google Scholar 

  4. Boss, S., Galletta, D., Lowry, P.B., Moody, G.D., Polak, P.: What do systems users have to fear? using fear appeals to engender threats and fear that motivate protective security behaviors. MIS Q. (MISQ) 39(4), 837–864 (2015)

    Article  Google Scholar 

  5. Braun, V., Clarke, V.: Using thematic analysis in psychology. Qualitative Res. Psychol. 3(2), 77–101 (2006)

    Article  Google Scholar 

  6. Bulgurcu, B., Cavusoglu, H., Benbasat, I.: Information security policy compliance: an empirical study of rationality-based beliefs and information security awareness. MIS Quarterly 34(3), 523–548 (2010)

    Article  Google Scholar 

  7. Burris, J., Deneke, W., Maulding, B.: Activity simulation for experiential learning in cybersecurity workforce development. In: Nah, F.F.-H., Xiao, B.S. (eds.) HCIBGO 2018. LNCS, vol. 10923, pp. 17–25. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-91716-0_2

    Chapter  Google Scholar 

  8. Campbell, D.T.: Factors relevant to the validity of experiments in social settings. Psychol. Bull. 54(4), 297 (1957)

    Article  Google Scholar 

  9. Choi, K.H., Lee, D.H.: A study on strengthening security awareness programs based on an rfid access control system for inside information leakage prevention. Multimed. Tools Appl. 74(20), 8927–8937

    Google Scholar 

  10. Cole, J.R., Pence, T., Cummings, J., Baker, E.: Gamifying security awareness: a new prototype. In: Moallem, A. (ed.) HCII 2019. LNCS, vol. 11594, pp. 115–133. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-22351-9_8

    Chapter  Google Scholar 

  11. Cuchta, T., et al.: Human risk factors in cybersecurity, pp. 87–92

    Google Scholar 

  12. Cybint: (2020) https://www.cybintsolutions.com/cyber-security-facts-stats/

  13. Desman, M.B.: The ten commandments of information security awareness training. Inf. Secur. J. A Glob. Perspect. 11(6), 39–44 (2003)

    Google Scholar 

  14. Dincelli, E., Chengalur-Smith, I.: Choose your own training adventure: designing a gamified seta artefact for improving information security and privacy through interactive storytelling. European Journal of Information Systems

    Google Scholar 

  15. EC-Council: (2019). https://blog.eccouncil.org/the-top-types-of-cybersecurity-attacks-of-2019-till-date/

  16. Eck, J.E., Liu, L.: Contrasting simulated and empirical experiments in crime prevention. J. Exp. Criminol. 4(3), 195–213 (2008)

    Article  Google Scholar 

  17. Gjertsen, E.G.B., Gjaere, E.A., Bartnes, M., Flores, W.R.: Gamification of Information Security Awareness and Training. Icissp (2017)

    Google Scholar 

  18. Gokul, C.J., Pandit, S., Vaddepalli, S., Tupsamudre, H., Banahatti, V., Lodha, S., Acm: PHISHY - a serious game to train enterprise users on phishing awareness. In: Proceedings of the 2018 Annual Symposium on Computer-Human Interaction in Play Companion Extended Abstracts (2018)

    Google Scholar 

  19. Gundu, T.: Acknowledging and Reducing the Knowing and Doing gap in Employee Cybersecurity Compliance, pp. 94–102. International Conference on Cyber Warfare and Security (2019)

    Google Scholar 

  20. Huynh, D., Luong, P., Iida, H., Beuran, R.: Design and evaluation of a cybersecurity awareness training game. In: Munekata, N., Kunita, I., Hoshino, J. (eds.) ICEC 2017. LNCS, vol. 10507, pp. 183–188. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-66715-7_19

    Chapter  Google Scholar 

  21. Jayakrishnan, G.C., Sirigireddy, G.R., Vaddepalli, S., Banahatti, V., Lodha, S.P., Pandit, S.S.: Passworld: a serious game to promote password awareness and diversity in an enterprise. In: (SOUPS 2020), pp. 1–18 (2020)

    Google Scholar 

  22. Jesson, J., Matheson, L., Lacey, F.M.: Doing your literature review: Traditional and systematic techniques. Sage (2011)

    Google Scholar 

  23. Joinson, A., van Steen, T.: Human aspects of cyber security: behaviour or culture change? Cyber Secur. Peer-Reviewed J. 1(4), 351–360 (2018)

    Google Scholar 

  24. Kunz, A., Volkamer, M., Stockhardt, S., Palberg, S., Lottermann, T., Piegert, E.: Nophish: evaluation of a web application that teaches people being aware of phishing attacks, vol. P-259, pp. 509–518 (2016)

    Google Scholar 

  25. Lastdrager, E., Gallardo, I.C., Hartel, P., Junger, M.: How effective is anti-phishing training for children? pp. 229–239 (2017)

    Google Scholar 

  26. Lim, I.K., Park, Y.G., Lee, J.K.: Design of security training system for individual users. Wirel. Personal Commun. 90(3), 1105–1120 (2016)

    Google Scholar 

  27. Meline, T.: Selecting studies for systematic review: inclusion and exclusion criteria. Contemporary Issues in Communication Science and Disorders 33(21–27) (2006)

    Google Scholar 

  28. Micallef, N., Arachchilage, N.A.G.: Involving users in the design of a serious game for security questions education. arXiv preprint arXiv:1710.03888 (2017)

  29. Moreno-Fernández, M.M., Blanco, F., Garaizar, P., Matute, H.: Fishing for phishers. improving internet users’ sensitivity to visual deception cues to prevent electronic fraud. Comput. Hum. Behav. 69, 421–436 (2017)

    Google Scholar 

  30. Paré, G., Kitsiou, S.: Methods for literature reviews. In: Handbook of eHealth Evaluation: An Evidence-based Approach [Internet]. University of Victoria (2017)

    Google Scholar 

  31. Parsons, K., Butavicius, M.A., Lillie, M., Calic, D., McCormac, A., Pattinson, M.R.: Which individual, cultural, organisational and interventional factors explain phishing resilience? In: HAISA, pp. 1–11 (2018)

    Google Scholar 

  32. Puhakainen, P., Siponen, M.: Improving employees’ compliance through information systems security training: an action research study. MIS quarterly, pp. 757–778 (2010)

    Google Scholar 

  33. Rastenis, J., Ramanauskaitė, S., Janulevičius, J., Čenys, A.: Impact of information security training on recognition of phishing attacks: A case study of vilnius gediminas technical university. vol. 1243. CCIS, pp. 311–324

    Google Scholar 

  34. Reinheimer, B., et al.: An investigation of phishing awareness and education over time: when and how to best remind users. In: (SOUPS 2020), pp. 259–284 (2020)

    Google Scholar 

  35. Renaud, K., Zimmermann, V.: Ethical guidelines for nudging in information security & privacy. Int. J. Hum. Comput. Stud. 120, 22–35 (2018)

    Article  Google Scholar 

  36. Safa, N.S., Von Solms, R.: An information security knowledge sharing model in organizations. Comput. Hum. Behav. 57, 442–451 (2016)

    Article  Google Scholar 

  37. Silic, M., Lowry, P.B.: Using design-science based gamification to improve organizational security training and compliance. J. Manage. Inf. Syst. 37(1), 129–161 (2020)

    Google Scholar 

  38. Siponen, M.T.: A conceptual foundation for organizational information security awareness. Information Management & Computer Security (2000)

    Google Scholar 

  39. Soare, B.: (2020). https://heimdalsecurity.com/blog/vectors-of-attack/

  40. Stockhardt, Simon, et al.: Teaching phishing-security: which way is best? In: Hoepman, Jaap-Henk., Katzenbeisser, Stefan (eds.) SEC 2016. IAICT, vol. 471, pp. 135–149. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-33630-5_10

    Chapter  Google Scholar 

  41. Takata, T., Ogura, K., IEEE: Confront Phishing Attacks - from a Perspective of Security Education, pp. 10–13. International Conference on Awareness Science and Technology (2019)

    Google Scholar 

  42. Taneski, V., Heričko, M., Brumen, B.: Impact of security education on password change, pp. 1350–1355 (2015)

    Google Scholar 

  43. Tichy, W.F.: Should computer scientists experiment more? Computer 31(5), 32–40 (1998)

    Article  Google Scholar 

  44. Tschakert, K.F., Ngamsuriyaroj, S.: Effectiveness of and user preferences for security awareness training methodologies. Heliyon 5(6), e02010 (2019)

    Google Scholar 

  45. Van Rensburg, W.J., Thomson, K.L., Futcher, L.: An educational intervention towards safe smartphone usage. In: HAISA 2018 (2018)

    Google Scholar 

  46. Vroom, C., Von Solms, R.: Towards information security behavioural compliance. Comput. Secur. 23(3), 191–198 (2004)

    Article  Google Scholar 

  47. Wen, Z.A., Lin, Z.Q., Chen, R., Andersen, E.: What. Hack: engaging anti-phishing training through a role-playing phishing simulation game. In: Chi 2019 (2019)

    Google Scholar 

  48. Wohlin, C., Runeson, P., Höst, M., Ohlsson, M.C., Regnell, B., Wesslén, A.: Experimentation in software engineering. Springer Science & Business Media (2012)

    Google Scholar 

  49. Xiong, A.P., Proctor, R.W., Yang, W.N., Li, N.H.: Embedding training within warnings improves skills of identifying phishing webpages. Human Factors 61(4), 577–595 (2019)

    Google Scholar 

  50. Yang, W., Xiong, A., Chen, J., Proctor, R.W., Li, N.: Use of phishing training to improve security warning compliance: Evidence from a field experiment. vol. Part F127186, pp. 52–61 (2017)

    Google Scholar 

  51. Zhou, L.M., Parmanto, B., Alfikri, Z., Bao, J.: A mobile app for assisting users to make informed selections in security settings for protecting personal health data: Development and feasibility study. Jmir Mhealth and Uhealth 6(12), e11210 (2018)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Skövde, Skövde, Sweden

    Joakim Kävrestad & Marcus Nohlberg

Authors
  1. Joakim Kävrestad
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Marcus Nohlberg
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Joakim Kävrestad .

Editor information

Editors and Affiliations

  1. University of Nottingham, Nottingham, UK

    Steven Furnell

  2. University of Plymouth, Plymouth, UK

    Nathan Clarke

Rights and permissions

Reprints and permissions

Copyright information

© 2021 IFIP International Federation for Information Processing

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Kävrestad, J., Nohlberg, M. (2021). Evaluation Strategies for Cybersecurity Training Methods: A Literature Review. In: Furnell, S., Clarke, N. (eds) Human Aspects of Information Security and Assurance. HAISA 2021. IFIP Advances in Information and Communication Technology, vol 613. Springer, Cham. https://doi.org/10.1007/978-3-030-81111-2_9

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-81111-2_9

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-81110-5

  • Online ISBN: 978-3-030-81111-2

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation