Skip to main content
SpringerLink
Log in

A Review of Methods for Evaluating Security Awareness Initiatives

  • Original Article
  • Published:
European Journal for Security Research Aims and scope Submit manuscript

Abstract

The ‘human factor’ is commonly considered to be the weakest link in an organization’s security chain, and a significant percentage of companies have implemented security awareness (SA) programs to address this vulnerability. However, an element whose usefulness is still underestimated is the importance to perform measurements of the different SA programs’ effectiveness in order to assess their adequateness for achieving the intended goals. This gap has serious consequences as most of the security awareness campaigns have resulted to be largely unsuccessful. Awareness measurement tools might be determinant in providing feedback on the outcome of a program as well as in helping with the strategic planning for endorsing security. This article will introduce and critically compare a set of measurement methods. It will then discuss their attributes and suggested applications.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16

Similar content being viewed by others

References

  • Abawajy J, Thatcher K, Kim TH (2008) Investigation of stakeholders commitment to information security awareness programs. In: 2008 international conference on information security and assurance (ISA 2008) IEEE, pp 472–476

  • Academic Frontier Project for Private Universities (2009) Survey on the internet security awareness. March. https://www.kansai-u.ac.jp/riss/en/shareduse/data/17_E_questionnaire.pdf. Accessed 4 Sept 2019

  • Agresti A (2018) An introduction to categorical data analysis. Wiley, New York

    Google Scholar 

  • Al-Awadi M (2009) A study of employees’ attitudes towards organisational information security policies in the UK and Oman (Doctoral dissertation, University of Glasgow)

  • BBC News (2004) The Chernobyl disaster. BBC special reports. http://news.bbc.co.uk/1/shared/spl/hi/guides/456900/456957/html/nn1page1.stm. Accessed 4 Sept 2019

  • Bradburn NM, Sudman S, Wansink B (2004) Asking questions: the definitive guide to questionnaire design–for market research, political polls, and social and health questionnaires. Wiley

  • Bresz FP (2004) People—often the weakest link in security, but one of the best places to start. J Health Care Compliance 6(4):57–60

    Google Scholar 

  • Brink DE (2017) Security awareness training: small investment, large reduction risk. Aberdeen Group. https://www.proofpoint.com/us/resources/analyst-reports/aberdeen-security-awareness-training. Accessed 4 Sept 2019

  • Brunner EM, Suter M (2008) International CIIP handbook 2008/2009. Center for Security Studies, ETH Zurich, Zurich

    Google Scholar 

  • Bulgurcu B, Cavusoglu H, Benbasat I (2010) Information security policy compliance: an empirical study of rationality-based beliefs and information security awareness. MIS Q 34(3):523–548

    Google Scholar 

  • Byres E, Lowe J (2004) The myths and facts behind cyber security risks for industrial control systems. In: Proceedings of the VDE kongress, vol 116, pp 213–218

  • Caputo DD, Pfleeger SL, Freeman JD, Johnson ME (2014) Going spear phishing: exploring embedded training and awareness. IEEE Secur Priv 12(1):28–38

    Google Scholar 

  • Choo KKR (2011) The cyber threat landscape: challenges and future research directions. Comput Secur 30(8):719–731

    Google Scholar 

  • Computer Fraud & Security (2016) News—employees prone to phishing. Comput Fraud Secur 2016(1):3. https://doi.org/10.1016/S1361-3723(16)30004-5

    Article  Google Scholar 

  • Cone BD, Irvine CE, Thompson MF, Nguyen TD (2007) A video game for cyber security training and awareness. Comput Secur 26(1):63–72

    Google Scholar 

  • Corbetta P (1999) Metodologia e tecniche della ricerca sociale. http://www.uniroma2.it/didattica/statistica_sociale_B/deposito/corbettametodologia_e_tecniche_della_ricerca_socialeriassunto.pdf. Accessed 4 Sept 2019

  • Crossler RE, Johnston AC, Lowry PB, Hu Q, Warkentin M, Baskerville R (2013) Future directions for behavioral information security research. Comput Secur 32:90–101

    Google Scholar 

  • Das SK, Kant K, Zhang N (2012) Handbook on securing cyber-physical critical infrastructure. Elsevier, Amsterdam

    Google Scholar 

  • Davie M (1986) The Titanic: the full story of a tragedy. Random House, London

    Google Scholar 

  • Davis FD (1989) Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q 13:319–340

    Google Scholar 

  • De Maggio MC, Mastrapasqua M, Tesei M, Chittaro A, Setola R (2017) How to improve the security awareness in complex organizations. Eur J Secur Res 4:1–17

    Google Scholar 

  • Durkheim É (1897) Le suicide: étude de sociologie. Alcan, Paris

    Google Scholar 

  • ENISA (2012) Introduction to return on security investment. https://www.enisa.europa.eu/publications/introduction-to-return-on-security-investment. Accessed 4 Sept 2019

  • Furnell SM, Jusoh A, Katsabas D (2006) The challenges of understanding and using security: a survey of end-users. Comput Secur 25(1):27–35

    Google Scholar 

  • Furnell SM, Bryant P, Phippen AD (2007) Assessing the security perceptions of personal Internet users. Comput Secur 26(5):410–417

    Google Scholar 

  • Granger S (2001) Social engineering fundamentals, part I: hacker tactics. Secur Focus

  • Groves RM, Fowler FJ Jr, Couper MP, Lepkowski JM, Singer E, Tourangeau R (2011) Survey methodology, vol 561. Wiley, New York

    Google Scholar 

  • Gubrium JF, Holstein JA (2001) Handbook of interview research: context and method. Sage Publications, Thousand Oaks

    Google Scholar 

  • Hansche S (2001) Designing a security awareness program: part I. Inf Syst Secur 9(6):14–23

    Google Scholar 

  • Herath T, Rao HR (2009) Encouraging information security behaviors in organizations: role of penalties, pressures and perceived effectiveness. Decis Support Syst 47(2):154–165

    Google Scholar 

  • Hills M, Anjali A (2017) A human factors contribution to countering insider threats: practical prospects from a novel approach to warning and avoiding. Secur J 30(1):142–152

    Google Scholar 

  • Information Security Forum (2014) From promoting awareness to embedding behaviours. https://www.securityforum.org/uploads/2015/03/From-Promoting-Awareness-ES-2014_Marketing.pdf. Accessed 4 Sept 2019

  • Karjalainen M, Siponen M (2011) Toward a new meta-theory for designing information systems (IS) security training approaches. Jo Assoc Inf Syst 12(8):518–555

    Google Scholar 

  • Kruger HA, Kearney WD (2006) A prototype for assessing information security awareness. Comput Secur 25(4):289–296

    Google Scholar 

  • Larsson R (1993) Case survey methodology: quantitative analysis of patterns across case studies. Acad Manag J 36(6):1515–1546

    Google Scholar 

  • Lee R (2017) Crashoverride: analysis of the threat to electric grid operations. Dragos Inc., Rome

    Google Scholar 

  • Mani D, Raymond Choo KK, Mubarak S (2014) Information security in the South Australian real estate industry: a study of 40 real estate organisations. Inf Manag Comput Secur 22(1):24–41

    Google Scholar 

  • Manke S, Winkler I (2012) The habits of highly successful security awareness programs: a cross-company comparison. Technical report, secure mentem, 2012. http://www.securementem.com/wp-content/uploads/2013/07/Habits_white_paper.pdf. Accessed 4 Sept 2019

  • Mishra S, Dhillon G (2006) Information systems security governance research: a behavioral perspective. In: 1st annual symposium on information assurance, academic track of 9th annual NYS cyber security conference, pp 27–35

  • Mitnick KD, Simon WL (2011) The art of deception: controlling the human element of security. Wiley, New York

    Google Scholar 

  • Moore T, Dynes S, Chang FR (2015) Identifying how firms manage cybersecurity investment. Southern Methodist University. http://blog.smu.edu/research/files/2015/10/SMU-IBM.pdf. Accessed 4 Sept 2019

  • Moteff J, Parfomak P (2004) Critical infrastructure and key assets: definition and identification. Library of Congress Washington DC Congressional Research Service

  • Muir A, Lopatto J (2004) Final report on the August 14, 2003 blackout in the United States and Canada: causes and recommendations

  • Murchison C (1935) A handbook of social psychology. Clark University Press, Worcester, pp 789–844

    Google Scholar 

  • NIST (1998) Information technology security training requirements: a role-and performance-based model (supersedes NIST Spec. Pub.500-172), SP 800-16, March

  • Parsons K, McCormac A, Pattinson M, Butavicius M, Jerram C (2014) A study of information security awareness in Australian government organisations. Inf Manag Comput Secur 22(4):334–345

    Google Scholar 

  • Pastor V, Díaz G, Castro M (2010). State-of-the-art simulation systems for information security education, training and awareness. In: 2010 IEEE education engineering (EDUCON). IEEE, pp 1907–1916

  • Patrick AS, Long AC, Flinn S (2003). HCI and security systems. In: CHI’03 extended abstracts on human factors in computing systems. ACM, pp 1056–1057

  • Peltier TR (2005) Implementing an information security awareness program. Inf Syst Secur 14(2):37–49

    Google Scholar 

  • Perrow C (2011) Normal accidents: living with high risk technologies-updated edition. Princeton University Press, Princeton

    Google Scholar 

  • Pescaroli G, Alexander D (2016) Critical infrastructure, panarchies and the vulnerability paths of cascading disasters. Nat Hazards 82(1):175–192

    Google Scholar 

  • Pfleeger SL, Sasse MA, Furnham A (2014) From weakest link to security hero: transforming staff security behavior. J Homel Secur Emerg Manag 11(4):489–510

    Google Scholar 

  • Ponemon Institute LLC (2012) The human factor in data protection. https://www.ponemon.org/blog/the-human-factor-in-data-protection. Accessed 4 Sept 2019

  • Rahim NHA, Hamid S, Mat Kiah ML, Shamshirband S, Furnell S (2015) A systematic review of approaches to assessing cybersecurity awareness. Kybernetes 44(4):606–622

    Google Scholar 

  • Reason J (2000) Human error: models and management. BMJ 320(7237):768–770

    Google Scholar 

  • Ruan K (2017) Introducing cybernomics: a unifying economic framework for measuring cyber risk. Comput Secur 65:77–89

    Google Scholar 

  • Ryan RM, Deci EL (2000) Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. Am Psychol 55(1):68

    Google Scholar 

  • Saaty TL (1988) What is the analytic hierarchy process? In: Mitra G (ed) Mathematical models for decision support. Springer, Berlin, pp 109–121

    Google Scholar 

  • SANS (2012) Security awareness survey. Sans Institute, April. https://www.sans.org/sites/default/files/2018-01/security-awareness-survey.pdf. Accessed 4 Sept 2019

  • Schein EH (2009) The corporate culture survival guide, vol 158. Wiley, New York

    Google Scholar 

  • Schultz E (2005) From the Editor-in-Chief: the human factor in security. Comput Secur 24(6):425–426

    Google Scholar 

  • Setola R, Mastrapasqua M, Tesei M, De Maggio MC, Corradini I, Pantaleo C, Capitello ME, De Simio F (2015) Study on security awareness in gas infrastructure. NITEL, March

  • Setola R, Rosato V, Kyriakides E, Rome E (2016) Managing the complexity of critical infrastructures. In: Janusz K (ed) Studies in systems, decision and control book series, vol 90. Springer, Berlin

    Google Scholar 

  • Siponen MT (2000) A conceptual foundation for organizational information security awareness. Inf Manag Comput Secur 8(1):31–41

    Google Scholar 

  • Siponen MT (2001) Five dimensions of information security awareness. SIGCAS Comput Soc 31(2):24–29

    Google Scholar 

  • Siponen M, Pahnila S, Mahmood MA (2010) Compliance with information security policies: an empirical investigation. Computer 43(2):64–71

    Google Scholar 

  • Solms BV (2000) Information security—the third wave? Comput Secur 19(7):615–615

    Google Scholar 

  • Soomro ZA, Shah MH, Ahmed J (2016) Information security management needs more holistic approach: a literature review. Int J Inf Manag 36(2):215–225

    Google Scholar 

  • Stone DH (1993) Design a questionnaire. BMJ 307(6914):1264–1266

    Google Scholar 

  • Szilagyi AD, Wallace MJ (1983) Organizational behavior and performance. Good Year Books, Culver

    Google Scholar 

  • Talib S, Clarke NL, Furnell SM (2010) An analysis of information security awareness within home and work environments. In: 2010 international conference on availability, reliability and security. IEEE, pp 196–203

  • Tsohou A, Kokolakis S, Karyda M, Kiountouzis E (2008) Investigating information security awareness: research and practice gaps. Inf Secur J Glob Perspect 17(5–6):207–227

    Google Scholar 

  • US-Canada Power System Outage Task Force (2004) Final report on the August 14, 2003 blackout in the United States and Canada: causes and recommendations. US-Canada Power System Outage Task Force

  • Velki T, Solic K, Ocevcic H (2014) Development of users’ information security awareness questionnaire (UISAQ)—ongoing work. In: 2014 37th international convention on information and communication technology, electronics and microelectronics (MIPRO). IEEE, pp 1417–1421

  • Verizon (2016) 2016 data breach investigations report. http://www.verizonenterprise.com/verizon-insights-lab/dbir/. Accessed 4 Sept 2019

  • Williams EJ, Hinds J, Joinson AN (2018) Exploring susceptibility to phishing in the workplace. Int J Hum Comput Stud 120:1–13

    Google Scholar 

  • Wilson M, Hash J (2003) Building an information technology security awareness and training program. NIST Spec Publ 800(50):1–39

    Google Scholar 

  • Workman M, Bommer WH, Straub D (2008) Security lapses and the omission of information security measures: a threat control model and empirical test. Comput Hum Behav 24(6):2799–2816

    Google Scholar 

  • Zimmerman R (2004). Decision-making and the vulnerability of interdependent critical infrastructure. In: 2004 IEEE international conference on systems, man and cybernetics, vol 5. IEEE, pp 4059–4063

Download references

Author information

Authors and Affiliations

  1. Complex System and Security Lab, Università Campus Bio-Medico di Roma, Via Alvaro del Portillo, 21, 00128, Rome, Italy

    Giacomo Assenza, Maria Carla De Maggio & Roberto Setola

  2. SNAM, Corporate Security, Milan, Italy

    Andrea Chittaro & Marzia Mastrapasqua

Authors
  1. Giacomo Assenza
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Andrea Chittaro
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Maria Carla De Maggio
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Marzia Mastrapasqua
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Roberto Setola
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Roberto Setola.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Assenza, G., Chittaro, A., De Maggio, M.C. et al. A Review of Methods for Evaluating Security Awareness Initiatives. Eur J Secur Res 5, 259–287 (2020). https://doi.org/10.1007/s41125-019-00052-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s41125-019-00052-x

Keywords

Search

Navigation