Abstract
Conducting accessibility evaluations with users with disabilities is an important part of developing accessible interactive systems. Conducting such evaluations of systems which require complex domain knowledge is often impossible, as disabled users do not exist or are very rare. This paper presents a user evaluation method to address this problem, consigliere evaluation. A consigliere evaluation has the disabled user as the main participant, but they are assisted by an advisor, or consigliere, who understands the complex domain; the consigliere is in turn, monitored by an accessibility expert, who acts as an enforcer. As in all user evaluations, the disabled participant undertakes a series of tasks. But in a consigliere evaluation, if the task requires some particular domain knowledge or skill, the role of the consigliere is to provide appropriate information. However, it is easy for the consigliere, who usually does not have knowledge of the accessibility domain, to provide information not specifically about the domain, but about how to do the task in general. So the role of the enforcer, who is an accessibility expert, is to ensure this does not happen, and also to provide assistance and explanation if accessibility issues arise that the disabled participant cannot solve. The paper illustrates the consigliere method with a case study, the evaluation of Skillsforge, an online system used by a number of universities to manage progress of postgraduate students. This system requires considerable domain knowledge of the terminology and progression requirements for these students. It is used by university administrative staff, academic staff who supervise postgraduate students or are involved in monitoring students and the students themselves. The case study illustrates how the consigliere evaluation method works and some of the things which need to be considered in conducting the evaluation appropriately.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
The use of this term in no way implies admiration for the mafia, but we were inspired by the Godfather film in which the consigliere whispers advice in the ear of the mafia boss.
References
World Wide Web Consortium – Web Accessibility Initiative (W3C WAI): Web Content accessibility Guidelines (WCAG) 2.1 (2018). https://www.w3.org/TR/WCAG21/. Accessed 12 Apr 2020
Android Developers. Make apps more accessible. https://developer.android.com/guide/topics/ui/accessibility/apps. Accessed 12 Apr 2020
Apple Human Interface Guidelines. Accessibility. https://developer.apple.com/design/human-interface-guidelines/accessibility/overview/introduction/. Accessed 12 Apr 2020
Centre for Excellence in Universal Design. Public access terminals, 1 July 2019 http://universaldesign.ie/technology-ict/irish-national-it-accessibility-guidelines/public-access-terminals/public-access-terminals.html. Accessed 12 Apr 2020
Centre for Excellence in Universal Design. Telecoms. http://universaldesign.ie/technology-ict/irish-national-it-accessibility-guidelines/telecoms/telecoms.html. Accessed 12 Apr 2020
AbleGamers. Design patterns for APX. https://accessible.games/accessible-player-experiences/design-patterns/. Accessed 12 Apr 2020
Sani, Z.H.A., Petrie, H.: User evaluation of an app for liquid monitoring by older adults. In: Antona, M., Stephanidis, C. (eds.) UAHCI 2017. LNCS, vol. 10279, pp. 86–97. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-58700-4_8
Power, C., Freire, A., Petrie, H., Swallow, D.: Guidelines are only half the story: accessibility problems encountered by blind users on the web. In: Proceedings of the 30th International Conference on Human Factors in Computing (CHI 2012). ACM Press, New York (2012)
Petrie, H., Bevan, N.: The evaluation of accessibility, usability and user experience. In: Stephanidis, C. (ed.) The Universal Access Handbook. Taylor and Francis, London (2009)
Sears, A., Hanson, V.L.: Representing users in accessibility research. ACM Trans. Access. Comput. 4(2), 7 (2012)
Aizpurua, A., Arrue, M., Harper, S., Vigo, M.: Are users the gold standard for accessibility evaluation? In: W4A: Proceedings of the 11th Web for All Conference (2014). Article 13
Power, C., Petrie, H.: Working with participants. In: Yesilada, Y., Harper, S. (eds.) Web Accessibility. HIS, pp. 153–168. Springer, London (2019). https://doi.org/10.1007/978-1-4471-7440-0_9
Acknowledgements
We would like to thank the participants who took part in this evaluation, they helped us clarify the consigliere evaluation method. We would also like to thank Skillsforge for the opportunity to work with them and the Engineering and Physical Sciences Research Council (EPSRC) and the University of York for funding the research through the Innovation Voucher Programme.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this paper
Cite this paper
Petrie, H., Samaddar, S., Power, C., Merdenyan, B. (2020). Consigliere Evaluation: Evaluating Complex Interactive Systems with Users with Disabilities. In: Miesenberger, K., Manduchi, R., Covarrubias Rodriguez, M., Peňáz, P. (eds) Computers Helping People with Special Needs. ICCHP 2020. Lecture Notes in Computer Science(), vol 12376. Springer, Cham. https://doi.org/10.1007/978-3-030-58796-3_5
Download citation
DOI: https://doi.org/10.1007/978-3-030-58796-3_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-58795-6
Online ISBN: 978-3-030-58796-3
eBook Packages: Computer ScienceComputer Science (R0)