Abstract
Affective computing, as a novel paradigm, adds a new feature into the human-computer interactions era that brings emotions into the process of design and evaluation. Such condition influences digital games’ not only structural dimension but also content and evaluation areas. This study aims to evaluate the effect of using emotions in the form of facial expressions as a User Interface (UI) and the input of a system in virtual scenarios. After studying the findings and methods of previous researches, a conceptual framework for Affective User Interface Design and Evaluation (AFFUIDE) has been proposed. Then, a game in the genre of shoot ‘em up (called Facial Battle) in four interaction modes has been developed. The evaluation of this study has been performed in terms of the players’ performance and experience. For this purpose, 18 male students of the Multimedia Faculty of Tabriz Islamic Art University took part in the experiments, who were asked to play all four modes of the game. Then, after analyzing the normalization of collected data, one-way ANOVA and U Mann Whitney tests were used for the statistical analysis. The findings indicate that the Traditional UI is the most usable and the Full Affective UI has the best sense of fun and user experience among other methods. However, the hybrid mode (affective shooting), is not only usable as traditional mode, but it also works better in terms of user experience compared to other modes.
Similar content being viewed by others
References
Andersson AT (2016) Facial feature tracking and head pose tracking as input for platform games
Bahreini K, van der Vegt W, Westera W (2019) A fuzzy logic approach to reliable real-time recognition of facial emotions. Multimed Tools Appl 78:18943–18966
Bannon L (2011) Reimagining HCI: toward a more human-centered perspective. Interactions 18:50–57
Benyon D (2019) Designing User Experience: a guide to HCI, UX and interaction design. Pearson, London
Bernays R, Mone J, Yau P et al (2012) Lost in the dark: emotion adaption. In: Adjunct proceedings of the 25th annual ACM symposium on User interface software and technology. ACM, New York, pp 79–80
Blom PM, Bakkes S, Tan CT et al (2014) Towards personalised gaming via facial expression recognition. In: Tenth Artificial Intelligence and Interactive Digital Entertainment Conference
Bogdanovych A, Trescak T, Simoff S (2016) What makes virtual agents believable? Conn Sci 28:83–108
Bowman ND, Pietschmann D, Liebold B (2017) The golden (hands) rule: Exploring user experiences with gamepad and natural-user interfaces in popular video games. J Gaming Virtual Worlds 9:71–85
Brooke J (1996) SUS-A quick and dirty usability scale. Usability Eval Ind 189:4–7
Browne C, Yannakakis GN, Colton S (2012) Guest editorial: Special issue on computational aesthetics in games. IEEE Trans Comput Intell AI Games 4:149–151
Burns A, Tulip J (2017) Detecting flow in games using facial expressions. In: 2017 IEEE Conference on Computational Intelligence, Games (CIG). IEEE, Piscataway, pp 45–52
Cai L, Liu B, Yu J, Zhang J (2017) Human behaviors modeling in multi-agent virtual environment. Multimed Tools Appl 76:5851–5871
Caschera MC, Ferri F, Grifoni P (2016) Sentiment analysis from textual to multimodal features in digital environments. In: Proceedings of the 8th International Conference on Management of Digital EcoSystems. pp 137–144
Čereković A, Pandžić IS (2011) Multimodal behavior realization for embodied conversational agents. Multimed Tools Appl 54:143–164
Collins K, Kapralos B, Kanev K (2014) Smart table computer interaction interfaces with integrated sound
De Byl P (2015) A conceptual affective design framework for the use of emotions in computer game design. Cyberpsychology J Psychosoc Res Cybersp 9
de Peuter A (2014) Development of an adaptive game by runtime adjustment of game parameters to EEG measurements
Dekker A, Champion E (2007) Please Biofeed the Zombies: Enhancing the Gameplay and Display of a Horror Game Using Biofeedback. In: DiGRA Conference. pp 43–52
Dewitt A, Bresin R (2007) Sound design for affective interaction. In: International Conference on Affective Computing and Intelligent Interaction. Springer, Berlin, pp 523–533
Drachen A, Nacke LE, Yannakakis G, Pedersen AL (2010) Correlation between heart rate, electrodermal activity and player experience in first-person shooter games. In: Proceedings of the 5th ACM SIGGRAPH Symposium on Video Games. ACM, New York, pp 49–54
Dudley JJ, Kristensson PO (2018) A review of user interface design for interactive machine learning. ACM Trans Interact Intell Syst 8:8
Dzedzickis A, Kaklauskas A, Bucinskas V (2020) Human emotion recognition: Review of sensors and methods. Sensors 20:592
Ekman P (1999) Basic emotions. Handb Cogn Emot :45–60
Ekman P, Friesen WV (1977) Facial action coding system
Fu LP, Landay J, Nebeling M et al (2018) Redefining natural user interface. In: Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, New York, p SIG19
Ghosh M, Kundu T, Ghosh D, Sarkar R (2019) Feature selection for facial emotion recognition using late hill-climbing based memetic algorithm. Multimed Tools Appl :1–27
Gilleade K, Dix A, Allanson J (2005) Affective videogames and modes of affective gaming: assist me, challenge me, emote me. DiGRA 2005 Chang Views–Worlds Play
Guthier B, Dörner R, Martinez HP (2016) Affective computing in games. In: Entertainment Computing and Serious Games. Springer, Berlin pp 402–441
Hamdy S, King D (2018) Affective games: a multimodal classification system. In: 19th annual European GAME-ON Conference (GAME-ON’2018) on Simulation and AI in Computer Games. EUROSIS, Ghent
Hassenzahl M, Burmester M, Koller F (2003) AttrakDiff: Ein Fragebogen zur Messung wahrgenommener hedonischer und pragmatischer Qualität. In: Mensch & computer 2003. Springer, Berlin, pp 187–196
Högberg J, Hamari J, Wästlund E (2019) Gameful Experience Questionnaire (GAMEFULQUEST): an instrument for measuring the perceived gamefulness of system use. User Model User-adapt Interact 1–42
Hubbard JA (2001) Emotion expression processes in children’s peer interaction: The role of peer rejection, aggression, and gender. Child Dev 72:1426–1438
i Badia SB, Quintero LV, Cameirão MS et al (2018) Toward emotionally adaptive virtual reality for mental health applications. IEEE J Biomed Health Inform 23:1877–1887
Ilves M, Gizatdinova Y, Surakka V, Vankka E (2014) Head movement and facial expressions as game input. Entertain Comput 5:147–156
Ip B (2011) Narrative structures in computer and video games: Part 1: Context, definitions, and initial findings. Games Cult 6:103–134
Jaimes A, Sebe N (2007) Multimodal human–computer interaction: A survey. Comput Vis Image Underst 108:116–134
Ji H, Yun Y, Lee S et al (2018) An adaptable UI/UX considering user’s cognitive and behavior information in distributed environment. Cluster Comput 21:1045–1058
Johnson D, Wiles J (2003) Effective affective user interface design in games. Ergonomics 46:1332–1345
Kleinginna PR, Kleinginna AM (1981) A categorized list of emotion definitions, with suggestions for a consensual definition. Motiv Emot 5:345–379
Korn O, Blatz M, Rees A et al (2017) Procedural content generation for game props? a study on the effects on user experience. Comput Entertain 15:1–15
Kromand D (2007) Avatar Categorization. In: DiGRA Conference
Landowska A, Miler J (2016) Limitations of emotion recognition in software user experience evaluation context. In: 2016 Federated Conference on Computer Science and Systems I (FedCSIS). IEEE, Piscataway, pp 1631–1640
Lang PJ (1995) The emotion probe: studies of motivation and attention. Am Psychol 50:372
Lankes M, Riegler S, Weiss A et al (2008) Facial expressions as game input with different emotional feedback conditions. In: Proceedings of the 2008 International Conference on Advances in Computer Entertainment Technology. ACM, New York, pp 253–256
Lim MY, Dias J, Aylett R, Paiva A (2012) Creating adaptive affective autonomous NPCs. Auton Agent Multi Agent Syst 24:287–311
Liu C, Agrawal P, Sarkar N, Chen S (2009) Dynamic difficulty adjustment in computer games through real-time anxiety-based affective feedback. Int J Hum Comput Interact 25:506–529
Mandel T (1997) The elements of user interface design. Wiley, Hoboken
Mandryk RL, Inkpen KM, Calvert TW (2006) Using psychophysiological techniques to measure user experience with entertainment technologies. Behav Inf Technol 25:141–158
Mansilla WA (2006) Interactive dramaturgy by generating acousmêtre in a virtual environment. In: International Conference on Entertainment Computing. Springer, Berlin, pp 90–95
McDuff D, Mahmoud A, Mavadati M et al (2016) AFFDEX SDK: a cross-platform real-time multi-face expression recognition toolkit. In: Proceedings of the 2016 CHI conference extended abstracts on human factors in computing systems. ACM, New York, pp 3723–3726
Mohammadi HS, Pirbabaei E, Sisi MJ, Sekhavat YA (2018) ExerBrain: A Comparison of Positive and Negative Reinforcement in Attention Training Using BCI based Computer Games. In: 2018 2nd National and 1st International Digital Games Research Conference: Trends, Technologies, and Applications (DGRC). IEEE, Piscataway, pp 167–171
Müller N, Eska B, Schäffer R et al (2018) Arch’n’Smile: A Jump’n’Run Game Using Facial Expression Recognition Control For Entertaining Children During Car Journeys. In: Proceedings of the 17th International Conference on Mobile and Ubiquitous Multimedia. ACM, New York, pp 335–339
Mungra D, Agrawal A, Sharma P et al (2019) PRATIT: a CNN-based emotion recognition system using histogram equalization and data augmentation. Multimed Tools Appl :1–23
Ng YY, Khong CW (2014) A review of affective user-centered design for video games. In: 2014 3rd International Conference on User Science and Engineering (i-USEr). IEEE, Piscataway, pp 79–84
Obaid M, Han C, Billinghurst M (2008) Feed the Fish: an affect-aware game. In: Proceedings of the 5th Australasian Conference on Interactive Entertainment. ACM, New York, p 6
Pan S, Brdiczka O, Carenini G et al (2016) What’s Hot in Intelligent User Interfaces. In: Thirtieth AAAI Conference on Artificial Intelligence
Partala T (2005) Affective information in human-computer interaction. Tampere University Press, Tampere
Pawar S, Tam F, Plass JL (2020) 14 Emerging Design Factors in Game-Based Learning: Emotional Design, Musical Score, and Game Mechanics Design. Handb Game-Based Learn 347
Picard RW (1995) Affective Computing-MIT Media Laboratory Perceptual Computing Section Technical Report No. 321. Cambridge, MA 2139
Picard RW (2000) Affective computing. MIT Press, Cambridge
Plutchik RE, Conte HR (1997) Circumplex models of personality and emotions. American Psychological Association, Washington, D.C.
Preece J, Rogers Y, Sharp H et al (1994) Interface metaphors and conceptual models. Human-Computer Interation 9
Prema P, Ramadoss B (2008) A Classification Scheme for Game Input and Output. World Acad Sci Eng Technol Int J Comput Electr Autom Control Inf Eng 2:1870–1876
Przybylski AK, Weinstein N, Murayama K et al (2012) The ideal self at play: The appeal of video games that let you be all you can be. Psychol Sci 23:69–76
Raheel A, Anwar SM, Majid M (2019) Emotion recognition in response to traditional and tactile enhanced multimedia using electroencephalography. Multimed Tools Appl 78:13971–13985
Robison J, McQuiggan S, Lester J (2009) Evaluating the consequences of affective feedback in intelligent tutoring systems. In: 2009 3rd international conference on affective computing and intelligent interaction and workshops. IEEE, Piscataway, pp 1–6
Ruan S, Chen L, Sun J, Chen G (2009) Study on the change of physiological signals during playing body-controlled games. In: Proceedings of the International Conference on Advances in Computer Enterntainment Technology. ACM, New York, pp 349–352
Russell JA (1980) A circumplex model of affect. J Pers Soc Psychol 39:1161
Sawyer R, Smith A, Rowe J et al (2017) Enhancing student models in game-based learning with facial expression recognition. In: Proceedings of the 25th conference on user modeling, adaptation and personalization. ACM, New York, pp 192–201
Sekhavat YA (2019) Battle of minds: a new interaction approach in BCI games through competitive reinforcement. Multimed Tools Appl :1–16
Sekhavat YA (2020) Collaboration or battle between minds? An attention training game through collaborative and competitive reinforcement. Entertain Comput 100360
Sekhavat YA, Roohi S, Mohammadi HS, Yannakakis GN (2020) Play with One’s Feelings: A Study on Emotion Awareness for Player Experience. IEEE Trans Games
Sharma M, Jalal AS, Khan A (2019) Emotion recognition using facial expression by fusing key points descriptor and texture features. Multimed Tools Appl 78:16195–16219
Shaw M (1986) An input-output model of interactive systems. In: Proceedings of the SIGCHI conference on Human factors in computing systems. pp 261–273
Shilling R, Zyda M, Wardynski EC (2002) Introducing emotion into military simulation and videogame design: America’s Army: Operations and VIRTE. In: Proceedings of the GameOn Conference
Song W, Cai X, Xi Y et al (2017) Real-time single camera natural user interface engine development. Multimed Tools Appl 76:11159–11175
Souza AD, Souza RD (2019) Affective interaction based hybrid approach for emotion detection using machine learning. In: 2019 International Conference on Smart Systems and Inventive Technology (ICSSIT). IEEE, Piscataway, pp 337–342
Takatalo J, Häkkinen J, Kaistinen J, Nyman G (2010) Presence, involvement, and flow in digital games. In: Evaluating user experience in games. Springer, Berlin, pp 23–46
Turečková Š (2016) Using player’s facial emotional expressions as a game input: Effects on narrative engagement
Valli A (2008) The design of natural interaction. Multimed Tools Appl 38:295–305
Von Ahn L, Dabbish L (2008) Designing games with a purpose. Commun ACM 51:58–67
Werbach K, Hunter D (2012) For the win: How game thinking can revolutionize your business. Wharton Digital Press
Xie L, Jia J, Meng H et al (2015) Expressive talking avatar synthesis and animation. Multimed Tools Appl 74:9845–9848
Yannakakis GN, Togelius J (2015) Experience-driven procedural content generation. In: 2015 International Conference on Affective Computing and Interaction I (ACII). IEEE, Piscataway, pp 519–525
Yeh Y, Lai SC, Lin C-W (2016) The dynamic influence of emotions on game-based creativity: An integrated analysis of emotional valence, activation strength, and regulation focus. Comput Human Behav 55:817–825
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
This paper is an excerpt from the master’s thesis of Milad Jafari Sisi.
Rights and permissions
About this article
Cite this article
Sekhavat, Y.A., Sisi, M.J. & Roohi, S. Affective interaction: Using emotions as a user interface in games. Multimed Tools Appl 80, 5225–5253 (2021). https://doi.org/10.1007/s11042-020-10006-4
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-020-10006-4