Skip to main content
SpringerLink
Log in

BARGAIN: behavioral affective rule-based games adaptation interface–towards emotionally intelligent games: application on a virtual reality environment for socio-moral development

  • Published:
User Modeling and User-Adapted Interaction Aims and scope Submit manuscript

Abstract

This paper presents a framework for adapting game elements to the player’s affective state and the integration of the framework in a virtual reality environment for moral development. These game elements include gestural and facial expressions of avatars during dialogues with the player, background music, the score, game mechanics, aesthetics and learning. The framework BARGAIN (Behavioral Affective Rule-based Games Adaptation Interface) is an authoring tool for affective game design providing a visual interface based on finite state machine (FSM) technique to represent the affective rules as state transitions graph dependent on the player emotional state assessed using facial expression recognition system based on electroencephalography (EEG) data. We conducted a user study (n = 29) examining the effects of the resulting affective virtual reality game on players’ experience using the Game experience Questionnaire (GEQ) (IJsselsteijn et al. in The game experience questionnaire, Technische Universiteit Eindhoven, Eindhoven, 2013). The results show significant correlation between the GEQ dimensions and the player's facial expressions during his interaction with the Non-Player Characters (NPCs) within the VR game. These findings highlight that adapting games to user's emotions enhance the players’ experience.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21
Fig. 22
Fig. 23
Fig. 24
Fig. 25
Fig. 26
Fig. 27
Fig. 28
Fig. 29
Fig. 30

Similar content being viewed by others

Notes

  1. www.oculus.com.

  2. www.emotiv.com.

  3. We have not used Emotiv affective suite that detects facial states like: blink, frown, raised eyebrows, wink and smile.

  4. https://imotions.com/emotient/.

  5. Unity's Prefab allows to create, configure and store a gameobject complete with all its components, property values and child gameobjects.

  6. Very joyful Music: https://www.playonloop.com/2017-music-loops/doggo-brothers/.

  7. Moderately joyful Music: www.audiomicro.com/feelin-good-wav-royalty-free-stock-music-1438027.

  8. Less joyful Music: https://freesound.org/people/axtoncrolley/sounds/172707/.

  9. https://en.wikipedia.org/wiki/Bonferroni_correction.

  10. https://jasp-stats.org/.

  11. https://en.wikipedia.org/wiki/Beta_distribution#Jeffreys'_prior_probability_(Beta(1/2,1/2)_for_a_Bernoulli_or_for_a_binomial_distribution).

  12. https://forum.cogsci.nl/index.php?p=/discussion/2161/setting-priors-in-jasp.

References

  • Abdessalem, H.B., Frasson, C.: Real-Time Brain Assessment for Adaptive Virtual Reality Game: a Neurofeedback Approach. Springer, Cham (2017)

    Google Scholar 

  • Ahmad, I., Hamid, E., Abdullasim, N., Jaafar, A.: Game Interface Design: Measuring the Player’s Gameplay Experience. Springer, Cham (2017)

    Google Scholar 

  • Arellano, D.G., Tokarchuk, L., Gunes, H.: Measuring Affective, Physiological and Behavioural Differences in Solo, Competitive and Collaborative Games. Springer, Cham (2016)

    Google Scholar 

  • Bekinschtein, T., Niklison, J., Sigman, L., Manes, F., Leiguarda, R., Armony, J., Owen, A., Carpintiero, S., Olmos, L.: Emotion processing in the minimally conscious state. J Neurol Neurosurg Psychiatry 75(5), 788–788 (2004)

    Article  Google Scholar 

  • Benlamine, M., Chaouachi, M., Frasson, C., Dufresne, A.: Physiology-based recognition of facial micro-expressions using EEG and identification of the relevant sensors by emotion. In: Proceedings of the 3rd International Conference on Physiological Computing Systems—Volume 1: PhyCS (2016a)

  • Benlamine, M.S., Chaouachi, M., Frasson, C., Dufresne, A.: Predicting Spontaneous Facial Expressions from EEG. Springer, Berlin (2016b)

    Google Scholar 

  • Benlamine, M.S., Villata, S., Ghali, R., Frasson, C., Gandon, F., Cabrio, E.: Persuasive Argumentation and Emotions: An Empirical Evaluation with Users. Springer, Cham (2017)

    Google Scholar 

  • Bernhaupt, R., Eckschlager, M., Tscheligi, M.: Methods for Evaluating Games: How to Measure Usability and User Experience in Games? ACM, New York (2007)

    Book  Google Scholar 

  • Böckle, M., Micheel, I., Bick, M., Novak, J.: A Design Framework for Adaptive Gamification Applications. Proceedings of the 51st Hawaii international conference on system sciences (HICSS 2018), 1227–1236 (2018)

  • Bontchev, B.: Adaptation in affective video games: a literature review. Cybern Inf Technol 16(3), 3–34 (2016)

    MathSciNet  Google Scholar 

  • Chaouachi, M., Frasson, C.: Mental Workload, Engagement and Emotions: An Exploratory Study for Intelligent Tutoring Systems. Springer, Berlin (2012)

    Google Scholar 

  • Chaouachi, M., Chalfoun, P., Jraidi, I., Frasson, C.: Affect and Mental Engagement: Towards Adaptability for Intelligent Systems. AAAI Press, Palo Alto (2010)

    Google Scholar 

  • Chaouachi, M., Jraidi, I., Frasson, C.: MENTOR: A Physiologically Controlled Tutoring System. Springer, Cham (2015)

    Google Scholar 

  • Chiasson, V., Vera-Estay, E., Lalonde, G., Dooley, J., Beauchamp, M.: Assessing social cognition: age-related changes in moral reasoning in childhood and adolescence. Clin Neuropsychol 31(3), 515–530 (2017)

    Article  Google Scholar 

  • Choi, J., Peters, M., Mueller, R.O.: Correlational analysis of ordinal data: from Pearson’s r to Bayesian polychoric correlation. Asia Pac Educ Rev 11(4), 459–466 (2010)

    Article  Google Scholar 

  • Christy, T., Kuncheva, L.I.: Technological advancements in affective gaming: a historical survey. GSTF J Comput (JoC) 3(4), 32–41 (2018)

    Google Scholar 

  • de Byl, P.: A conceptual affective design framework for the use of emotions in computer game design. Cyberpsychol J Psychosoc Res Cyberspace 9(3), 4 (2015). https://doi.org/10.5817/CP2015-3-4

    Article  Google Scholar 

  • Dekker, A., Champion, E.: Please Biofeed the Zombies: Enhancing the Gameplay and Display of a Horror Game using Biofeedback. In: Proceedings of DiGRA 2007 Conference: Situated Play, 550–558 (2007)

  • Drachen, A., Nacke, L.E., Yannakakis, G., Pedersen, A.L.: Correlation Between Heart Rate, Electrodermal Activity and Player Experience in First-Person Shooter Games. ACM, New York (2010)

    Book  Google Scholar 

  • Fairclough, S.H.: Physiological Computing: Interfacing with the Human Nervous System. Springer, New York (2010)

    Google Scholar 

  • Garner, T.A.: Game Sound from Behind the Sofa: An Exploration into the Fear Potential of Sound and Psychophysiological Approaches to Audio-Centric, Adaptive Gameplay. Ph D Thesis, Aalborg University (2013)

  • Gibbs, J.C.: Moral Development and Reality: Beyond the Theories of Kohlberg, Hoffman, and Haidt. Oxford University Press, Oxford (2013)

    Book  Google Scholar 

  • Goldberg, L.R.: The development of markers for the Big-Five factor structure. Psychol Assess 4(1), 26 (1992)

    Article  Google Scholar 

  • Hagemann, D., Naumann, E., Thayer, J.F., Bartussek, D.: Does resting electroencephalograph asymmetry reflect a trait? An application of latent state-trait theory. J Personal Soc Psychol 82(4), 619 (2002)

    Article  Google Scholar 

  • Hamari, J., Shernoff, D.J., Rowe, E., Coller, B., Asbell-Clarke, J., Edwards, T.: Challenging games help students learn: an empirical study on engagement, flow and immersion in game-based learning. Comput Hum Behav 54, 170–179 (2016)

    Article  Google Scholar 

  • Harmon-Jones, E., Gable, P.A.: On the role of asymmetric frontal cortical activity in approach and withdrawal motivation: an updated review of the evidence. Psychophysiology 55(1), e12879 (2018)

    Article  Google Scholar 

  • Harmon-Jones, E., Gable, P.A., Peterson, C.K.: The role of asymmetric frontal cortical activity in emotion-related phenomena: a review and update. Biol Psychol 84(3), 451–462 (2010)

    Article  Google Scholar 

  • i Badia, S.B., Quintero, L.V., Cameirao, M.S., Chirico, A., Triberti, S., Cipresso, P., Gaggioli, A.: Towards emotionally-adaptive virtual reality for mental health applications. IEEE J Biomed Health Inform 23(5), 1877–1887 (2018)

    Article  Google Scholar 

  • IJsselsteijn, W., De Kort, Y., Poels, K.: The Game Experience Questionnaire. Technische Universiteit Eindhoven, Eindhoven (2013)

  • Kataria, M.: Confirmation: What’s in the evidence? J Behav Exp Econ 65, 9–15 (2016)

    Article  Google Scholar 

  • Kosunen, I., Salminen, M., Järvelä, S., Ruonala, A., Ravaja, N., Jacucci, G.: RelaWorld: Neuroadaptive and Immersive Virtual Reality Meditation System. ACM, New York (2016)

    Book  Google Scholar 

  • Lehman, B., Matthews, M., D’Mello, S., Person, N.: What are You Feeling? Investigating Student Affective States During Expert Human Tutoring Sessions. Springer, Berlin (2008)

    Book  Google Scholar 

  • Molho, C., Tybur, J.M., Güler, E., Balliet, D., Hofmann, W.: Disgust and anger relate to different aggressive responses to moral violations. Psychol Sci 28(5), 609–619 (2017)

    Article  Google Scholar 

  • Mulder, J.: Bayes factors for testing order-constrained hypotheses on correlations. J Math Psychol 72, 104–115 (2016)

    Article  MathSciNet  Google Scholar 

  • Nacke, L., Lindley, C.A.: Flow and Immersion in First-Person Shooters: Measuring the Player’s Gameplay Experience. ACM, New York (2008)

    Book  Google Scholar 

  • Nacke, L.E., Kalyn, M., Lough, C., Mandryk, R.L.: Biofeedback Game Design: Using Direct and Indirect Physiological Control to Enhance Game Interaction. ACM, New York (2011)

    Book  Google Scholar 

  • Nogueira, P.A., Rodrigues, R.A., Oliveira, E.C., Nacke, L.E.: Guided Emotional State Regulation: Understanding and Shaping Players' Affective Experiences in Digital Games. In: The 9th Annual AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment (AIIDE-13), AAAI Press, 51–58 (2013)

  • Philip, L., Martin, J.-C., Clavel, C.: Rapid facial reactions in response to facial expressions of emotion displayed by real versus virtual faces. i-Perception 9(4), 2041669518786527 (2018)

    Article  Google Scholar 

  • Picard, R.W.: Affective Computing. MIT press, Cambridge (1997)

    Google Scholar 

  • Pope, A.T., Bogart, E.H., Bartolome, D.S.: Biocybernetic system evaluates indices of operator engagement in automated task. Biol Psychol 40(1–2), 187–195 (1995)

    Article  Google Scholar 

  • Putman, P., van Peer, J., Maimari, I., van der Werff, S.: EEG theta/beta ratio in relation to fear-modulated response-inhibition, attentional control, and affective traits. Biol Psychol 83(2), 73–78 (2010)

    Article  Google Scholar 

  • Quintana, D.S., Williams, D.R.: Bayesian alternatives for common null-hypothesis significance tests in psychiatry: a non-technical guide using JASP. BMC Psychiatry 18(1), 178 (2018)

    Article  Google Scholar 

  • Simpson, J., Carter, S., Anthony, S.H., Overton, P.G.: Is disgust a homogeneous emotion? Motiv Emot 30(1), 31–41 (2006)

    Article  Google Scholar 

  • Sitaram, R., Lee, S., Ruiz, S., Rana, M., Veit, R., Birbaumer, N.: Real-time support vector classification and feedback of multiple emotional brain states. Neuroimage 56(2), 753–765 (2011)

    Article  Google Scholar 

  • Tato, A.A.N., Nkambou, R., Dufresne, A., Beauchamp, M.H.: Convolutional neural network for automatic detection of sociomoral reasoning level. In: Proceedings of the 10th International Conference on Educational Data Mining (EDM2017), 284–289 (2017)

  • Tato, A.A.N., Dufresne, A., Nkambou, R., Morasse, F., Beauchamp, M.H.: Preliminary Evaluation of a Serious Game for Socio-Moral Reasoning. Springer, Berlin (2018)

    Google Scholar 

  • Villata, S., Cabrio, E., Jraidi, I., Benlamine, S., Chaouachi, M., Frasson, C., Gandon, F.: Emotions and personality traits in argumentation: an empirical evaluation 1. Argum Comput 8(1), 61–87 (2017)

    Article  Google Scholar 

  • Wagenmakers, E.-J., Love, J., Marsman, M., Jamil, T., Ly, A., Verhagen, J., Selker, R., Gronau, Q.F., Dropmann, D., Boutin, B.: Bayesian inference for psychology. Part II: Example applications with JASP. Psychon Bull Rev 25(1), 58–76 (2018)

    Article  Google Scholar 

Download references

Acknowledgements

The authors acknowledge support of the NSERC (Natural Sciences and Engineering Research Council of Canada), the FRQSC (Fonds de Recherche du Québec Société et Culture) and BMU Labs for funding this research.

Author information

Authors and Affiliations

  1. Heron Lab, University of Montreal, Montreal, Canada

    Mohamed S. Benlamine, Aude Dufresne & Claude Frasson

  2. Ste-Justine Hospital Research Center, Montreal, Canada

    Miriam H. Beauchamp

Authors
  1. Mohamed S. Benlamine
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Aude Dufresne
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Miriam H. Beauchamp
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Claude Frasson
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mohamed S. Benlamine.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Benlamine, M.S., Dufresne, A., Beauchamp, M.H. et al. BARGAIN: behavioral affective rule-based games adaptation interface–towards emotionally intelligent games: application on a virtual reality environment for socio-moral development. User Model User-Adap Inter 31, 287–321 (2021). https://doi.org/10.1007/s11257-020-09286-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11257-020-09286-0

Keywords

Search

Navigation