Keywords

1 Introduction

Emotion is an important psychological indicator that can affect human cognition and decision-making. Emotional disability as a serious mental disorders is concerned by psychiatrists widely.

As we all know, games can regulate emotion, making players stressed, excited or happy. With the recent development of computer technologies, new VR games can bring the players a more realistic experience that is useful in the field of psychological therapy for the treatment of anxiety disorders, phobia, etc. For instance, Rothbaum et al. have successfully used VR technologies typical imaginal exposure treatment for Vietnam combat veterans with posttraumatic stress disorder (PTSD) [1].

In this paper, we study whether audio-based VR games can be effectively used for emotion regulation. To this end we designed and implemented the first 3D-audio-based VR game that aims at emotion regulation, called “Escape from the Dark Jungle”. We recorded the player’s emotion before and after playing game, and use it to evaluate the game’s effectiveness. Our main contributions include:

  1. 1.

    A novel 3D-sound-based VR game designed for emotion regulation;

  2. 2.

    An extensive user study that evaluates the effectiveness of the game.

2 Related Work

2.1 Emotion and Emotion Regulation

Following the “ecological rationality” theory proposed by Gigerenzer and Todd [2], extensive research has been conducted on emotions and decision making [3, 4]. Research show that emotion can influence our decision-making behavior to a great extent. In a study related to emotion and economic decision-making [5, 6], Harle and Sanfey asked subjects to participate UG tasks after induced positive, negative and neutral emotions. The results showed that subjects in the sad level had lower acceptance rates of unfair offers than others. On the other hand, when the subjects were induced to the four kinds of emotions are amusement (positive, approach), serenity (positive, withdrawal), anger (negative, approach), disgust (negative, withdrawal), emotional impact on the decision-making has more significant effect.

In the field of emotion regulation, Panney Baker’s paradigm [7] on emotional storytelling is a method of regulating strong negative emotions. It requires the subjects to make self-disclosure of emotional volatility using writing. This emotional storytelling has a quite strong long-term positive effect. Besides, in contrast to less hostile people, a highly hostile person (an individual who is difficult to manage emotions) exhibits a more aggressive immune response, and people with high levels of alexithymia (difficult to recognize and understand emotions) experience more benefits than those with low levels of alexithymia [8]. Those who cannot handle emotional events in their lives will benefit most from Panney Baker’s paradigm.

2.2 Games for Emotional Regulation

There exist a few games for emotional regulation. These games mainly target on people with special needs, such as regulating emotion of adolescents with emotional disorders like depression or anxiety disorder [9, 10]; regulating the eating mood of people with anorexia or bulimia eating disorder or treatment of other neuropsychiatric disorder [11, 12]. Other games are used to train players’ emotional self-regulation through biofeedback, in order to achieve better job performance [13,14,15].

Despite all these efforts, there are not many emotion regulation games that target on regular users. To the best of knowledge, there is no emotion regulation games that use “Stimulus - Response” theory of behavioral method as we do in this work.

2.3 Psychoacoustics and Immersive Audio Technology

Psychoacoustics studies have measured the human perception of sound using psychophysical method, including loudness level, tone, direction hearing, hearing sensitivity, etc. [16] The most typical characteristics of sound perception include the binaural effect and the auricle effect. The binaural effect refers to the ability of localizing the sound source position with two ears. The auricle effect refers to the ability to determine the sound source position assisted by the shape characteristics of the auricle. Immersive audio technology is an application of psychological acoustics research, which uses head-related transfer function to render real-time 3D surround sound [17, 18]. Nowadays, there are many low-cost real-time 3D rendering technologies that allow users to enjoy immersive 3D audio though regular headsets.

3 Game Design

Our goal is to design an audio-only game that has no visual elements. Compared with a regular VR game that contains both visual and audial components, we believe a sound-only game could have the following unique advantages on emotion regulation:

  1. 1.

    The amount of visual information in a regular VR game could be overwhelming. In contrast, a sound-only game leaves a much greater degree of freedom to players’ imagination. It can even wake the players’ subconscious like hypnosis, bringing greater emotional impact on them.

  2. 2.

    Creating realistic VR game is expensive and time-consuming. In contrast, low-cost real-time 3D audio rendering technology based on psychoacoustics has been available for years.

  3. 3.

    Rich visual information may introduce too many uncontrollable factors in our user study. It is instead easier to control and evaluate only sound information for emotion regulation.

3.1 Scenario Design

Considering the above reasons, we designed a game scenario in which the player plays the game only by listening in the dark. In this game, the player falls into a dark jungle due to a plane crash, with temporary blindness caused by injury. The player needs to explore the jungle and find a way to escape from it. The start interface of this game is shown in Fig. 1. The player can hear different sounds based on his/her location in the jungle, and make real-time navigation decisions based on the sound he/she is hearing.

Fig. 1.
figure 1

Start Interface of “Escape from the dark jungle”

Full size image

3.2 Core Play Rule

To play the game, the player sits alone in the dark, wearing a headset and uses a game paddle, as shown in Fig. 2. The player uses the joystick on the game paddle to move north/south/east/west on the virtual map. When the player moves to a specific location on the map, some sound sources that are nearby will be triggered to play through the headset, such as intensive explosion, beast roaring, wind and rain, farm dog barking, etc. The player needs to move away from the dangerous sounds, move towards safe sounds to find a safe path to escape from the jungle. In the process of the game, the player relies mainly on his/her own exploration to make progress, and the system will give them a little voice guidance when necessary (e.g. stuck at the same location for too long). During the game, the system will automatically record the players’ trajectory and all the sound sources that are triggered. The game operation flow is shown as Fig. 3.

Fig. 2.
figure 2

Players play with a headset and game paddle

Full size image
Fig. 3.
figure 3

Game operation flow chart of “Escape from the dark jungle”

Full size image

3.3 Audio Material Selection

This Game requires a lot of sound sources with positive and negative attributes, which need to be carefully selected and prepared.

Firstly, we selected 88 original sounds from an audio material website [sc.chinaz.com], which are suitable for our game scenario, like explosion, fire, jungle, farm, etc.

Next, we invited seven subjects to evaluate these sound pieces from two aspects: recognizability and positivity. The subjects gave a score from 1 to 7 for each aspect. Based on the evaluation results, we deleted sounds that could not be recognized by more than half of the subjects. 44 sounds are removed in this step. For the remaining ones, we mark those sounds whose average positivity score is more than 5 as positive sound, less than 3 as negative sound and between 3 and 5 as neutral sound. The results of audio material selection experiment are shown in Table 1.

Table 1. Results of audio material selection
Full size table

3.4 Map Design

We designed three types of audio maps: Positive, Negative and Neutral. The Map design principle is based on the behavioral method of the “Stimulus - Response” theory.

Behaviorists believe that human behavior and emotion heavily depends on the feedback form the environment. People feel good when the results match with their expectations, and feel bad otherwise. In our game, the positive sounds represent positive feedback, encouraging players to continue with their current movement, while the negative sounds are negative feedback that urges the players to change their current moving pattern.

For the design of the Positive Map, we follow the “Stimulus-Response” pattern of behavior: when the player triggers a positive sound, he/she will receive positive feedback, meaning that the player will encounter more positive sounds. When the player triggers a negative sound and dodge, he will avoid more negative sounds. In this way, the player can gradually build up confidence during the game and their emotion will hopefully become more positive. On the other hand, the Negative Map is completely opposite to the “Stimulus-Response” behavior mode. If the player triggers a positive sound and decides to continue to move forward, a negative sound will be triggered. If the player triggers a negative sound and dodges, he/she will trigger more negative sounds. This will force the player to make unnatural decisions constantly and their emotion may be negatively affected. Besides, the Neutral Map is designed with random sound distribution.

The size of the audio map is 200 × 200, the center of the map is the player’s starting position. The audio map, as shown in Fig. 4, is divided into four areas: the Explosion Zone (red), the Vacuum Zone (yellow), the Battle Zone (green) and the Safe Zone (blue). The number of three types of sound sources in the map is shown in Table 2. The Explosion Zone is where the crash happens, and the second layer is a vacuum region due to radiation of the explosion. The third layer is the battle area full of beasts from the vacuum area, and the most outer layer is the safe zone with some villages and farms. The player wins the game by stepping into the Safe Zone and triggering the first positive sound source.

Fig. 4.
figure 4

Audio Map of “Escape from the dark Jungle” (Color figure online)

Full size image
Table 2. Number of three sound sources in the map
Full size table

The Neutral Map in Fig. 5, the Positive Map in Fig. 6, the Negative Map in Fig. 7, are all designed based on “Stimulus - Response” behavioral theory and design.

Fig. 5.
figure 5

Audio source distribution of the Neutral Map

Full size image
Fig. 6.
figure 6

Audio source distribution of the Positive Map

Full size image
Fig. 7.
figure 7

Audio source distribution of the Negative Map

Full size image

4 Evaluation

We conducted a user study to find out whether or not our VR Game can effectively regulate players’ emotion. We set up our system at a five-day university art exhibition, and invited people who were attending the exhibition to play our game. We recorded their emotions both before and after playing game for analysis.

As shown in Table 3, we arrange a 2.5 m * 1.5 m dark space enclosed by black cloth. A computer with a 28-in. display is placed in the dark room. A regular Philip headset and a Sony PlayStation4 game paddle are used for the game play.

Table 3. List of equipment and materials
Full size table

When a subject enters the dark space, the display shows simple instructions on how to play the game, and reminds the player to keep the headset on all the time when playing the game. After that, we conduct an emotion test on the player before starting the game. Next, the player watches the opening CG and starts the game adventure using the game paddle. After the player finishes the game, we conduct another emotion test on this player, and save all the playing history and emotion test results. The experiment flow chat is shown in Fig. 8 and the instruction audio source list is shown in Table 4. In the emotion test, we use the core emotions concise measure map revised by Barrett and Russell [20]. Our emotion test interface is shown in Fig. 9. Every subject takes about 10 to 15 min to complete our user study.

Fig. 8.
figure 8

Experiment flow chat

Full size image
Table 4. Instruction sound source list
Full size table
Fig. 9.
figure 9

Emotion test interface.

Full size image

We recruited 150 participants from the visitors to the art exhibition. Among them, 14 subjects did not complete the game, therefore their data are excluded. In the remaining 134 game sessions, 29 used the positive sound map, 56 used the negative sound map and 49 used the neutral map. We did not record their gender nor their age. Table 5 shows the mean and standard deviation of the 134 pre and post emotion test.

Table 5. Emotion test results
Full size table

As shown in Figs. 10 and 11, we compared the emotional status of the participants before and after playing the game. According to a 2D emotion model [19], the emotional status can be divided into two dimensions of valence and arousal. We found that in games using negative and neutral sound map, the emotional status of the participants significantly changed after the game play. The scores of our participants’ emotional status in both dimensions were significantly lower after the playing negative map game (Valence: M before  = 529.55, SD before  = 30.43, M after  = 434.82, SD after  = 33.67, t(55) = 2.089, p = 0.041; Arousal: M before  = 1367.29, SD before  = 38.57, M after  = 1265.32, SD after  = 36.45, t(55) = 2.002, p = 0.05). This result indicates that participants were less pleasant and calmer after playing the game. When playing the neutral map game, the emotional arousal scores were significantly lower after the game (M before  = 1445.20, SD before  = 49.60, M after  = 1228.67, SD after  = 41.05, t(48) = 4.294, p < 0.001), and there is no significant difference before and after the game in emotional valence (M before  = 452.76, SD before  = 32.44, M after  = 379.29, SD after  = 34.16, t(48) = 1.631, p = 0.109). No significant difference is observed in both dimensions of valence and arousal after playing the positive map game (Valence: M before  = 443.97, SD before  = 48.68, M after  = 455.69, SD after  = 46.13, t(28) = −0.180, p = 0.858; Arousal: M before  = 1337.41, SD before  = 38.47, M after  = 1361.90, SD after  = 55.78, t(28) = −0.401, p = 0.692).

Fig. 10.
figure 10

Emotional valance change. (* represents p < 0.05 in the paired sample t-test)

Full size image
Fig. 11.
figure 11

Emotional arousal change (* represents p < 0.05 in the paired sample t-test, *** represents p < 0.001 in the paired sample t-test)

Full size image

We conducted an ANOVA using the score of emotional arousal as the dependent variable, and game and map type as independent variables, as shown Fig. 12. We found a very significant effect on emotional arousal before and after game (F(1, 131) = 9.155, p = 0.003, η2 = 0.065). The participants were calmer after playing game in all types of maps. We did not find the main effect of the map type (F(2, 131) = 0.230, p = 0.795, η2 = 0.003). The connection between emotional arousal and the map type was significant (F(2, 131) = 4.155, p = 0.018, η2 = 0.06). The effect of the negative and neutral map types was very significant (F negative (1, 131) = 4.486, p = 0.036, η2 = 0.033; F neutral (1, 131) = 17.701, p < 0.001, η2 = 0.119). No significant difference was observed for the positive map type (F positive (1, 131) = 0.134, p = 0.715, η2 = 0.001).

Fig. 12.
figure 12

Map comparison

Full size image

Figure 10 shows emotional valance change before and after playing game with different map type. From this figure, we can get following conclusions:

  1. (a)

    Paired-sample t-test on the changes in emotional valance before and after playing game, the results shows:

    1. i.

      The emotional valance change of positive map shows no significant difference before and after playing game, t(24) = −1.140, p = 0.266.

    2. ii.

      The emotional valance change of negative maps shows significant difference before and after playing game, t(42) = 2.287, p = 0.027. Which means, after playing game with negative map, the player’s mood was significantly worsen.

    3. iii.

      The emotional valance change of the neutral map shows no significant difference between before and after playing game, p = 0.109.

  2. (b)

    Two-way ANOVA results show that,

    1. i.

      The main effect of emotional valance change was not significant, F(1, 113) = 1.132, p = 0.290, η2 = 0.01. This shows that there was no positive change nor negative change regardless of the map type, which is basically in line with our expectation because positive map, neutral map and negative map can offset the emotional change.

    2. ii.

      The main effect of map type was not significant F(2, 113) = 0.701, p = 0.498, η2 = 0.012. This result shows that playing game with the three types of map were basically similar, which means the influence on players’ emotion was relatively homogeneous.

    3. iii.

      Interaction between the emotional valance and map types was Significant, F(2, 113) = 2.703, p = 0.071, η2 = 0.046. However, when analyzing each simple effect, we found that playing with negative maps could significantly decrease the valance scores of players, which made the players’ emotion more negative. Besides, the neutral maps and positive maps had little effect on the emotional valance of the players.

Figure 11 shows emotional arousal change before and after playing game with different map type. From this figure, we can get following conclusions:

  1. (a)

    Paired-sample t-test on the changes in emotional arousal before and after playing game, the results shows:

    1. i

      The emotional arousal change of the positive map shows no significant difference before and after playing game, t(24) = −1.437, p = 0.164.

    2. ii

      The emotional arousal change of the negative map shows no significant difference before and after playing game, t(42) = 1.661, p = 0.104.

    3. iii

      The emotional arousal change of the neutral map shows the extremely significant difference between before and after playing game, t(47) = 4.307, p < 0.001.

  2. (b)

    Two-way ANOVA results show that,

    1. i

      The main effect of emotional arousal change was significant F(1, 113) = 4.393, p = 0.038, η2 = 0.037. The emotional arousal score of players was significantly decreased after playing the game. This result shows that the emotion of players became calmer after playing the game regardless of map type.

    2. ii

      The main effect of the map type was not significant F(2, 113) = 0.617, p = 0.542, η2 = 0.011. This shows that the emotional arousal level change after playing game with three different map type was basically similar, which means the influence on players’ emotion was relatively homogeneous.

    3. iii

      The interaction between map types and emotional arousal was extremely significant F(2, 113) = 5.496, p = 0.005, η2 = 0.089. This shows that the impact of different maps on the player’s emotional arousal was different. when analyzing each simple effect, we find that there was no change on the players’ emotional arousal before and after playing game with the positive map and the negative map, that is, the player always maintained a high emotional arousal (continuous excitement), while the neutral map significantly decreased the player’s emotional arousal. The players became calmer after playing game with neutral map.

We conducted another ANOVA using the score of emotional valence as dependent variable and game and map type as independent variables. No significant difference was observed in the main effect of emotional valence before-and-after game (F(1, 131) = 3.033, p = 0.084, η2 = 0.023) and with the map type (F(2, 131) = 1.886, p = 0.156, η2 = 0.028). There is no significant difference in the interaction between the two independent variables (F(2, 131) = 1.007, p = 0.368, η2 = 0.015).

In order to exclude the impact of the game itself (such as game is boring, too difficult and so on) on the players’ emotion, we use the neutral map (random distribution of sound source) as a reference. We compare the positive map and the negative map with the neutral map respectively, to observe the impact of the positive map and negative map on the players’ emotion.

Figure 13 shows emotional valance change before and after playing game with the positive map and the neutral map;

Fig. 13.
figure 13

Valance comparison between positive map and neutral map (* represents p < 0.05 in the paired sample t-test)

Full size image

In this figure, an independent sample t-test was conducted on the positive map and the neutral map, which illustrates the following conclusions:

  1. (1)

    Before playing game, there was no significant difference in emotional valance between the positive map and neutral map, t(71) = −0.348, p = 0.729.

  2. (2)

    After playing game, Emotional valance of the positive map and the neutral map was significantly different, t(71) = 2.025, p = 0.047.

Figure 14 shows emotional arousal change before and after playing game with the positive map and the neutral map;

Fig. 14.
figure 14

Arousal comparison between positive map and neutral map (* represents p < 0.05 in the paired sample t-test)

Full size image

In this figure, an independent sample t-test was conducted on the positive map and the neutral map, which illustrates the following conclusions:

  1. (1)

    Before playing game, there was no significant difference in emotional arousal between the positive map and neutral map, t(71) = −1.418, p = 0.161.

  2. (2)

    After playing game, Emotional arousal of the positive map and the neutral map was significantly different, t(71) = 2.611, p = 0.011.

Figure 15 shows emotional valance change before and after playing game with the negative map and the neutral map;

Fig. 15.
figure 15

Valance comparison between negative map and neutral map

Full size image

In this figure, an independent sample t-test was conducted on the negative map and the neutral map, which illustrates the following conclusions:

  1. (1)

    Before playing game, there was no significant difference in emotional valance between the negative map and neutral map, t(89) = 0.833, p = 0.407.

  2. (2)

    After playing game, here was no significant difference in emotional valance between the negative map and neutral map, t(89) = −0.005, p = 0.996.

Figure 16 shows emotional arousal change before and after playing game with the negative map and the neutral map;

Fig. 16.
figure 16

Arousal comparison between negative map and neutral map

Full size image

In this figure, an independent sample t-test was conducted on the negative map and the neutral map, which illustrates the following conclusions:

  1. (1)

    Before playing game, there was no significant difference in emotional arousal between the negative map and neutral map, t(89) = −1.040, p = 0.301.

  2. (2)

    After playing game, here was no significant difference in emotional arousal between the negative map and neutral map, t(89) = 0.431, p = 0.667.

Figure 17 shows emotional valance change before and after playing game with the positive map and the negative map;

Fig. 17.
figure 17

Valance comparison between negative map and positive map (* represents p < 0.05 in the independent sample t-test)

Full size image

In this figure, an independent sample t-test was conducted on the positive map and the negative map, which illustrates the following conclusions:

  1. (1)

    Before playing game, there was no significant difference in emotional valance between the positive map and negative map, t(66) = −1.016, p = 0.314.

  2. (2)

    After playing game, Emotional valance of the positive map and the negative map was significantly different, t(66) = 2.188, p = 0.032.

5 Discussion

Finding 1:

This VR Game makes players more calm, especially using the neutral map.

As our common sense, we typically think that the game would make players exciting, thus the emotional arousal score should increase after playing game. However, our user study suggests that our VR game works in an opposite way: it calms the players down significantly. We think the main reason is that our VR game is not competitive and there is no failure punishment. We believe that other games with a similar design would effectively make the player calmer. It is also possible that the dark environment of the audio game also makes players calmer.

We also find that using the positive map, the players’ emotional arousal score does not decrease. This suggests that the Positive map is effective to keep players’ excitement level during the game.

Finding 2:

The negative map makes players feel obviously worse, while the Positive Map makes people feel slightly better.

Our user study results show that the negative map can significantly reduce the players’ emotional valence score. In contrast, the positive map makes people’s emotional valence somewhat better, though not significantly. This finding is largely in line with our experimental hypothesis. The reason why the effect of the positive map is insignificant is inconclusive. One possible explanation is that the game is not challenging enough to play using the positive map, so it does not have a strong impact on the players’ mood.

6 Conclusion

In this paper, we introduce a novel 3D-sound-based VR game that is designed to regulate players’ emotion, based on the behavioral method of the “Stimulus - Response” theory. We also evaluate the effectiveness of this design method through a user study. Although our study is preliminary, the results suggest that we can potentially influence the plays’ emotion purposefully through specially-designed audio games, so that negative players can be changed to positive and excited players can calm down. In the future, we plan to design and compare more types of games, and to find more effective design method to regulate players’ emotion.