Keywords

1 Introduction: Angest, a VR Walking Simulator

Playing a game is an interactive experience. Due to its highly interactive nature there are a series of inputs that the players address to the system. The system, by its turn, process all the players inputs, providing, as response, the outputs that the they should decode. Being able to give inputs means that the player has agency. Being an experience it means that it is more than just the sequence of the small events that compound it, as a narrative it’s the sequence of the events that give meaning for the whole and for each of its steps, this characteristic can create a deep immersion in the experience. Because of these two aspects, the agency and the immersion, games can trigger strong social emotions. As any other media, while the person who is consuming it advance through the experience, it is possible that the events unfolded produces angry, happiness, sadness or a sense of belong. “Ultimately, I think the power of a game lies in its ability to bring us close to the subject. There is no other medium that has this power” [1].

The game genre is a key aspect for the immersion and evolvement of the player within the experience. Regarding the capability to construct an emotional bond between the player and the game characters, ‘walking simulator’ are gaining more and more popularity. A walking simulator is a genre of video game which lacks many of the traditional aspects of a game (such as a goal, win/loss conditions, any kind of game system to interact with) despite taking the form of a video game. Walking simulator are low pace games, covering topics about contemplation and exploration with some level of emotional evolvement from the player with the narrative [2].

The elements that defines if a game fits this category usually are too open and wide, what led to many different games being categorized as walking simulator. For that reason, some specialist even defends the term should be avoided [3]. However, as many terms in the mass culture, the term captures well the spirit of the experience despite its lacks of consistency. The game concept is not new, the first game as such date from the 1980’s, developed for ZX Spectrum computers, with the title The Forest [4], but walking simulators are gain lots popularity in the recently years (2010’s). Firewatch has sold more than 1 million copies [5] and Gone Home [6] have reached more than 250,000 downloads. Each and receive prizes and recognition from the game industry, particularly the specialized critic.

Walking simulators and Virtual Reality games have many common characteristics. Some of these similarities between the platform and the genre are: slow pace, immersive experiences and strong emotional bond. These elements are deliberated decided by the game designer, the professional responsible for defining all core elements of player experience and how well the game supports and provides the type of fun players want to have [7]. To do so, game designers had a set of tools that can be used to provide a good and immersive experience for the players: avatars, first person view, non-players characters and plot twists in the narrative are some of the most common used resources [8]. How successful a game can be in exploring these tools is correlated with the quality of the player experience. Since Virtual Reality is a computer-generated self-contained digital world, consequently, Virtual Reality in a game is another powerful resource for the game designer to provides a richer and deeper player experience for the player because the entire “world” that the game is taken place is under his/her control.

Furthermore, it is possible to say that even the strengths of the walking simulators and the Virtual Reality are so complementary that they obfuscate their weakness. For example, ‘slow pace’ is one of the main critic for walking simulators, but fast pace is one of the main challenges in VR because too fast pace leads to motion sicknesses, to avoid it, low pace experiences is a recommendation [9]. Other aspect is that walking simulators and VR are about immersion, the journey itself is more important than achieving a specific rank or beat an enemy [1]. Regarding the immersion, a deeper immersion is not dependent of hyper realistic graphics, some walking simulators make use of really simple 3D graphics called low-poly (from low number of polygons, consequently, low details and less hardware demand) and for VR it is an interesting option because its better rendered; high resolution graphics is a currently corner stone for VR performance. Besides been a performance improvement solution, low-poly is also an aesthetic, such as 8 bits games that started as a solution for technological constrains but became a deliberated choice for games. VR and walking simulators are about exploring a space and have an immersive experience, together and well combined they are strong resources for game designers develop meaningful player experiences.

Angest is a Virtual Reality game developed for Samsung Gear VR and distributed in the Oculus app store (Fig. 1). Samsung Gear VR is a headset in which some high-end Samsung mobile devices can be plugged in, transforming the device in a VR platform. Making use of the sensors of the mobile phone, such as accelerometer and gyroscope, it can provide a three degree of freedom VR experience (the user can see the all scenario that appear rendered in the three axes which simulate depth with high fidelity but it is not possible to walk in this scenario) [10].

Fig. 1.
figure 1

Screenshot of Angest.

Full size image

Angest is a Virtual Reality game that has many of the elements that characterizes itself as a walking simulator. In the game, the player control Valentina, a cosmonaut that perform a series of ordinaries activities in a spaceship with the company/guidance/vigilance of Konstantin, an artificial intelligent robot that interacts through voice and text information that appear in monitors fixed in different locations of the spaceship. Nonetheless, the game intentionally lacks lots of elements that normally compound the player experience as ‘fun to beat obstacles’, ‘intrinsic reward’, ‘increase workload’ and the premise that ‘humans need to be challenged’ [1]. Angest provide the opposite experience: there is no big obstacle, on the days that Valentina pass by on the spaceship there is nothing that imped or challenge her on her routine. The activities that are performed are intentionally boring and repetitive, no meaning or fulfillment felling can be extract from them. The narrative becomes more confusing while player advance on it and there is no skill to be mastered or any puzzle to be solved that can explain this confusion. During the game, the burden of a routine, its repetition and inevitability are not a fault or error in game design, in fact, they are exposed and explored as a narrative resource. More than choices the player is presented to open and wide questions that at first sight are addressed to the game character, Valentina, but are generic and intimate enough to be extended to the player.

The Angest particularities arouses some interesting issues. Since ‘meaningful choices’ is a well-known pillar of good game design, once the choices are not available for the player, do the other decisions in game design compensate this lack? If winning condition is a broadly used paradigm in the game industry, what about the player experience in a game that is not possible to win or to lose?… and for the research perspective, what is the definition of success in a game like this? All of these questions will be carefully addressed in the next item, methodology.

2 Constructing the Object and the Methodology

A game is an artefact, it means that a game is not a given thing, it is result of human practice. This practice itself can have a high or low degree of expertise, i.e., the level of domain of the skill necessary to perform a practice. A high degree of expertise is called proficiency. Concerning the practice of game development, we can point out two kinds of proficiency: ‘structural proficiency’ and ‘thematic proficiency’ [11]. The first, structural proficiency, is about the technical execution of the game: the performance of the software on a given hardware, the quality of graphics, absence of bugs, if the ‘gulfs of execution and evaluation’ [12] are consistent and if the game can sustain the ‘magical circle’ (the magical circle is a symbolic space, a context in which it is possible to blend fantasy and reality [13]). Structural proficiency is about the sharpness and capability of game development team to execute what was planned. The former, thematic proficiency, is about the quality of what was planned: gameplay, narrative, atmosphere, how the sound track is appropriate to game pace etc. Once we agree that it is necessary a well-done structure to sustain the “magical circle”, structural proficiency could be comprehended as the structure that maintain the circle and thematic proficiency is about what is inside the circle. For the objectives here underlined, the evaluation will focus on the thematic proficiency of the player experience.

An evaluation consists on extracting the relation between two things. Those things are identified as variable, the control of certain variables to inquiry the relation and causality of those independent variables with dependent variables make clear the relation between them. When we talk about Game evaluation, the most commons variables are: the player satisfaction and pleasure in play the game. The evaluation itself is called ‘playtest’: an experience of playing a game to evaluate it structural and/or thematic proficiency.

To proper evaluate the player experience it is necessary to have, maintain and update a formalized research procedure to conduct the playtest [14]. The objective of a playtest is to highlight and make evident player’s opinion and attitudes regarding the game being tested [15]. Playtest helps to validate if the player perception of the game corresponds to that what was planning by the game designer, it helps to check if this communication is fluid. To keep the research objectives pretty clear for the game development team, the researcher should make an document that summarizes all the issues regarding the game. This document is called ‘research plan’ and it describes the procures and the rationale behind the approach chosen for the playtest. To do so “research plan” must contain the research questions, the hypothesis and the overall methodology that was choose for this research.

2.1 Defining Research Questions

Defining research questions is the starting point of many researches. A research is conducted to answer the questions and doubts that the researches think is worth solving and reply. These questions help to clarify what is the challenge to be solved and why the research is being conducted, its object and what is the problem that need to be explained. Angest was developed by Black River Studio, the game division of Sidia, a Research and Development institute in Brazil [16]. The research was conducted by another division of Sidia, UX & Design, though the research was an in-house demand. Because the research team and the development team do not interact till the final playtest of Angest, we spent almost one week building an alignment to understand and keep clear all the doubts that should be addressed in playtest. The “clients” of the research were the game designer and game developer of Angest, thus it was conducted many rounds of interviews with the game designers and game producers in charge of the game and sporadic talks with other members of the development team.

Angest is a game that was planned and executed based on a series of assumptions that, by the perspective of the development team, justified the elaboration of the game. Exposing those assumptions was the first step to start the research plan. Many of them were related with the opportunity of walking simulators in a Virtual reality platform (as presented in item 2). Based on the decisions made for the game, the game designer and the producer wanted to confirm: (i) if the game fits in walking simulator game category for players that appreciate this game gender; (ii) if players are able to identify that the game has many ends and how does it motivate people to play the game again; (iii) if the game was deep enough to make the player reflect about his/her own life. With these clarification, it was defined that the test scope should addressed participants’ perception of game style, value and quality; game playability (mechanics, immersion, interactions, etc.) and overall game satisfaction; consequently, other topics that impact on the player experience were keep out of scope, such as acquisition process (search, download, etc.), Game set-up (install, paring controller, etc.) and adherence for players out of the target player profile or game niche.

2.2 Defining Research Techniques

Research techniques are like tools that the researcher chose in order to solve the research problem. As exposed previously, in this walking simulator, the reflection and questioning of the subject’s own condition, discomfort and anguish is part of the narrative. Heuristics are nor useful for walking simulators. The difficult aspect of evaluating walking simulators can be easily perceived when the researcher tries to apply a heuristic evaluation as the one proposed by Korhonen and Koivisto (2006) [17]. The game objective is not clear and, since it is not clear the end point or the victory conviction, tracking player progress is also complicated. Rewards, strategy and feeling of control (regarding structural proficiency) sometimes make no sense as well the questions related with repetitive, boring tasks or stagnation of the game. It implies that most of the legitimated and well-known game’s evaluation methodology and techniques do not fit the need of the faced challenge here because the satisfaction that walking simulator players pursuit is not exactly the same the one that most of the games usually offer (explosions, beat an enemy, solve a puzzle, etc.) To select out tools we kept in mind that subject approaches work better to identify attitudes and preferences of gamers, consequently it was expected that the interview after the playtest would be the main instrument to answer the research questions.

2.3 About Player Profile

Recruiting participants was a central aspect for test success, a short questionnaire was developed to guarantee it. Besides social demographic questions, the main question was about the game style of the participant. Walking simulators are strong biases games, in other words, or people hate or people love it [3]. The recruiting questionnaire asked the favorite games of the potential participant and if he/she had played some walking simulators, these questions showed to be appropriate to validate the player profile. As expected, most of the players that like walking simulators included them in their top three favorite games. Another characteristic to be confirmed was ‘previous experience playing VR’, we didn’t want non-VR players because the fascination with the technology should obfuscate some aspects and characteristics of the game. As a ‘nice to have’ we search for players that actively comment on forums and discussion groups about games, critical players that can be categorized as ‘extreme users’.

After the interviews with the game development team it was clear that the evaluation should focus on player satisfaction, players’ perception of game immersion, narrative and quality. In our research we had to take into account some aspects regarding the game playability and structural proficiency (mechanics, graphics, interactions, etc.) However, once the structural proficiency was considered as a secondary aspect for player satisfaction and most of its issues were validated by Sidia’s Q&A team, we decided not focus so much on these technical issues. Based on that, we design a couple of question for the user after he/she finishes the game session, those are the questions: (i) what was your first impression of the game? (ii) how easy to understand was the game; was it intuitive, interesting and immersive? (iii) was the game becoming easier to play while you were advancing on it? (iv) there was something confusing or that was not clear for you? (v) what would you change in the game? (vi) what should be maintained in a hypothetic new release of the game? (vii) did you think that the game could have another ending? (viii) what parts of the game do you liked the most… and what do you liked least? (ix) would you play this game again, why?

2.4 Running the Playtest

The playtest was set in São Paulo, during three days of August of 2017 in a VR arcade (Fig. 2). We had 14 participants from which 12 had validated data, that number had been proof to be appropriated for a qualitative research with a homogeneous group [5]. All participants were gamers with more than 8 years of ‘serious’ gameplay. They have had previous experience with more than one game console, read reviews online and follow games news. Based on the request and details provide by the game development team, the participants sample were fully representative of the game target audience.

Fig. 2.
figure 2

Playtest of Angest.

Full size image

The playtest followed the overall structure of most of user validation tests: a moderator and a facilitator welcomed the participants; they were requested to sign a nondisclosure agreement form; the instructions were presented for the participants; they played the game; there was the debriefing and then each participant was thanked for its participation. As described before, the test focused on the thematic proficiency of the game, so it was decided to do not use the “think aloud” verbal protocol because it could break the “magic circle”.

The planning, execution and report of the results took four weeks altogether. The methodological walkthrough was: (i) understand the challenge and define research questions; (ii) define the player profile; (iii) select methodologies; (iv) define a chronogram; (v) recruit participants; (vi) set-up playtest location, documentation and devices; (vii) run the playtest; (viii) collect the data; (ix) analyze the data; (x) partial report; (xi) final report and results presentation.

Until this point in this article we had presented and discussed the main issues from the items ‘i’ to ‘vii’. In the next part, we are going to present the evaluation results and then discuss, based on our experience with Angest, the effectiveness of the research technique chosen, its weakness and strengths, when evaluating walking simulator VR games.

3 Findings

Here we are going to present the findings of the playtest as input to discuss the efficiency and efficacy of proposed methodological approach to proper respond how to evaluate a virtual reality game experience.

The responses given in the debriefing session were recorded for late being tabulated for the analysis of player experience. On average playtest sessions lasted 60 min. Maximum gameplay was 124 min, minimum was 22 min. From 20 participants, 10 finished the game during the playtest. The data from debriefing could be analyzed through a bottom-up or top down approach. In the first one, the categories of analysis were defined before the data consolidation (there are some categories in game evaluation that enables it approach). Due the intrinsic complexity of the game and the particularities of VR and walking simulators, it was decided to user a bottom-up categorization. Players comments were categorized in the following categories: The debriefing session was helpful to identify user satisfaction and to respond the research questions. There were no critical or major bugs in the playtest what justified the choice of evaluating the thematic proficiency with less focus on the structural proficiency. The players enjoyed the Russian retro futuristic atmosphere, the look and feel of the characters (Kostantin and Valentina) and the philosophic nihilist atmosphere. The sounds effects were an important element in the experience and player satisfaction as well.

The participants pleasure the scenario, interactions and the activity outside the spaceship. The movements in the game was by teleport points: the player was located in one point at time, he/she can’t move around the scenario step by step, when he/she can move by gazing in some specific points that works as teleport anchors, the gaze has a loading, it fade-out and them fade-in in the new position. It was not high pleased by all the participants however it was not disappointing. In fact, it was a game design decision to avoid motion sickness, since no player reported it, the decision has been shown correct. In Aeroponics (a scenario of the game) the leaves grew in same sequence what is not the common behavior of a plant; when the player change water PH it doesn’t change anything (the plants wouldn’t die or grow faster). In VR, it is important that everything that look real and afford to behave as something in the real world should have to behave this way, if not, there should be a good explanation.

About gameplay, the was some minor problems. In players opinion, there was some information that was not explicit enough and it was not possible to retrieve the instructions from Konstantin. A dyslexic player informed that it was had trouble to read the ‘russian-nish’ typeface that was used to spaceship signalization. The storytelling was another good point of the game, the small hints about Valentina nature and the nightmares were well-used storytelling resources. Nightmares were like another virtual reality inside the virtual reality of the game, it was a good way to present new scenarios for a narrative that happens inside a spaceship. The tests that Valentina should perform during her routine were a good way to address the philosophical existentialist questions to the player. However, it’s know that walking simulators are criticized for its low pace, but in Angest there were some critics about the narrative being too fast, the players also would like to have more logs to read and more tests to perform. Some players realize that the game has different ends based on the choices made in the test and the way that the interaction with Konstantin were build. According with the give feedback, it encourages most of the players to play the game again to confirm this assumption.

Immersion is a key feature in VR and in walking simulators and Angest had proof that explored it well. Samsung Gear VR is the least potent of the main commercial VR hardware. The processor of Gear VR is the mobile phone and even the most powerful ones are far behind of other VR computers or video games. Even so, there were lots of feedbacks saying that the game was as good as any other VR games and far better than the average VR mobile game. The overall evaluation of the game was positive, besides the problems with some instructions there were no complains. The doubts were about specific points in the narrative that were intentionally open to interpretation.

Nonetheless, the assumptions were not fully validated or refuted. About the first one, some of them need a more deeply research. Angest was clear recognized as a walking simulator, the participants really appreciated the graphics, sounds and the atmosphere of the game. The mechanics and the characters were much praised. The first interaction at the lab was the unique pain point but it didn’t interfere in the engagement or immersion. Some users reported that they suppose that the game could have many ends and that the choices during the game could determine which will be the end. However, this point was not clear for all. The participants that had this guess reported that they probably will play the game again.

Now that the game had been released it is possible to confirm that most of the topics related to the thematic proficiency were high praised, even for those that don’t appreciate the game itself, these players recognized that they were not the target profile and that the game was a niche game [19, 20]. The game has a score of 4.4 of 5 in Oculus play store. One objective aimed by the game development team was that through the game the player could reflect about his/her own life. Some players recognized that the questions were in fact addressed to them, but they did not reflect too much about it. Comments on Oculus store reinforce many of the findings presented here.

4 Conclusion and Next Steps

VR games evaluation is not a simple thing. It’s not simple because it’s not well-known. It’s not well-known yet nor because it’s complicated but because it’s new. Although, there are many procedures that can be used from other areas as game evaluation and user experience evaluation to help in this effort. For example, as in most of the social science researches, the methodology used to evaluate our object (player satisfaction of the game Angest) was constructed based on research questions that identified by the research team together with the “client” (game development team). Most of the research questions and doubts were directly related with the player satisfaction and subjective perception of the plot and the overall experience. Besides that, there were identified other questions and issues that could impact the player experience. Other point that was stressed by the game development team was the category of the game, a walking simulator, and many choices in game design due to VR potentialities and limitations.

After the research questions were identified, the next challenge was to choose a methodology that could handle all these issues. The usage of a research document to organize the test, detail its structure and deliverables had been proved good enough to map the player satisfaction and to reply most of the doubts. The debriefing session was a key point and open questions design specific to the game proposal had worked well with the players. The feedback was natural and riche. The participants profile was very specific and the research focused on the thematic proficiency of the game, with this focus, the number of participants was adequate, as indicated by most of the bibliography about qualitative research. Because there are no clear, specific and dedicated categories to evaluate VR walking simulators, open questions are more indicated than closed questions. The experience of playing a deeper VR game provides complex emotions in the players and enables each one to construct its own perspective of that experience. That’s why bottom-up analyses are recommended in this kind of evaluation.

The test set-up for VR games is another technical challenge, specially one running in Samsung Gear VR. The hardware sometimes overheats and shutdown the device, disturbing the experience and the test. It didn’t happen in this playtest because of the quality of software but it is a common constrains. There was the possibility to cast the screen during the game session to a TV but it is not a simple set-up and there was concerns about hardware and battery performance. We decided to drop out this possibility to reduce the attrition in the experience.

The feedback from the development tem about the evaluation was positive. There was a lot of assumptions that could be confirmed and the report was used to justify some market and promotion decisions. The choice to develop a walking simulator for VR, a huge risk at first sight since walking simulator still niche games, had been confirmed as a good strategic decision. The comments on Oculus store and other specialized reviews where very similar and reinforced the research findings. The research also helped to set the expectation of game success to the high manager team.

This kind of research demands planning, organization and mainly the alignment between the development and the research team. It was extremely necessary that the research team understood the game proposal, mechanics and the intentions of its creators. VR is not as familiar as many other platforms as desktop and mobile but it is growing fast in many different areas, not only games. It is important that games and UX researches be prepared to deal with this technology and its characteristic and we hope that this article helps new VR researches to be prepared for it. Our perception is that when the object of research is a new challenge in a given field the first procedure is to return to the basis of the field, review its fundamentals and check how relevant they remain. In case, we selected techniques from the User Experience and from Game Design area. There is a double gain with this strategy, (i) first, however the basis of a field sometimes is not enough to exhaust all the doubts in a research, they are open and generic enough to enable the research to start to address the problem; (ii) second, it is possible to clarify how much the object of the research still belonging to that given field of knowledge and; (iii) if this relation is too weak, keep it clear the need of news approaches and methods. Human-computer interaction provides guidance and tools to start to approach any object that emerge from the interaction between a human and a computer. It’s possible to apply it from classic computer desktop interactions to Virtual Reality and wearable devices. About the second and third topic, one clear example is the arise of other specific areas of the human-computer interaction in recent years, such as User experience and Player experience with focus on digital games. There are still relations between these new areas of knowledge and the one from where they emerged, they have a common ground at the same time that each one justifies itself as a proper research area due its particularities. The know-how of the former field is necessary for the new areas, yet it is not sufficient to solve the new challenges. Probably, in a near future, VR game evaluation will have it owns methods and apparatus. Till there, researches like this depurates legitimated methods and apparatus of co-related fields to help this construction.