Abstract
This chapter is dedicated to the use of motion-sensing controllers in video games, with particular attention to gameplay mechanics that link body movement, sound, an music. First, we present a review of the state of the art. This includes an overview of recent motion controllers describing the different motion-sensing technologies employed and the implications such technologies have on game design and gameplay. This is followed by a set of examples of relationships between motion control, sound, music, and gameplay mechanics in recent video games. Secondly, we present a survey on recent research in expressive movement analysis and motion-based interaction, introducing the concepts of motion descriptor and parameter mapping. We then report on two serious games designed for children affected by autism using full-body motion capture and emotion recognition, and studies of player engagement in motion-controlled video games. In light of the interdisciplinary research presented, we finally discuss perspectives for motion-based game design.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
This calibration procedure suggests that direction tracking is achieved by using drift-corrected orientation data obtained from the built-in IMU. A similar calibration procedure was previously used also in games employing the Wii MotionPlus expansion for the Wii Remote.
- 2.
- 3.
From the video trailer of the game (at 00:13) on Die Gute Fabrik Vimeo channel: https://vimeo.com/31946199.
- 4.
References
Alaoui, S.F., Caramiaux, B., Serrano, M., Bevilacqua, F.: Movement qualities as interaction modality. In: Proceedings of the Designing Interactive Systems Conference, pp. 761–769. ACM (2012)
Bernardo, F., Arner, N., Batchelor, P.: O Soli Mio : exploring millimeter wave radar for musical interaction. In: Erkut, C. (ed.) NIME’17 International Conference on New Interfaces for Musical Expression, pp. 283–286, Copenhagen (2017). http://homes.create.aau.dk/dano/nime17/papers/0054/paper0054.pdf
Bianchi-Berthouze, N.: Understanding the role of body movement in player engagement. Hum. Comp. Interact. 28(1), 40–75 (2013). http://www.tandfonline.com/doi/abs/10.1080/07370024.2012.688468
Brown, D., Renney, N., Stark, A., Nash, C., Mitchell, T.: Leimu: gloveless music interaction using a wrist mounted leap motion. Proc. Int. Conf. New Interf. Mus. Expr. 16, 300–304 (2016). http://www.nime.org/proceedings/2016/nime2016{\_}paper0059.pdf
Burges, C.J.C.: A tutorial on support vector machines for pattern recognition. Data Min. Knowl. Disc. 2, 1–43 (1998)
Camurri, A., Mazzarino, B., Ricchetti, M., Timmers, R., Volpe, G.: Multimodal analysis of expressive gesture in music and dance performances. In: Camurri, A., Volpe, G. (eds.) Gesture-based Communication in Human-Computer Interaction, pp. 20–39, vol. 2915, Springer, Berlin (2004). http://link.springer.com/10.1007/978-3-540-24598-8{\_}3
Camurri, A., Coletta, P., Demurtas, M., Peri, M., Ricci, A., Sagoleo, R., Simonetti, M., Varni, G., Volpe, G.: A platform for real-time multimodal processing. In: Proceedings International Conference Sound and Music Computing, pp. 11–13 (2007). http://www.smc-conference.org/smc07/SMC07Proceedings/SMC07Paper59.pdf
Caramiaux, B., Françoise, J., Schnell, N., Bevilacqua, F.: Mapping through listening. Comput. Music J. 38(3), 34–48 (2014)
CBS Interactive: Dance central for Xbox 360 reviews - Metacritic (2017a). http://www.metacritic.com/game/xbox-360/dance-central
CBS Interactive: Just dance for Wii reviews - Metacritic (2017b). http://www.metacritic.com/game/wii/just-dance
Champy, A.: Elements of motion: 3D sensors in intuitive game design. Analog. Dialogue. 41(2), 11–14 (2007). http://www.analog.com/en/analog-dialogue/articles/3d-sensors-in-intuitive-game-design.html
Collins, K.: Game Sound : An Introduction to the History, Theory, and Practice of Video Game Music and Sound Design. MIT Press (2008). https://mitpress.mit.edu/books/game-sound
Cook, P.: Principles for designing computer music controllers. In: Proceedings of the International Conference on New Interfaces for Musical Expression 2001 (NIME 2001), pp. 3–6 (2001). http://www.nime.org/proceedings/2001/nime2001{\_}003.pdf. http://dl.acm.org/citation.cfm?id=1085152.1085154
Die Gute Fabrik: Presskit — Johann Sebastian Joust (2014). http://www.jsjoust.com/presskit/
George, R.: Behind the Scenes of Zelda: Skyward Sword (2011). http://uk.ign.com/articles/2011/11/11/behind-the-scenes-of-zelda-skyward-sword
Glowinski, D., Dael, N., Camurri, A., Volpe, G., Mortillaro, M., Scherer, K.: Toward a minimal representation of affective gestures. IEEE Trans. Affect. Comput. 2(2), 106–118 (2011). http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=5740837. http://ieeexplore.ieee.org/document/5740837/
Godøy, R.I., Leman, M. (eds.): Musical Gestures: Sound, Movement, and Meaning. Routledge (2010)
Google ATAP: Project soli (2017). https://atap.google.com/soli/
Halston, T.: New PS4 camera vs old PS4 camera comparison what’s the difference? (2016). https://www.psu.com/news/31081/New-PS4-camera-vs-old-PS4-camera-comparison---whats-the-difference
Holz, D.: Leap motion sets a course for VR (2014). http://blog.leapmotion.com/leap-motion-sets-a-course-for-vr/
Hunt, A.D., Wanderley, M.M., Paradis, M.: The importance of parameter mapping in electronic instrument design. In: Casey, C., Schneider, K., Hammond, E. (eds.) Proceedings of the 2002 Conference on New Instruments for Musical Expression (NIME-02), pp. 88–93, Dublin (2002). http://www.nime.org/proceedings/2002/nime2002{\_}088.pdf
Jensenius, A.R.: To gesture or Not? An Analysis of Terminology in NIME Proceedings 2001–2013. In: Proceedings of the International Conference on New Interfaces for Musical Expression, pp. 217–220 (2014). http://www.nime.org/proceedings/2014/nime2014{\_}351.pdf
Jensenius, A.R., Lyons, M.J.: A NIME Reader, Current Research in Systematic Musicology, vol. 3. Springer, Cham (2017). http://link.springer.com/10.1007/978-3-319-47214-0
Kuchera, B.: Nintendo Switch’s World of Goo shows off system’s Wii-style pointer controls (2017). https://www.polygon.com/2017/3/16/14947750/nintendo-switch-world-of-goo-controls-wii
Larboulette, C., Gibet, S.: A review of computable expressive descriptors of human motion. In: Proceedings of the 2nd International Workshop on Movement and Computing - MOCO ‘15. pp. 21–28. ACM Press, New York, NY (2015). http://dl.acm.org/citation.cfm?id=2790994.2790998. http://dl.acm.org/citation.cfm?doid=2790994.2790998
Lazzaro, N.: Why We Play Games: Four Keys to More Emotion Without Story. Technical Report, XEODesign Inc., Oakland, CA (2004)
Nintendo: How to calibrate the controllers (2017). http://en-americas-support.nintendo.com/app/answers/detail/a{\_}id/22340/{-}/how-to-calibrate-the-controllers
Piana, S., Stagliano, A., Camurri, A., Odone, F.: A set of full-body movement features for’ emotion recognition to help children affected by autism spectrum condition. In: IDGEI International Workshop, Crete (2013)
Piana, S., Stagliano, A., Odone, F., Camurri, A.: Adaptive body gesture representation for’ automatic emotion recognition. ACM Trans. Interact. Intell. Syst. 6(1), 1–31 (2016). http://dl.acm.org/citation.cfm?id=2896319.2818740
Rovan, J.B., Wanderley, M.M., Dubnov, S., Depalle, P.: Instrumental gestural mapping strategies as expressivity determinants in computer music performance. In: Proceedings of the AIMI International Workshop, pp. 68–73. Kansei-The Technology of Emotion Workshop (1997)
Serafin, S., Erkut, C., Kojs, J., Nilsson, N.C., Nordahl, R.: Virtual reality musical instruments: state of the art, design principles, and future directions. Comput. Music. J. 40(3), 22–40 (2016). http://www.mitpressjournals.org/doi/10.1162/COMJ{\_}a{\_}00372
Sixense Entertainment: Razer hydra (2011). http://sixense.com/razerhydra-3
STMicroelectronics: Semiconductor solutions from STMicroelectronics selected by Nintendo for Nintendo switch (2017). http://www.st.com/content/st{\_}com/en/about/media-center/press-item.html/t3934.html
Takahashi, D.: Nintendo Switch has high-tech Joy-Con controllers with motion detectioncamera (2017). https://venturebeat.com/2017/01/12/nintendo-switch-has-high-tech-joy-con-controllers-with-motion-detection/
Trojaniello, D., Cereatti, A., Pelosin, E., Avanzino, L., Mirelman, A., Hausdorff, J.M., Della Croce, U.: Estimation of step-by-step spatio-temporal parameters of normal and impaired gait using shank-mounted magneto-inertial sensors: application to elderly, hemiparetic, parkinsonian and choreic gait. J. Neuroeng. Rehabil. 11(1), 152 (2014). http://jneuroengrehab.biomedcentral.com/articles/10.1186/1743-0003-11-152. http://www.jneuroengrehab.com/content/11/1/152
Van Nort, D., Wanderley, M.M., Depalle, P.: Mapping control structures for sound synthesis: functional and topological perspectives. Comput. Music J. 38(3), 6–22 (2014)
Visi, F.: Methods and technologies for the analysis and interactive use of body movements in instrumental music performance (2017). https://pearl.plymouth.ac.uk/handle/10026.1/8805?show=full
Visi, F., Schramm, R., Miranda, E.: Gesture in performance with traditional musical instruments and electronics. In: Proceedings of the 2014 International Workshop on Movement and Computing - MOCO ‘14. pp. 100–105. ACM Press, New York, NY (2014). http://dl.acm.org/citation.cfm?id=2618013. http://dl.acm.org/citation.cfm?doid=2617995.2618013
Visi, F., Caramiaux, B., Mcloughlin, M., Miranda, E.: A knowledge-based, data-driven method for action-sound mapping. In: NIME’17 International Conference on New Interfaces for Musical Expression (2017a)
Visi, F., Coorevits, E., Schramm, R., Miranda, E.R.: Musical instruments, body movement, space, and motion data: music as an emergent multimodal choreography. Hum. Technol. 13(1), 58–81 (2017b). http://humantechnology.jyu.fi/archive/vol-13/issue-1/musical-instruments-body-movement-space-and-motion-data/@@display-file/fullPaper/Visi{\_}Coorevits{\_}Schramm{\_}Reck-Miranda.pdf
Visi, F., Georgiou, T., Holland, S., Pinzone, O., Donaldson, G., Tetley, J.: Assessing the accuracy of an algorithm for the estimation of spatial gait parameters using inertial measurement units: application to healthy subject and hemiparetic stroke survivor. In: 4th International Conference on Movement Computing (2017c)
Wasenmuller, O., Stricker, D.: Comparison of kinect v1 and v2 depth images in terms of¨ accuracy and precision. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). vol. 10117 LNCS, pp. 34–45. Springer, Cham (2017). http://link.springer.com/10.1007/978-3-319-54427-4{\_}3
Wattanasoontorn, V., Boada, I., Garcia, R., Sbert, M.: Serious games for health. Entertain. Comput. 4(4), 231–247 (2013). http://linkinghub.elsevier.com/retrieve/pii/S1875952113000153
Weichert, F., Bachmann, D., Rudak, B., Fisseler, D.: Analysis of the accuracy and robustness of the leap motion controller. Sensors. 13(5), 6380–6393 (2013). http://www.ncbi.nlm.nih.gov/pubmed/23673678. http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=PMC3690061. http://www.mdpi.com/1424-8220/13/5/6380/
Wessel, D., Wright, M.: Problems and prospects for intimate musical control of computers. In: Poupyrev, I., Lyons, M.J., Fels, S.S., Blaine, T. (eds.) Proceedings of the 2001 International Conference on New Interfaces for Musical Expression (NIME-01), pp. 11–14, Seattle, WA (2001). http://www.nime.org/proceedings/2001/nime2001{\_}011.pdf
Wöllner, C.: Body, Sound and Space in Music and Beyond: Multimodal Explorations. Routledge (2017)
Yang, L.: Knuckles cap sense overview (2017a). http://steamcommunity.com/sharedfiles/filedetails/?id=943495896
Yang, L.: Knuckles quick start (2017b). http://steamcommunity.com/sharedfiles/filedetails/?id=943406651
Zamborlin, B., Bevilacqua, F., Gillies, M., D’inverno, M.: Fluid gesture interaction design. ACM Trans. Interact. Intell. Syst. 3(4), 1–30 (2014). http://dl.acm.org/citation.cfm?id=2543921. http://research.gold.ac.uk/9619/1/TiiS2013.pdf. http://dl.acm.org/citation.cfm?doid=2567808.2543921
Zappi, V., McPherson, A.: Dimensionality and appropriation in digital musical instrument design. In: Caramiaux, B., Tahiroglu, K., Fiebrink, R., Tanaka, A. (eds.) Proceedings of the International Conference on New Interfaces for Musical Expression, pp. 455–460. Goldsmiths, University of London, London (2014). http://www.nime.org/proceedings/2014/nime2014{\_}409.pdf
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG
About this chapter
Cite this chapter
Visi, F., Faasch, F. (2018). Motion Controllers, Sound, and Music in Video Games: State of the Art and Research Perspectives. In: Williams, D., Lee, N. (eds) Emotion in Video Game Soundtracking. International Series on Computer Entertainment and Media Technology. Springer, Cham. https://doi.org/10.1007/978-3-319-72272-6_8
Download citation
DOI: https://doi.org/10.1007/978-3-319-72272-6_8
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-72271-9
Online ISBN: 978-3-319-72272-6
eBook Packages: Computer ScienceComputer Science (R0)