Abstract
This paper introduces a novel framework, Gesture and Appearance Cutout Embedding (GACE), that supports real-time integration of human appearance and gesture-guided control within a game. It aims to enhance immersion since it allows game users to see their personal appearance in a real-time manner. In addition, we exploit the gesture-based control to allow user interaction with other in-game characters. With the goal to make implementation easier, we address the challenges in the whole pipeline of video processing, gesture recognition, and communication. The system is successfully integrated into both entertainment and serious games. Extensive experiments show that the proposed system runs reliably and comfortably with a commodity setting. Meanwhile, the user impression study indicates our system is favored by end users.
Similar content being viewed by others
Notes
Accuracy is (T r u e P o s i t i v e + T r u e N e g a t i v e)/T o t a l.
References
Aggarwal J, Ryoo M (2011) Human activity analysis: A review. ACM Comput Surv 43(3):16
Cavazza M, Charles F, Mead SJ, Martin O., Marichal X., Nandi A. (2004) Multimodal acting in mixed reality interactive storytelling. IEEE MultiMedia 11(3):30–39
Chambel T, Viana P, Bove VM, Strover S, Thomas G (2014) Immersiveme’14: 2nd ACM international workshop on immersive media experiences. In: ACM Multimedia, pp 1255–1256
Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: IEEE Conference on computer vision and pattern recognition, pp 886–893
Dede C (2009) Immersive interfaces for engagement and learning. Science 323 (5910):66–69
Dondlinger MJ (2007) Educational video game design: A review of the literature. Journal of Applied Educational Technology 4(1):21–31
Henry P, Krainin M, Herbst E, Ren X, Fox D (2010) Rgb-d mapping: Using depth cameras for dense 3d modeling of indoor environments. In: International symposium on experimental robotics, pp 477–491
Izadi S, Kim D, Hilliges O, Molyneaux D, Newcombe RA, Kohli P, Shotton J, Hodges S, Freeman D, Davison AJ, Fitzgibbon AW (2011) Kinectfusion: real-time 3d reconstruction and interaction using a moving depth camera. In: ACM Symposium on User Interface Software and Technology, pp 559–568
Jörg S, Normoyle A, Safonova A (2012) How responsiveness affects players’ perception in digital games. In: Proceedings of the ACM Symposium on Applied Perception. ACM, pp 33–38
Lai K, Bo L, Ren X, Fox D (2011) Sparse distance learning for object recognition combining rgb and depth information. In: IEEE International Conference on Robotics and Automation, pp 4007–4013
Lang C, Nguyen TV, Katti H, Yadati K., Kankanhalli MS, Yan S (2012) Depth matters: Influence of depth cues on visual saliency. In: European conference on computer vision (2), pp 101–115
Lange B, Suma EA, Newman B, Phan T, Chang C-Y, Rizzo A, Bolas M (2011) Leveraging unencumbered full body control of animated virtual characters for game-based rehabilitation. In: International conference, virtual and mixed reality, pp 243–252
Likert R (1932) A technique for the measurement of attitudes. Archives of Psychology
Liu S, Nguyen TV, Feng J, Wang M, Yan S (2012) Hi, magic closet, tell me what to wear!. In: ACM Multimedia, pp 1333–1334
Lu J, Nguyen V, Niu Z, Singh B, Luo Z, Do M (2011) Cutechat: a lightweight tele-immersive video chat system. In: ACM Multimedia, pp 1309–1312
Nguyen TV, Feng J, Yan S (2014) Seeing human weight from a single rgb-d image. J Comput Sci Technol 29(5):777–784
Nguyen TV, Li L, Tan J, Yan S (2012) 3DME: 3d media express from rgb-d images. In: ACM Multimedia, pp 1331–1332
Nguyen TV, Tan YH, Sepulveda J (2014) GACE: Gesture and appearance cutout embedding for gaming applications. In: Proceedings of the 2nd ACM International Workshop on Immersive Media Experiences, ImmersiveMe ’14, pp 41–44. ACM, New York
Shotton J, Fitzgibbon AW, Cook M, Sharp T, Finocchio M, Moore R, Kipman A, Blake A (2011) Real-time human pose recognition in parts from single depth images. In: IEEE Conference on computer vision and pattern recognition, pp 1297–1304
Slater M, Spanlang B, Sanchez-Vives MV, Blanke O et al (2010) First person experience of body transfer in virtual reality. PloS one 5(5):e10564
Soltani F, Eskandari F, Golestan S (2012) Developing a gesture-based game for deaf/mute people using microsoft kinect. In: International Conference on Complex, Intelligent, and Software Intensive Systems, pp 491–495
Tang S, Wang X, Lv X, Han TX, Keller JM, He Z, Skubic M, Lao S (2012) Histogram of oriented normal vectors for object recognition with a depth sensor. In: Asian conference on computer vision, pp 525–538
Wang J, Liu Z, Wu Y, Yuan J (2012) Mining actionlet ensemble for action recognition with depth cameras. In: IEEE Conference on Computer Vision and Pattern Recognition, pp 1290–1297
Acknowledgments
This work was supported by Singapore Ministry of Education under research Grant MOE2012-TIF-2-G-016.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Nguyen, T.V., Sepulveda, J. Augmented immersion: video cutout and gesture-guided embedding for gaming applications. Multimed Tools Appl 75, 12351–12363 (2016). https://doi.org/10.1007/s11042-016-3435-x
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11042-016-3435-x