Abstract
Vision-based full-body 3D reconstruction for tele-immersive applications generates large amount of data points, which have to be sent through the network in real time. In this paper, we introduce a skeleton-based compression method using motion estimation where kinematic parameters of the human body are extracted from the point cloud data in each frame. First we address the issues regarding the data capturing and transfer to a remote site for the tele-immersive collaboration. We compare the results of the existing compression methods and the proposed skeleton-based compression technique. We examine the robustness and efficiency of the algorithm through experimental results with our multi-camera tele-immersion system. The proposed skeleton-based method provides high and flexible compression ratios from 50:1 to 5000:1 with reasonable reconstruction quality (peak signal-to-noise ratio from 28 to 31 dB) while preserving real-time (10+ fps) processing.
Similar content being viewed by others
References
Adobe. Quicktime 7.0 h.264 implementation (2006)
Aggarwal, J.K., Cai, Q.: Human motion analysis: a review. Comput. Vis. Image Underst. 73(3), 428–440 (1999)
Arikan, O.: Compression of motion capture databases. ACM Trans. Graph. 25(3), 890–897 (2006)
Baker, H., Tanguay, D., Sobel, I., Gelb, D., Gross, M., Culbertson, W., Malzenbender, T.: The coliseum immersive teleconferencing system. In: Proceedings of International Workshop on Immersive Telepresence, Juan-les-Pins, France (2002)
Bentley, J.L.: Multidimensional binary search trees used for associative searching. Commun. ACM 18(9), 509–517 (1975)
Besl, P.J., McKay, N.D.: A method for registration of 3-D shapes. IEEE Trans. Pattern Anal. Mach. Intell. 14(2), 239–256 (1992)
Cheung, K.M., Baker, S., Kanade, T.: Shape-from-silhouette of articulated objects and its use for human body kinematics estimation and motion capture. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, June 2003
CMU. Graphics Lab Motion Capture Database. Carnegie Mellon University. http://mocap.cs.cmu.edu/
Demirdjian, D., Darrell, T.: 3-D articulated pose tracking for untethered deictic reference. In: ICMI’02: Proceedings of the 4th IEEE International Conference on Multimodal Interfaces, Washington, DC, p. 267. IEEE Comput. Soc., Los Alamitos (2002)
Dewaele, G., Devernay, F., Horaud, R.: Hand motion from 3D point trajectories and a smooth surface model. In: ECCV (1), pp. 495–507 (2004)
Gavrila, D.M.: The visual analysis of human movement: a survey. Comput. Vis. Image Underst. 73(1), 82–98 (1999)
Gross, M., Würmlin, S., Naef, M., Lamboray, E., Spagno, C., Kunz, A., Koller-Meier, E., Svoboda, T., Gool, L.V., Lang, S., Strehlke, K., Moere, A.V., Staadt, O.: Blue-c: a spatially immersive display and 3D video portal for telepresence. ACM Trans. Graph. 22(3), 819–827 (2003)
Gumhold, S., Karni, Z., Isenburg, M., Seidel, H.-P.: Predictive point-cloud compression. In: Siggraph 2005 Sketches (2005)
Hasenfratz, J., Lapierre, M., Sillion, F.: A real-time system for full-body interaction with virtual worlds. In: Proceedings of Eurographics Symposium on Virtual Environments, pp. 147–156. Eurographics Association, Aire-la-Ville (2004)
Holden, M.K.: Virtual environments for motor rehabilitation: review. Cyberpsychol. Behav. 8(3), 187–211 (2005)
Ibarria, L., Rossignac, J.: Dynapack: space-time compression of the 3D animations of triangle meshes with fixed connectivity. In: SCA’03: Proceedings of the 2003 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, Aire-la-Ville, Switzerland, pp. 126–135. Eurographics Association, Aire-la-Ville (2003)
Isard, M., Blake, A.: Condensation—conditional density propagation for visual tracking. Int. J. Comput. Vis. 29(1), 5–28 (1998)
Jung, S., Bajcsy, R.: A framework for constructing real-time immersive environments for training physical activities. J. Multimed. 1(7), 9–17 (2006)
Kalra, P., Magnenat-Thalman, N., Moccozet, L., Sannier, G., Aubel, A., Thalman, D.: Real-time animation of realistic virtual humans. IEEE Comput. Graph. Appl. 18(25), 42–56 (1998)
Karni, Z., Gotsman, C.: Compression of soft-body animation sequences. Comput. Graph. 28(1), 25–34 (2004)
Keshner, E.A.: Virtual reality and physical rehabilitation: a new toy or a new research and rehabilitation tool? J. Neuroengineering Rehabil. 1(8) (2004)
Keshner, E., Kenyon, R.: Using immersive technology for postural research and rehabilitation. Assist. Technol. 16, 54–62 (2004)
Knoop, R.D.S., Vacek, S.: Sensor fusion for 3D human body tracking with an articulated 3D body model. In: Proceedings of the IEEE International Conference on Robotics and Automation, Walt Disney Resort, Orlando, Florida, 15 May 2006
Kum, S.-U., Mayer-Patel, K.: Real-time multidepth stream compression. ACM Trans. Multimedia Comput. Commun. Appl. 1(2), 128–150 (2005)
Lamboray, E., Würmlin, S., Gross, M.: Real-time streaming of point-based 3D video. In: VR’04: Proceedings of the IEEE Virtual Reality 2004, Washington, DC, p. 91. IEEE Comput. Soc., Los Alamitos (2004)
Lanier, J.: Virtually there. Sci. Am. 4, 52–61 (2001)
Lengyel, J.E.: Compression of time-dependent geometry. In: SI3D’99: Proceedings of the 1999 Symposium on Interactive 3D Graphics, New York, NY, pp. 89–95. ACM, New York (1999)
Li, L., Zhang, M., Xu, F., Liu, S.: Ert-vr: an immersive virtual reality system for emergency rescue training. Virtual Real. 8(3), 194–197 (2005)
Lien, J.-M., Bajcsy, G.K.R.: Skeleton-based data compression for multi-camera tele-immersion system. In: Proceedings of the International Symposium on Visual Computing (ISVC), pp. 347–354 (2007)
Lopez, E.J.L.: Finite element surface-based stereo 3D reconstruction, April 2006. Poster presentation given at the Trust NSF Site Visit
Marc Alexa, W.M.: Representing animations by principal components. Comput. Graph. Forum 19(3), 411–418 (2000)
Mason, H., Moutahir, M.: Multidisciplinary experiential education in second life: a global approach. In: Second Life Education Workshop, San Francisco, California, pp. 30–34 (2006)
McComas, J., Pivik, J., Laflamme, M.: Current uses of virtual reality for children with disabilities. Virtual Environments in Clinical Psychology and Neuroscience (1998)
Morency, L.-P., Darrell, T.: Stereo tracking using ICP and normal flow constraint. In: Proceedings of International Conference on Pattern Recognition (2002)
Mulligan, J., Daniilidis, K.: Real-time trinocular stereo for tele-immersion. In: Proceedings of 2001 International Conference on Image Processing, Thessaloniki, Greece, pp. 959–962 (2001)
Ochotta, T., Saupe, D.: Compression of point-based 3D models by shape-adaptive wavelet coding of multi-height fields. In: Symposium on Point-Based Graphics, pp. 103–112 (2004)
Patel, K., Bailenson, J.N., Hack-Jung, S., Diankov, R., Bajcsy, R.: The effects of fully immersive virtual reality on the learning of physical tasks. In: Proceedings of the 9th Annual International Workshop on Presence, Ohio, USA, pp. 87–94 (2006)
Piccardi, M.: Background subtraction techniques: a review. In: Proceedings of IEEE International Conference on Systems, Man and Cybernetics, Hague, Netherlands, pp. 3099–3104 (2004)
Point Grey Research Inc, Vancouver, Canada
Robb, R.: Virtual reality in medicine: A personal perspective. J. Vis. 5(4), 317–326 (2002)
Rusinkiewicz, S., Levoy, M.: Efficient variants of the ICP algorithm. In: Proceedings of the Third International Conference on 3-D Digital Imaging and Modeling (3DIM), pp. 145–152 (2001)
Sattler, M., Sarlette, R., Klein, R.: Simple and efficient compression of animation sequences. In: SCA’05: Proceedings of the 2005 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, New York, NY, pp. 209–217. ACM, New York (2005)
Simon, D., Hebert, M., Kanade, T.: Real-time 3-D pose estimation using a high-speed range sensor. In: Proceedings of IEEE International Conference on Robotics and Automation (ICRA’94), vol. 3, pp. 2235–2241 (1994)
Tsai, R.: A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses. IEEE J. Robot. Autom. RA3(4), 323–344 (1987)
William Pennebaker, J.M.: JPEG: Still Image Data Compression Standard. Springer, Berlin (1992)
Würmlin, S., Lamboray, E., Gross, M.: 3D video fragments: dynamic point samples for real-time free-viewpoint video. Comput. Graph. 28, 3–14 (2004)
Yang, Y., Wang, X., Chen, J.X.: Rendering avatars in virtual reality: integrating a 3D model with 2D images. Comput. Sci. Eng. 4(1), 86–91 (2002)
Yang, Z., Cui, Y., Anwar, Z., Bocchino, R., Kiyanclar, N., Nahrstedt, K., Campbell, R.H., Yurcik, W.: Real-time 3D video compression for tele-immersive environments. In: Proc. of SPIE/ACM Multimedia Computing and Networking (MMCN’06), San Jose, CA (2006)
Zhang, J., Owen, C.B.: Octree-based animated geometry compression. In: DCC’04: Proceedings of the Conference on Data Compression, Washington, DC, p. 508. IEEE Comput. Soc., Los Alamitos (2004)
Zhang, D., Nomura, Y., Fujii, S.: Error analysis and optimization of camera calibration. In: Proceedings of IEEE/RSJ International Workshop on Intelligent Robots and Systems (IROS’91), Osaka, Japan, pp. 292–296 (1991)
Zhao, W., Nandhakumar, N.: Effects of camera alignment errors on stereoscopic depth estimates. Pattern Recogn. 29(12), 2115–2126 (1996)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Lien, JM., Kurillo, G. & Bajcsy, R. Multi-camera tele-immersion system with real-time model driven data compression. Vis Comput 26, 3–15 (2010). https://doi.org/10.1007/s00371-009-0367-8
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00371-009-0367-8