Abstract
A modular real-time diminished reality pipeline for indoor applications is presented. The pipeline includes a novel inpainting method which requires no prior information of the textures behind the object to be diminished. The inpainting method operates on rectified images and adapts to scene illumination. In typically challenging illumination situations, the method produces more realistic results in indoor scenes than previous approaches. Modularity enables using alternative implementations in different stages and adapting the pipeline for different applications. Finally, practical solutions to problems occurring in diminished reality applications, for example interior design, are discussed.
Similar content being viewed by others
References
Alvar: A library for virtual and augmented reality. http://www.vtt.fi/multimedia/alvar.html
Bao, S.Y., Furlan, A., Fei-Fei, L., Savarese, S.: Understanding the 3d layout of a cluttered room from multiple images. In: 2014 IEEE Winter Conference on Applications of Computer Vision (WACV), Steamboat Springs, CO, 24–26 March 2014, pp. 690–697 (2014)
Bhatewara, N., Kumar, P., Agrawal, A.: Intelligent video inpainting system for texture reconstruction. In: 2013 4th International Conference on Computer and Communication Technology (ICCCT), Allahabad, 20–22 Sept 2013, pp. 60–65 (2013)
Criminisi, A., Perez, P., Toyama, K.: Object removal by exemplar-based inpainting. In: 2013 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), 18–20 June 2003, vol. 2, pp. II-721–II-728 (2003)
Dou, M., Fuchs, H.: Temporally enhanced 3d capture of room-sized dynamic scenes with commodity depth cameras. In: Virtual Reality (VR), IEEE, Minneapolis, MN, USA, 29 March-2 April 2014, pp. 39–44 (2014)
Efros, A.A., Leung, T.K.: Texture synthesis by non-parametric sampling. In: Proceedings of the International Conference on Computer Vision, vol. 2. ICCV ’99, p. 1033. IEEE Computer Society, Washington, DC (1999)
Hartley, R.I., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press, Cambridge (2000)
Henry, P., Krainin, M., Herbst, E., Ren, X., Fox, D.: Rgb-d mapping: using depth cameras for dense 3d modeling of indoor environments. In: Khatib, O., Kumar, V., Sukhatme, G. (eds.) Experimental Robotics, Springer Tracts in Advanced Robotics, vol. 79, pp. 477–491. Springer, Berlin (2014)
Herling, J., Broll, W.: Advanced self-contained object removal for realizing real-time diminished reality in unconstrained environments. In: 2014 9th IEEE International Symposium on Mixed and Augmented Reality (ISMAR) Seoul, South Korea, 13–16 Oct 2010, pp. 207–212 (2010)
Herling, J., Broll, W.: Pixmix: a real-time approach to high-quality diminished reality. In: 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) Atlanta, GA, USA, 5–8 Nov 2012, pp. 141–150 (2012)
Herling, J., Broll, W.: High-quality real-time video inpainting with pixmix. IEEE Trans. Vis. Comput. Graph. 20(6), 866–879 (2014). ID: 1
Jarusirisawad, S., Hosokawa, T., Yokoya, N.: Diminished reality using plane-sweep algorithm with weakly-calibrated cameras. Prog. Inform. 7, 11–20 (2010)
Kawai, N., Sato, T., Yokoya, N.: From image inpainting to diminished reality. In: Shumaker, R., Lackey, S. (eds.) Virtual, Augmented and Mixed Reality. Designing and Developing Virtual and Augmented Environments. Lecture Notes in Computer Science, vol. 8525, pp. 363–374. Springer International Publishing, Berlin (2014)
Kawai, N., Yamasaki, M., Sato, T., Yokoya, N.: Ar marker hiding based on image inpainting and reflection of illumination changes. In: 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 293–294 (2012)
Kholgade, N., Simon, T., Efros, A., Sheikh, Y.: 3d object manipulation in a single photograph using stock 3d models. ACM Trans. Graph. 33(4), 127:1–127:12 (2014)
Korkalo, O., Aittala, M., Siltanen, S.: Light-weight marker hiding for augmented reality. In: 2010 9th IEEE International Symposium on Mixed and Augmented Reality (ISMAR) Seoul, South Korea, 13–16 Oct 2010, pp. 247–248 (2010)
Li, Z., Wang, Y., Guo, J., Cheong, L.F., Zhou, S.: Diminished reality using appearance and 3d geometry of internet photo collections. In: 2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) Adelaide, SA, 1–4 Oct 2013, pp. 11–19 (2013)
Liu, H., Philipose, M., Sun, M.T.: Automatic objects segmentation with rgb-d cameras. J. Vis. Commun. Image Represent. 25(4), 709–718 (2014)
Lu, B.V., Kakuta, T., Kawakami, R., Oishi, T., Ikeuchi, K.: Foreground and shadow occlusion handling for outdoor augmented reality. In: 2010 9th IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Seoul, South Korea, 13–16 Oct 2010 pp. 109–118 (2010)
Mann, S.: Mediated reality with implementations for everyday life.PRESENCE: Teleoperators and Virtual Environments, MIT Press (2002)
Mura, C., Mattausch, O., Villanueva, A.J., Gobbetti, E., Pajarola, R.: Automatic room detection and reconstruction in cluttered indoor environments with complex room layouts. Comput. Graph. 44, 20–32 (2014)
Pintore, G., Gobbetti, E.: Effective mobile mapping of multi-room indoor structures. Vis. Comput. 30(6–8), 707–716 (2014)
Shepard, D.: A two-dimensional interpolation function for irregularly-spaced data. In: Proceedings of the 1968 23rd ACM National Conference. ACM ’68, pp. 517–524. ACM, New York (1968)
Siltanen, S.: Texture generation over the marker area. In: IEEE/ACM International Symposium on Mixed and Augmented Reality, 2006. ISMAR 2006, Santa Barbara, CA, 22–25 Oct 2006, pp. 253–254 (2006)
Siltanen, S., Oksman, V., Ainasoja, M.: User-centered design of augmented reality interior design service. Int. J. Arts Sci. 6(1), 547–563 (2013)
Siltanen, S., Saraspaa, H., Karvonen, J.: [Demo] a complete interior design solution with diminished reality. In: 2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Munich, Germany, 10–12 Sept 2014, pp. 371–372 (2014)
Simon, G., Berger, M.O.: Interactive building and augmentation of piecewise planar environments using the intersection lines. Vis. Comput. 27(9), 827–841 (2011)
Yang, M.D., Chao, C.F., Huang, K.S., Lu, L.Y., Chen, Y.P.: Image-based 3d scene reconstruction and exploration in augmented reality. Autom. Constr. 33, 48–60 (2013). Augmented Reality in Architecture, Engineering, and Construction
Zou, C., Peng, X., Lv, H., Chen, S., Fu, H., Liu, J.: Sketch-based 3-d modeling for piecewise planar objects in single images. Comput. Graph. 46, 130–137 (2015)
Acknowledgments
Dr. Timo Tossavainen provided valuable comments during the writing of this article.
Author information
Authors and Affiliations
Corresponding author
Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
About this article
Cite this article
Siltanen, S. Diminished reality for augmented reality interior design. Vis Comput 33, 193–208 (2017). https://doi.org/10.1007/s00371-015-1174-z
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00371-015-1174-z