Skip to main content
SpringerLink
Log in

Tangible displays for the masses: spatial interaction with handheld displays by using consumer depth cameras

  • Original Article
  • Published:
Personal and Ubiquitous Computing Aims and scope Submit manuscript

Abstract

Spatially aware handheld displays are a promising approach to interact with complex information spaces in a more natural way by extending the interaction space from the 2D surface to the 3D physical space around them. This is achieved by utilizing their spatial position and orientation for interaction purposes. Technical solutions for spatially tracked displays already exist in research laboratories, e.g., embedded in a tabletop environment. Along with a large stationary screen, such multi-display systems provide a rich design space with a variety of benefits to users, e.g., the explicit support of co-located parallel work and collaboration. As we see a great future in the underlying interaction principles, the question is how the technology can be made accessible to the public. With our work, we want to address this issue. In the long term, we envision a low-cost tangible display ecosystem that is suitable for everyday usage and supports both active displays (e.g., the iPad) and passive projection media (e.g., paper screens and everyday objects such as a mug). The two major contributions of this article are a presentation of an exciting design space and a requirement analysis regarding its technical realization with special focus on a broad adoption by the public. In addition, we present a proof of concept system that addresses one technical aspect of this ecosystem: the spatial tracking of tangible displays with a consumer depth camera (Kinect).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. While the correction of such distortions is usually trivial, the tricky part is to grab the graphical output of a rendering framework before it is sent to the framebuffer of the projector.

References

  1. OpenKinect—imaging information. http://openkinect.org/wiki/Imaging_Information

  2. Akaoka E, Ginn T, Vertegaal R (2010) DisplayObjects: prototyping functional physical interfaces on 3d styrofoam, paper or cardboard models. In: Proceedings of the fourth international conference on Tangible, embedded, and embodied interaction (TEI), ACM, pp 49–56

  3. Bandyopadhyay D, Raskar R, Fuchs H (2001) Dynamic shader lamps: painting on movable objects. In: Proceedings of the International Symposium on augmented reality (ISAR), IEEE, pp 207–216

  4. Bier EA, Stone MC, Pier K, Buxton W, DeRose TD (1993) Toolglass and magic lenses: the see-through interface. In: Proceedings of SIGGRAPH, ACM, pp 445–446

  5. Butler DA, Izadi S, Hilliges O, Molyneaux D, Hodges S, Kim D (2012) Shake’n’sense: reducing interference for overlapping structured light depth cameras. In: Proceedings of the 2012 ACM annual conference on human factors in computing systems (CHI), ACM, pp 1933–1936

  6. Clark A, Piumsomboon T (2011) A realistic augmented reality racing game using a depth-sensing camera. In: Proceedings of the International Conference on virtual reality continuum and its applications in industry (VRCAI), ACM, pp 499–502

  7. Costanza E, Giaccone M, Kueng O, Shelley S, Huang J (2010) Ubicomp to the masses: a large-scale study of two tangible interfaces for download. In: Proceedings of the ACM international conference on Ubiquitous computing (Ubicomp), ACM (2010), pp 173–182

  8. Fischler MA, Bolles RC (1981) Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Comm ACM 24:381–395

    Article  MathSciNet  Google Scholar 

  9. Fitzmaurice GW (1993) Situated information spaces and spatially aware palmtop computers. Comm ACM Special Issue Comput Augment Environ Back Real World 36(7):39–49

    Google Scholar 

  10. Haller M, Leitner J, Seifried T, Wallace JR, Scott SD, Richter C, Brandl P, Gokcezade A, Hunter S (2010) The NiCE discussion room: integrating paper and digital media to support co-located group meetings. In: Proceedings of the International ACM Conference on human factors in computing systems (CHI), ACM, pp 609–618

  11. Han JY (2005) Low-cost multi-touch sensing through frustrated total internal reflection. In: Proceedings of the 18th annual ACM symposium on user interface software and technology (UIST), ACM, pp 115–118

  12. Harrison C, Dey AK (2008) Lean and zoom: proximity-aware user interface and content magnification. In: Proceedings of the International ACM Conference on Human Factors in Computing Systems (CHI), ACM, pp 507–510

  13. Harrison C, Benko H, Wilson AD (2011) OmniTouch: wearable multitouch interaction everywhere. In: Proceedings of the International ACM Symposium on user interface software and technology (UIST), ACM, pp 441–450

  14. Holman D, Vertegaal R, Altosaar M, Troje N, Johns D (2005) Paper windows: interaction techniques for digital paper. In: Proceedings of the International ACM conference on human factors in computing systems (CHI), ACM, pp 591–599

  15. Huber J, Steimle J, Liao C, Liu Q, Mühlhäuser M (2012) LightBeam: interacting with augmented real-world objects in pico projections. In: Proceedings of the International Conference on mobile and ubiquitous multimedia (MUM). ACM, Article 16

  16. Ishii H, Ullmer B (1997) Tangible bits. Towards seamless interfaces between people, bits and atoms. In: Proceedings of the International ACM Conference on human factors in computing systems (CHI), ACM, pp 234–241

  17. Izadi S, Hodges S, Taylor S, Rosenfeld D, Villar N, Butler A, Westhues J (2008) Going beyond the display: a surface technology with an electronically switchable diffuser. In: Proceedings of the 21st annual ACM symposium on user interface software and technology (UIST), ACM, pp 269–278

  18. Junuzovic S, Inkpen Quinn K, Blank T, Gupta A (2012) IllumiShare: sharing any surface. In: Proceedings of the International ACM Conference on human factors in computing systems (CHI), ACM, pp 1919–1928

  19. Khalilbeigi M, Lissermann R, Mühlhäuser M, Steimle J (2011) Xpaaand: interaction techniques for rollable displays. In: Proceedings of the International ACM Conference on human factors in computing systems (CHI), ACM, pp 2729–2732

  20. Khalilbeigi M, Lissermann R, Kleine W, Steimle J (2012) FoldMe: interacting with double-sided foldable displays. In: Proceedings of the Sixth International Conference on tangible, embedded and embodied interaction (TEI), ACM, pp 33–40

  21. Kirsh D (1995) The intelligent use of space. Artificial intelligence—special volume on computational research on interaction and agency, part 2, 73(1–2):31–68

  22. Kray C, Rohs M, Hook J, Kratz S (2008) Group coordination and negotiation through spatial proximity regions around mobile devices on augmented tabletops. In: Proceedings of the 3rd International Workshop on horizontal interactive human computer systems (TABLETOP), IEEE, pp 1–8

  23. Lee JC, Hudson SE, Summet JW, Dietz PH (2005) Moveable interactive projected displays using projector based tracking. In: Proceedings of the International ACM Symposium on user interface software and technology (UIST), ACM, pp 63–72

  24. Lee JC, Hudson SE, Tse E (2008) Foldable interactive displays. In: Proceedings of the ACM Symposium on user interface software and technology (UIST), ACM, pp 287–290

  25. Linder N, Maes P (2010) LuminAR: portable robotic augmented reality interface design and prototype. In: Adjunct Proceedings of the International ACM Symposium on user interface software and technology (UIST), ACM, pp 395–396

  26. Lissermann R, Olberding S, Petry B, Mühlhäuser M, Steimle J (2012) PaperVideo: interacting with videos on multiple paper-like displays. In: Proceedings of the ACM International Conference on Multimedia (MM), ACM, pp 129–138

  27. Merrill D, Kalanithi J, Maes P (2007) Siftables: towards sensor network user interfaces. In: Proceedings of the 1st international conference on tangible and embedded interaction (TEI), ACM, pp 75–78

  28. Newcombe RA, Izadi S, Hilliges O, Molyneaux D, Kim D, Davison AJ, Kohli P, Shotton J, Hodges S, Fitzgibbon A (2011) KinectFusion: real-time dense surface mapping and tracking. In: Proceedings of the International Symposium on mixed and augmented reality (ISMAR), IEEE, pp 127–136

  29. Rekimoto J (1998) A multiple device approach for supporting whiteboard-based interactions. In: Proceedings of the International ACM Conference on human factors in computing systems (CHI), ACM, pp 344–351

  30. Spindler M, Stellmach S, Dachselt R (2009). PaperLens: advanced magic lens interaction above the tabletop. In: Proceedings of the International ACM Conference on interactive tabletops and surfaces (ITS), ACM, pp 69–76

  31. Spindler M, Dachselt R (2009) Towards pen-based annotation techniques for tangible magic lenses above a tabletop. In: Accompanying DVD of the ACM International Conference on interactive tabletops and surfaces (ITS), ACM

  32. Spindler M, Tominski C, Schumann H, Dachselt R (2010) Tangible views for information visualization. In: Proceedings of the International ACM Conference on interactive tabletops and surfaces (ITS), ACM, pp 157–166

  33. Spindler M, Martsch M, Dachselt R (2012) Going beyond the surface: studying multi-layer interaction above the tabletop. In: Proceedings of the International ACM Conference on human factors in computing systems (CHI), ACM, pp 1277–1286

  34. Spindler M, Büschel W, Dachselt R (2012). Towards spatially aware tangible displays for the masses. In: Proceedings of Workshop on designing collaborative interactive spaces for e-Creativity, e-Science and e-Learning (DCIS) at AVI

  35. Spindler M, Büschel W, Dachselt R (2012) Use your head: tangible windows for 3D information spaces in a tabletop environment. In: Proceedings of the International ACM Conference on interactive tabletops and surfaces (ITS), ACM, pp 245–254

  36. Sellen AJ, Harper RHR (2001) The myth of the paperless office. The MIT Press, Cambridge, MA

    Google Scholar 

  37. Tarun AP, Wang P, Girouard A, Strohmeier P, Reilly D, Vertegaal R (2013) PaperTab: an electronic paper computer with multiple large flexible electrophoretic displays. In: CHI extended abstracts on human factors in computing systems (CHI EA), ACM, pp 3131–3134

  38. Tsang M, Fitzmaurice GW, Kurtenbach G, Khan A, Buxton B (2002) Boom chameleon: simultaneous capture of 3D viewpoint, voice and gesture annotations on a spatially-aware display. In: Proceedings of the International ACM Symposium on user interface software and technology (UIST), ACM, pp 111–120

  39. Ullmer B, Ishii H (1997) The metaDESK: models and prototypes for tangible user interfaces. In: Proceedings of the International ACM Symposium on user interface software and technology (UIST), ACM, pp 223–232

  40. Wallace JR, Scott SD, Stutz T, Enns T, Inkpen K (2009) Investigating teamwork and taskwork in single- and multi-display groupware systems. Pers Ubiquitous Comput 13(8):569–581

    Article  Google Scholar 

  41. Weiser M (1991) The computer for the 21st century. Sci Am 265(3):66–75

    Article  Google Scholar 

  42. Wilson D (2010) Using a depth camera as a touch sensor. In: Proceedings of the International ACM Conference on interactive tabletops and surfaces (ITS), ACM, pp 69–72

  43. Yee K (2003) Peephole displays: pen interaction on spatially aware handheld computers. In: Proceedings of the International ACM Conference on human factors in computing systems (CHI), ACM, pp 1–8

  44. Zhang Z, Wu Y, Shan Y, Shafer S (2001) Visual panel: virtual mouse, keyboard and 3D controller with an ordinary piece of paper. In: Proceedings of the Workshop on perceptive user interfaces (PUI), ACM, pp 1–8

Download references

Acknowledgments

This article is a revised and extended version of a paper that was presented at the DCIS 2012 workshop on Designing Collaborative Interactive Spaces for e-Creativity, e-Science, and e-Learning held at AVI 2012 [34]. Our work was funded by the German Ministry of Education and Science (BMBF) project ViERforES-II (01IM1000 2B) and the German Research Foundation (DFG) project IPAR (DA 1319/2-1).

Author information

Authors and Affiliations

  1. University of Magdeburg, Magdeburg, Germany

    Martin Spindler & Charlotte Winkler

  2. Technische Universität Dresden, Dresden, Germany

    Wolfgang Büschel & Raimund Dachselt

Authors
  1. Martin Spindler
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Wolfgang Büschel
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Charlotte Winkler
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Raimund Dachselt
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Martin Spindler.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Spindler, M., Büschel, W., Winkler, C. et al. Tangible displays for the masses: spatial interaction with handheld displays by using consumer depth cameras. Pers Ubiquit Comput 18, 1213–1225 (2014). https://doi.org/10.1007/s00779-013-0730-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00779-013-0730-7

Keywords

Search

Navigation