Keywords

1 Introduction

In recent years, many innovative human computer interaction methods have been proposed. Natural User Interfaces (NUI) try to reflect human traditional interaction with physical elements and are comprising of whole body elements including finger (touching), eye (tracking), body (moving) and even brain (signaling). In motion-based interactions, more and more full-body interactions are supported and many innovative interaction devices started entering the market (e.g., Myo, Leap Motion). With an increasing number of devices and plethora of interaction patterns, formal modelling of body movements has become essential. Interaction devices benefit from machine-readable descriptions for pattern detection and exchange in the Internet of Things with potentially several devices used as interaction ENSEMBLES [1]. Users benefit from human-readable modelling of body movements for learning the requirements of the interaction for (often varying) devices in use. Designers benefit by being enabled to describe and exchange design patterns during construction of innovative products.

Body movements for interaction can be considered as special forms of dance. Like musical scores can be used to compose and perform music, dance scores are used by theatre and music professionals to compose and perform dances. We propose to map this approach for HCI and have developed various methods and tools with the help of a modified version of Labanotation [2, 3].

2 Modelling Motion-Based Interactions with Labanotation

Labanotation is a movement language developed by Rudolf Laban in 1928, revised and improved multiple times, most notably at the Dance Notation Bureau [4]. Labanotation is very similar to music notation languages and notes body positions and movement commands on time lines. The symbol set covers all relevant body parts and joints and describes relations between each other as well as the position in space. It supports the notation of multiple motions in parallel and can therefore be used for simple but also quite complex interaction movements. It is therefore commonly used by professional choreographers and dancers.

By focusing on and extending essential structural elements, it is perfectly suited to describe body movements even in fine-granular movements. The use of Labanotation for the purpose of designing motion-based interaction has been suggested [5] and XML-based machine-readable implementations have been proposed as well (e.g., LabanWriter [6], Tiniklink/MovementXML [7], LabanEditor [8]).

Fig. 1.
figure 1

Designer and Analyser tools for creating and editing the XML model of the Labanotation score.

Full size image

Current methods used to describe and share interactions suffer from a number of problems, including the inability to capture a detailed description of the movements required and the inability to illustrate the timing and dynamic flow of said movements adequately. We have therefore developed an extended version of Labanotation based on XML, which also supports physical ability profiles in order to enhance the adaptability of movement-based interactions to a user’s individual physical context [9]. We have also developed tools for manually designing motion-based interactions using a graphical Editor [10] and an Analyser for automatic analysing interaction performances using cameras based on depth sensors (Kinect, OpenCV) [11]. Both tools can be used to create and edit XML-based models of interaction, either by conceiving movements using the editor or by observing movements using the analyser (Fig. 1).

3 Labanotation Visualisation

XML-based descriptions might be appropriate for machines for exchanging patterns between interconnected HCI devices in interactive IoT-scenarios (Internet of Things) and graphical visualisation of the Laban scores might be appropriate for skilled designers of interaction schemes. But end-users need a more intuitive representation of the expected body movements.

Literature reports about some early approaches to generate animations from Labanotation, mostly with the goal to ease the design of movements and give direct feedback to the user on a Labanotation score. The LabanEditor [8] animates Laban scores, but was designed with a specific focus on dance applications and does not support other output modalities. It also only supports a reduced set of symbols and the code is not publically available. The Mocap Data Editing approach [12] converts from motion captured (mocap) material to Labanotation and from there to animation. This gives the ability to edit motion captured sequences via Labanotation and can be used for motion editing. Since their framework uses the mocap data to extrapolate the Labanotation it functions on already given data and does not entirely create a new data set for animation.

4 Generator

We therefore have developed a new flexible system architecture in Java allowing for the creation of arbitrary modalities (Fig. 2). Our current implementation supports text and 3D character animations, but can easily be extended to other modalities, like e.g., speech output or graphical projections of cartoon animations on ambient displays in the respective smart space, etc. The systems contains an XML parser which converts the model into a set of serialized objects in the Movement Generator Engine. The parser is taken from our Editor [10] in order to guarantee interoperability. Every Labanotation symbol is taken with its parameters and type to create a new BodyPartMovement or JointMovement object to relate to a body part or joint and written into a list to keep the constraints of parallel and consecutive movements. The representation contains the direction and amount to be moved and the duration the whole motion should take to have a primitive representation for one motion. To represent a whole movement with these motions a correlation with each other is encoded.

Fig. 2.
figure 2

Creating human-readable visualisation using the Generator. (Color figure online)

Full size image

The Engine is then configured to convert these objects into the modality of choice. Depending on the context, the resulting media output stream can either be stored in a local file or transmitted to other entities for further processing or visualisation. For this, a set of generator types can be utilized in the system.

4.1 Generating Text

Textual description can suffice for a basic understanding of the required movement for simple interaction schemes. Nevertheless, describing multiple parallel movements with natural languages can result in confusing long sentences as all related movements need to be named and put in relation with each other. Therefore some type of text formatting might be an option to separate different blocks of parallel movements with appropriate keywords to signal the parallelism.

One feature of Labanotation is that the side of a limb, its height or general direction and the limb itself can be obtained from a stage without much effort. The columns are used to indicate sides. The limbs used can be obtained by the presign of a symbol or from the current column. With the given model for movements a primitive text generation can be implemented by reading the data and paraphrase them directly with the given limbs. Due to the differentiation between turn movements and translating movements a keyword of either ‘turn’ or ‘move’ can be added to the text. The qualitative direction stored in the movement model, as obtained through the Laban symbols, can be given to describe the movements direction. The narrative of the given motions creates a movement path, which is not always particularly detailed due to the range of detail in Labanotation. Furthermore, the user reenacting the movement will not always be following the directions perfectly, but usually the general directions of movements are relevant and not the exact distance or points travelled to.

An example on how the text is displayed below. First the user is given his starting position. Then each block of synchronous movement is listed:

figure a

4.2 Generating Animations

Animations using human-like figures support a more intuitive description of required movements, especially for more complex patterns and parallel movements of several body parts. Even though Java provides libraries for 3D rendering, the possibilities are limited. We therefore use a more general approach of generating a movement description that can be used as input for any 3D graphic engine. This even allows for distributing the Generator and the rendering processes to different devices, which might be useful in some scenarios, where the interactive device could delegate the render process to some cloud server. As an example rendering engine we deployed Unity3D. The Unity3D engine uses the MonoDevelopment API which can be compile to different platforms, including Windows, MacOS and Linux.

To create an animation model, a kinematic skeleton is needed. The kinematic skeleton is an unsorted tree whose root is usually the hip. The hip is a particularly suited root node since it is the centre of mass of the human body. Movements within the body can be related to the hip, while a movement of the whole body, usually walking, can be represented by the translation of the hip, since all other joints are attached to this node. Combined these can be seen as a walking animation where the legs propel the body.

Fig. 3.
figure 3

Kinematic skeleton with highlighted kinematic chain in 2D (left) and as tree structure (right). (Color figure online)

Full size image

The Generator creates a kinematic skeleton tree from scratch by adding vectors (‘bones’) including the position and orientation of the human bones (see Fig. 3 with a kinematic chain outlined in red as a subtree of the overall body tree). With the paths inside the subtrees, the inverse kinematic algorithm can determine the angles of each bone for a specific endpoint position in each node. For this, the ‘vecmath’ library of Java is used. The objects and parameters of the resulting animation is then compiled into JSON strings, serialized using the ‘fastjson’ library and send to our Rendering Engine using a TCP socket connection using the Mono (.NET) API of Unity3D. Deserialisation is based on the ‘MiniJSON’ library. The following listing shows an example object:

figure b

5 Conclusions

The Documentation Generation Tool presented in this paper supports an intuitive learning process for future interactive interfaces. We have implemented two example modalities, namely text and animation. Currently, we are working on combining modalities to further increase usability, e.g., animation with oral explanations. We are also extending the system towards an ambient reflection framework [13] supporting self-describing interface devices and automatically combined multimodal rendering processes in ambient spaces.