Keywords

1 Introduction

Puppetry, a special form of dramatic art, is a common art form in all over the world. As an inheritance of intangible cultural heritage, puppetry has left a deep impression in human history and culture. The forms of puppetry vary in different parts of the world, but generally they undergo the process of transition from religious worship performances to entertainment. For example, Italian dramatist Fligny explored puppetry of ancient Egypt along the Nile Valley. In his book “History of Puppet”, he proposed that the priests controlled the puppet to induce blind faith among the public, suggesting that religious worship endowed puppet with life. Besides, in the description of Greek historian Herodotus, the priest stepped onto the altar in the music along with the puppet and used dramas, songs and dances to entertain the God [1]. The ancient Chinese puppet shows began in Han Dynasty and enjoyed a rapid development in Tang Dynasty. Gradually the shows gave birth to various and different art forms. Among various branches, the marionette, as a representative branch, has passed down until today.

Marionette, also known as line puppet or hanging puppet, is made up of board, lines and puppet. The principle is that the puppet is maneuvered by the lines so as to make corresponding movements, which imitates human actions and creates vivid dramatic scene [2]. The puppet, similar to anatomic human body, is consisted of head, belly, limbs, etc. Among all kinds of puppetry, the performance of marionette is closest to humanity. Puppetry is a kind of art form relying on human control of the puppet. During the performance of marionette, the performer binds lines into the joints of the puppet and with the help of the board, the lines are maneuvered by the performer so as to make the puppet present different movements. Therefore, the performer should not only use his hands in balance to make sure the puppet moves flexibly, but also use fingers to move the puppet’s limbs.

During the process, whether the performer can control the lines proficiently and know how to bind the lines well are keys to the flexibility of marionette [1]. Nowadays people live a fast-paced life, fans of puppetry have little time to learn cumbersome practices or study skills and principles of binding lines. Besides, decrease of performers overtime leads to gradual disappearance of puppetry, the intangible cultural heritage. By studying the operating characteristics and skills of traditional puppetry, we apply gesture recognition technology to puppetry performance. In doing so, not only puppet operation is simplified, but also learning cost is reduced. In this article, we choose Leapmotion, a gesture recognition hardware to help users learn puppetry easily. Puppets can present sophisticated movements with a few simple gestures. In addition, this study not only combines traditional puppetry culture with contemporary culture under the new technology, but also explores traditional cultural expressions in the new media environment by studying operating modes of two ages (Fig. 1).

Fig. 1.
figure 1

Marionette

Full size image

2 Technical Theory and Background

Gesture recognition technology is based on computer science and was first born in the 1980s and 1990s. This technique explains human gestures through pattern recognition. Initially, this research is focused on development input of special hardware. For instance, B. Thomas developed his own digital glove in 1993. When users put on the glove, the computer can not only read the position of their hands immediately, but also detect whether their hands stretch and even the direction of the fingers. Though the technology has realized the purpose of input, the experimenter must wear hardware devices and natural Human–Computer Interaction cannot be achieved. Therefore, in the following research, developers began to focus on non-labeling gesture recognition technology, which can capture location and shape of hands without wearing devices. In 1994, Gaowen et al. of Harbin Institute of Technology proposed the capture and recognition of hands under a static and complex background. In May of 1995, they proposed the capture and recognition of hands under a dynamic and complex background. In the 21st century, gesture recognition technology has enjoyed a rapid development and presented tremendous scientific research and commercial value in robot control, automatic navigation, interactive education as well as healthcare, etc. With the intensive research of gesture recognition and promotion of related products, hardware devices based on non-labeling gesture recognition technology spring up like mushrooms and gradually enter into public life (Fig. 2).

Fig. 2.
figure 2

Digital glove

Full size image

2.1 Three Types of Gesture Recognition Technology

Gesture recognition technology is divided into three types, namely two-dimensional space hand recognition, two-dimensional space gesture recognition and three-dimensional space gesture recognition. The former two is mainly different from the latter in space. In two-dimensional space gesture recognition technology, computer-captured information includes width in horizontal line as well as height in vertical line (X and Y in geometry) and this space only extends in the two-dimensional scope [6]. On the other hand, computer-captured information using three-dimensional space gesture recognition technology includes length, width as well as height (X, Y and Z in geometry). In this case, we apply three-dimensional space gesture recognition technology.

Gestures and Actions.

Three-dimensional space gesture recognition technology uses special hardware to collect in-depth information instead of single common camera because the latter is unable to provide in-depth information. At present, there are three kinds of hardware around the world, namely Structure Light, Time of Flight and Multi-camera [4]. Three-dimensional space gesture recognition system solves depth problems, which enables hands to move freely within the recognition area. Fortunately, there is no need for users to wear any devices to be recognized. In addition, the technology has high accuracy, large scope as well as various HCI modes, and therefore, UX is highly improved.

2.2 Principles and Characteristics of Leapmotion

In this case, we choose Leapmotion, a hardware developed by Leapmotion company. This hardware mainly takes Multi-camera recognition technology, which captures images with various cameras at the same time. The principle is similar to compound eye insect [5]. By comparing with the difference among images captured by cameras in different angles, the technology calculates in-depth information in order to form a three-dimensional image (Fig. 3).

Fig. 3.
figure 3

Leapmotion

Full size image

For example, the principle of dual camera is based on geometry to calculate in-depth information as described in the figure. Camera 1 and camera 2 take two pictures of different angles respectively at the same time in the same environment. This process simulates the principle of human eyes. We can calculate the distances between the object and cameras if we can find out the location of the object as parameters of the two cameras and their relative location are known (Fig. 4).

Fig. 4.
figure 4

Multi-camera recognition technology

Full size image

The hardware demand for Multi-camera is lowest among all three-dimensional gesture recognition technology. Besides, extra special devices are not required as the technology is totally relied on computer vision algorithm. As a result, the Leapmotion is smaller and cheaper than many other hardware. It is convenient for users to carry and install [5].

3 Case Study

This study is based on a social science research project of Shandong Province called “Research on Integration and Development of Excellent Traditional Culture in Shandong Province and New Media”. The case is “Puppet”, which includes a large number of traditional elements of Laixi puppet show. According to our investigation, Laixi county of Shandong Province is one of the birthplaces of Chinese puppetry with important research values. Our research is divided into four phases. The first phase is investigation and research. Our members went to Laixi county of Shandong Province for data collection and artists’ interviews. The second phase is data aggregation and culture studies. By referring to artistic features of Laixi puppetry, we study characters, binding as well as stage in the puppet show. The third phase is HCI theoretic research. During the second phase, we have been looking for reasonable hardware support. Therefore, we can sketch the model, solve the problem of HCI and programming. The last stage is for the perfection of the technology. We combine the puppet and the stage, install sensor and single-chip microcomputer, look for BUG problems in the program in order to make the technology perfect and practical. In the following part, this article will present specific content of this study stage by stage.

3.1 Preliminary Research

Our group arrived in Laixi county, Shandong Province from 15th to 20th, April, 2016. We not only explore the local puppetry history and puppet making, but also communicate with inheritors of intangible cultural heritage. During our research, we have found out that Laixi puppet mainly uses characters from the Peking opera such as male role, female role, painted face, clown, etc. to represent different characters in different stories. The eyes, ears, nose and mouth of the puppet can move. The most three important operations of puppet show are how to lift up the puppets, rotate the stick as well as make the puppet move flexibly. During the interview with Laisheng Wang, an inheritor of intangible cultural heritage, we knew that a tremendous puppet was unearthed from the wooden-chambered tomb of Western Han Dynasty in Daishu village of Laixi county in 1978, which has attracted significant attention in the archaeology field both at home and abroad. The puppet, composed of 13 sections of wooden limbs, is 193 cm high. Its joints can move so as to sit, stand or kneel. The puppet from the tomb is by far the earliest and largest wooden puppet, which was regarded as the most famous Chinese puppet. The discovery is also a proof that Laixi is the birthplace for puppetry. In addition, after interviewing with Yang Zhang, a puppet show performer, we have had a general understanding of making puppets. A puppet is composed of a head, a body, hands and clothes. First of all, the artisan designs a puppet model and then makes a clay mold according to the drawing. Secondly, the artisan repeatedly pastes newspapers onto the mold before taking down the dried newspapers (form a cardboard), which has formed the shape of a puppet. Thirdly, the artisan fills mud into the cardboard that becomes smooth and sturdy after polishing. This process is commonly known as puttying. At last, the artisan draws eyes, eyebrows, mouth, etc. on the cardboard. If making a gimmick puppet, the artisan needs to leave two holes on its head in order to put Ping-Pong as eyes. In addition, it also takes some time to make the operating stick, clothes as well as accessories. Therefore, it takes at least months to make a puppet [1].

3.2 Research on Puppet Show and Binding Skills

During the performance of marionette, there are four basic binding skills. The first is to control the lines to make the puppet move. The second is to clear up the lines so they are in perfect order. The third is to clamp the unused lines timely during the performance. The fourth is to prepare lines for the next movements. Due to years of performance, the performer has gradually developed various skills such as lifting, rubbing, twisting, hooking, shaking, turning, spotting, swinging, dialing, buckling, turning, etc. [1] For instance, lifting can make the puppet nod or bow. In certain area, the artisan will put lines into special body parts of the puppet. For example, the artisan links lines with his belly button of the clown, the laughing role has lines in his hip, and Monk Sanhua has lines in his shoulders. What’s more, there are lines to control the fan when the puppet is performing a fan dancing and there are also lines to control the crutch when the puppet needs to be on crutches. Therefore, special characters are able to make special movements in different shows due to this kind of special lines. As a result, though line-controlling skills become increasingly complicated, artistic performance becomes more vivid. Lines to lift up the puppet are different in numbers, but there are six basic lines put in the limbs and body of the puppet.

We take “uproar in heaven” in “Journey to the West” as an example. The main puppet character is Monkey King, one of the most famous Chinese mythological figures who looks like a monkey and is called “stone monkey” because he was born from a stone since the creation of the world. The artisan put 10 lines in the head, wrists, elbows, hip, ankles and knees respectively in order to present his moving features.

3.3 Research on HCI Technology

After a preliminary investigation and research, we decide to use gesture recognition technology to realize HCI. The final work is mainly composed of 6 parts, which are PC, ARDUINO single-chip microcomputer, two stepping motors, LEAPMOTION sensor, the puppet and the stage.

First of all, we connect LEAPMOTION and single-chip microcomputer to the PC. LEAPMOTION and ARDUINO will establish a communication link after we program LEAPMOTION by using UNITY3D. During the process, we need to set two gestures in order to operate the two stepping motors. Opening the right hand and making a fist are two gestures we choose so as to lower the error. When the right hand is open, stepping motors rotate clockwise. On the contrary, stepping motors rotate anticlockwise when the right hand makes a fist. Unity3D is a comprehensive game tools and a professional game engine that allows players to make 3D videogames, realize architectural visualization or make real-time 3D animation. By using the software, we can program various HCI-based hardware such as LEAPMOTIN, KINECT depth camera as well as TOBII eye tracker. Virtual games or hardware machinery can therefore be controlled by the hardware. Unity3D uses C# language to program and the code scrip is as follows (Fig. 5):

Fig. 5.
figure 5

Unity3D programming

Full size image
figure a
figure b

Secondly, we connect two stepping motors to Arduino single-chip microcomputer. Single-chip microcomputer and stepping motors will establish a communication link after we program single-chip microcomputer by using PC. So far we have completed setting up hardware operating system. Users can then control stepping motors by using LEAPMOTION. ARDUINO uses C++ language to program and the code scrip is as follows (Fig. 6):

Fig. 6.
figure 6

Arduino connection

Full size image
figure c
figure d

Thirdly, we connect the puppet to stepping motors by lines in order to maneuver puppet’s movements. Due to the limited number of experimental equipment, only two stepper motors are used during the process. We use lines to connect hands and feet of the puppet to the two stepping motors respectively. The operating principle is as follows. When the user opens his right hand, PC detects its shape through LEAPMOTION and then sends the information to ARDUINO single-chip microcomputer. In this occasion, the two stepping motors rotate clockwise, which leads to the downside movement of puppet’s hands and feet. On the contrary, when user’s right hand makes a fist, the two stepping motors rotate anticlockwise, which leads to the upside movement of puppet’s hands and feet (Fig. 7).

Fig. 7.
figure 7

Stage drawing

Full size image

At last, we use wood to set up the stage. The stepping motors are installed on the roof of the stage. By controlling the length of the lines, the puppet can suspend in the air. During the performance, the user only needs to put his right hand in front of the stage can he control the puppet’s movements with different gestures (Fig. 8).

Fig. 8.
figure 8

Structure

Full size image

3.4 Research on Cultural Inheritance of Puppetry

The main task of the experiment is to study the cultural relationship between gesture recognition technology and traditional marionette show. During the millennium of traditional puppetry performance, performers maneuvered puppets with their hands, a form that has Chinese traditional features and connotation. The hands have become a bridge between new and old culture. Puppet show artists control puppets by using various skills such as lifting, rubbing, twisting, hooking, shaking, turning, spotting, swinging, dialing, buckling, turning, etc. The operation has become an integral part of puppetry culture because of various gestures and perfect cooperation of performer’s two hands. In this experiment, we still use two hands to control the puppet, which inherits the traditional acting media. The two gestures in the experiment, opening hand and making a fist, are similar to rubbing and twisting in the traditional performance. The old and new operating method is in contrast with each other from a cultural perspective. In future study, we will add more hardware devices and gestures (Fig. 9).

Fig. 9.
figure 9

Live demo

Full size image

3.5 User Experience and Feedbacks

We have conducted surveys among users of different groups in order to evaluate the prospect and market value of gesture recognition technology. 15 experimenters, ranging in age from 18 to 50, have taken part in our small-scale symposium. They also have different careers such as college students, university teachers, business owners and puppetry artists. The questionnaire includes the following main points. First, we ask them to write down their understanding of traditional puppetry. Second, we ask them whether they have ever used similar products. Third, we ask them whether the operation is practical and convenient. Fourth, we ask them what aspects of cultural inheritance we should pay attention to in our work. We hope that we can gain some suggestions for improvement from this survey.

In the symposium, we elaborated the concept, research process, operating methods, etc. to the experimenters. They can have the opportunity to participate in the interactive process and use gesture technology to perform puppet shows. During the meeting, we encouraged attendees to discuss in detail the various issues they encountered in the interactive experience and to gain a thorough understanding of interactive modes. In order to increase the practicability of the concept, we also hope that the participants can provide more ideas on integrating new media and traditional culture as well as HCI in this field in the future.

3.6 Findings and Conclusion

According to our results, most experimenters focused on HCI and hardware and they have provided a lot of positive feedbacks. On the one hand, students speak highly of the interactive experience because of the simple operating method and decreased time cost of learning. Business owners hope that accuracy of hardware recognition can be improved. On the other hand, puppetry artists and teachers has given their opinion on the relationship between HCI modes and cultural inheritance of puppetry. In their opinion, the experiment has not only established a connection between new media and traditional culture, but also basically realized mutual communication between two ages and two cultures. However, the connection is rather simple. In the future, we hope that we can have further study in this field, especially in finding out more breakthrough points to establish connections between new media and traditional culture.

3.7 Conclusion

In this study, we prove that gesture recognition technology can be applied in the field of puppetry. It can not only reduce user’s learning time and improve operating experience of the puppet performance, but also play an active role in publicizing traditional puppetry culture. In the cultural field, cultural features are not missing in the puppet show that uses gesture recognition technology. On the contrary, the technology has played a significant role in cultural inheritance. This experiment interprets the traditional cultural significance of gestures under the background of new technology and builds a bridge between new media art and traditional culture.