Keywords

These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

1 Introduction

Most elderly people want to live actively every day even after reaching the mandatory retirement age. Some people enjoy playing golf, fishing, climbing a mountain, playing a musical instrument as a hobby. These hobbies provide the elderly people a feeling of accomplishment. The feeling of accomplishment is effective even for decline of behavioral symptoms of dementia [1, 2]. However, some hobbies have a high threshold for novice people.

Although playing a keyboard instrument has a high threshold for novice people, people who practice it hardly is easy to get a feeling of accomplishment. If even people with dementia can try to play the keyboard with a support system/method, they will get a feeling of accomplishment and may live actively every day.

Therefore, we researched what kind of support method enables people with severe dementia to play the piano and get a feeling of accomplishment [3]. We conducted an experiment in which an elderly woman with “higher brain dysfunction” took piano lessons using educational video materials and found that the participant could play the piano as long as she imitated a model performance on a video. We concluded that the participant played the piano with enthusiasm. The number of pitch errors gradually decreased. Moreover, we recognized that she gained a feeling of accomplishment through her utterances and behaviors.

However, the participant complained that it was difficult to find the same key as the key the performer played in the video. Moreover, in one of the lessons, the model performer said the color of the sticker on the key immediately before hitting the key. It was in the same manner as “Lighted Keys [4]” which is a keyboard that lights the key the performer should hit next. In this case, the participant hit most of the keys with her index finger.

Therefore, we present another support method where users can play the piano laying their hands on the image of a model performer’s hands. In this paper, we describe this support method and the results of an experiment where ten healthy students tried to play the piano using this method.

2 Reference

There are many systems for healthy people to learn how to play musical instruments. A piano learning support system [5] refers to the key outlined in color and an adequate fingering number that a user should hit in the next timing. “The AR Piano Tutor [6]” also indicates the key that a user should hit in the next timing by a colored rectangle-shaped mark over the real keyboard with the Augmented Reality interface. “Piano Tutor for iPad [7]” and “Piano Marvel [8]” are applications for users who are novice piano learners. These systems show a keyboard on a small picture of a display. These systems have a high-score tracking function. Users can see what key they should hit next on the keyboard and where they made mistakes in their performances. However, it is difficult for elderly novice people to see the small picture on a display and to push the correct key while referring to the fingering numbers. “Family Ensemble [9]” is a piano duo support system that allows parents of novice child to hit keys correctly even with only one finger. Although the system supports only the parent’s performance directly, the system was developed on the assumption that the child gets to have the motivation to practice the piano. “The Phantom of the Piano [10]” shows a model performance when a user stops his/her performance.

3 Practice Method of Displaying a Model Performance

3.1 The User’s Hands Lay on Hands of the Model Performer

We present a practice method in which a user sets his/her hands on the hands of the model performer. As shown in Fig. 1, a half mirror is placed between an upward-facing 23.8-inch display (EIZO, FORIS FS2434) and an electric keyboard (YAMAHA, NP-30). As shown in Fig. 2, the display represents the model performance made using computer graphics (CG). A user playing the keyboard watches his/her hands laying on hands of the model performer’s hands in the mirror. We expected that the user would recognize the key and fingering that he/she should hit simultaneously and successively.

Fig. 1.
figure 1

A half mirror is placed between a display and a keyboad.

Full size image

3.2 Graphics of Model Performance

The musical piece is a hook-line from “Kawa no Nagare no Yoni (Like a Flowing Stream),” a Hibari Misora song. It is a very famous song in Japan. It consists of eighteen bars.

In our previous research [3], we made a video that a model performer plays a musical piece. If we employ the performer’s video to projected in the display, a strain is generated between each key in the video and on the real keyboard.

Therefore, virtual keys and hands of the model performance were made using the computer graphics software, “Blender.” Twenty-five frames per one second are generated. The dimension of each key made by Blender is same as the key of the real keyboard that we use in our experiment. The virtual keys and hands were reversed because the video was projected on the display as a mirror image.

Both hands of the model performance were on the keyboard constantly. A contrast between light and shadow on the surface of hands is coordinated according to movement of each finger. This operation allows the user of the system to know which finger will hit a key the next time.

Fig. 2.
figure 2

The model performance made by computer graphics. (Color figure online)

Full size image

Moreover, the key that the user should be playing displays as red, and the key that the user should play next displays as yellow. The user should release the key when the key returns to white.

The first author made MIDI (Musical Instrumental Digital Interface) data by playing the musical piece with an electronic piano. She played the piece about in one minute. The electronic piano outputs a “Standard MIDI (Musical Instrumental Digital Interface) File (SMF).” The SMF was converted a text file by a software “mft2.” In the text file, we can see a hitting time (Note on time) and a releasing time (Note off time). The time when each key or each finger starts to move is assigned by Note on/off time in making each frame of graphics with Blender.

4 Experiment

4.1 Participants

Ten university students in their early 20 s participated in an experiment. One of them was female. Six participants had no experience playing the piano. Three participants had a littel experience for half a year or one or two years. One participant had taken piano lessons for five years.

4.2 Informed Consent

The participants in this experiment were informed about the intentions of the experiment and the treatment of personal information. Moreover, they were informed that they could withdraw from the experiment at any time and no reward was prepared for them. However, after the experiment, they received one Madeleine. We obtained written consent from them.

4.3 Method

As shown in Fig. 3, the participants ware asked to play a musical piece along with the model performance by laying their hands on the CG performer’s hands in the mirror. A headphone output both the sounds of the model performer’s and the participants’ performances. The participants’ performances were recorded by video cameras and a midi editing software (Domino Ver.1.43) to analyze their performance and their mistakes in notes and fingerings. Three participants were asked to set wearable cameras (Pana-sonic HX-A500) on their heads so that we could see what they were watching when they were playing the piano.

Fig. 3.
figure 3

The participant watches his/her hands laying on hands of the model performer in the mirror.

Full size image

Figure 4 shows the method of the experiment. Before playing the musical piece, the participants were asked to lay their hands on the hands of the CG’s performer. For the model performances, the performer’s hands made with CG repeated the hook-line ten times. The first five times, the sounds of their performances and the model performances were not output. The participants may be a little familiar with playing the keyboard because they had been taught playing the keyboard harmonica when they were elementaly school students. We worried that they may try to find each correct key without watching the model performance when they noticed what the musical piece was. For the purpose of the experiment, participants had to depend on the movement of the model performer and the changing colors of the keys to play.

After five times performances, the participants were asked what the piece is. If the participant could not say a correct answer, we asked him/her same question after ten times performances.

The next five times, the sounds of their performances and the model performances were output. After ten performances, the participants played the piece by themselves. It took ten minutes for them to finish playing. We interviewed them regarding what kinds of things confounded them and where they were looking at while playing the musical piece along with the model performance.

Fig. 4.
figure 4

Method of the experiment.

Full size image

4.4 Analysis

We analyzed four of participants’ performances; the first, the fifth, and the tenth performances with the model performance, and the last performance without it. We counted correct hitting, extra hitting, and replaying and checked their fingering in each performance.

4.5 Result

Table 1 shows the results of the participants’ performances. The number of “wrong” indicates that they played notes that are not in the musical piece (extra, replaying). If they could accurately name the title of the musical piece after playing it five times, we marked “5” in the “answer title” and after playing ten times, we marked “10.” All of them answered it correctly.

Furthermore, by the mistake of the experimenter, six participants heard a noise when they were playing from the sixth to the tenth time. The noise was other participants’ performances collected previously. MIDI data from other participants’ performances were input into the keyboard. The “noise” column of concerned participants is indicated by “yes.”

Table 1. The results of the participants’ performances.
Full size table

“56” in the column “correct” means that the participants could play all the notes in the musical piece. Participant-I could play all the notes in the first performance in spite of having no prior experience playing the musical instrument. He hit only one extra note. After only five performances, nine participants could play the piece with a few mistakes in pitch and without a mistake in fingering. Moreover, Participant-F could play the piece without the model performance, even though she had no experience playing the piano.

Regarding the fingering, Participant-H mistook six notes when he played the musical piece without the model performance. The others could play with the correct fingerings in all performances.

All participants were told that they were watching the mirror while they played. The images from the wearable cameras showed that the participants had their eyes on the mirror the entire time they played.

Furthermore, four participants reported that they sometimes hit the wrong key when the CG model performer hit a key with his mid finger or ring finger. Figure 5 shows an example of these kinds of mistakes. The upper score shows the model performance. At the last two notes, the performer operated its fingers from the right mid finger (the fingering number is “3”) to the left mid finger (the fingering number is “3”). The last note should be the key of A (La) hit with a mid finger. The lower score shows the performances of Participants-D, F, G, and H. They played the key next to A, i.e., G (So) with their ring fingers by mistake.

Fig. 5.
figure 5

Confusion of fingering caused by mirror.

Full size image

5 Discussion

Although the participants had no or little experience playing the piano, almost all of them could hit all the notes of the musical piece correctly after playing it only five times.

We believe there are at least three reasons they were able to play the musical piece easily. First, they only imitated the model performance by laying their hands on the image of a model performer’s hands. Second, they were good at hitting keys when the color changed because they were used to playing tapping games. Third, they only needed to focus on one thing, the model performance in the mirror. During the practice stage, a common performer usually needs to see both the keys and a score while he/she is playing. Our system did not give such a burden to the participant.

Participant-I who could play all notes correctly in the first time flattened his fingers and hit the keys. However, in his the ten times performance, form of his hands became round and hit the keys deeply. He got used to keying as well as the hands of the model performance might lead him to the good form of hitting.

Moreover, after playing the piece ten times, Participants-D and F could play the musical piece with the correct fingering without the model performance. The participants practiced the musical piece with the correct fingering because they could see the hands of the model performer. If the keys’ color changes are only shown on the display without the hands of the model performer, they could not practice with the correct fingering and some of them might hit all the keys with one finger. We think that the correct fingering encourages a proficiency of playing the piano.

On the other hand, some participants hit the wrong keys when the model performer hit the key with her mid/ring finger because the model performance was represented like a mirror. We expected that the participants gradually come to move their fingers according to a feeling that integrates their fingers into the model performer’s fingers. However, being estimated from the participants’ mistakes, the participants, first, might catch which finger the model performer hit consciously, e.g. “The model performer used her mid finger to hit the key.” Then, they might move their appropriate finger to hit the key. If they practice for long time, they may be used to play the piano with the model performance.

6 Conclusion

In this paper, we presented a practice method in which a user sets his/her hands on the hands of the model performer to play a keyboard with a low threshold. We conducted an experiment where ten healthy students tried to play the piano using this method. They had no or little experience playing the piano. Although the sounds of their performances and the model performances were not output, almost all participants could hit all notes of the musical piece correctly in only five times practice. There are some reasons why users played a musical piece easily. First, they only imitated the model performance by laying their hands on the image of a model performer’s hands. Second, they were good at hitting keys when the color changed because they were used to playing tapping games. Third, they only needed to focus on one thing, the model performance in the mirror. On the other hand, the some participants sometimes hit wrong keys when the model performer hit the key with her mid/ring finger.

We aim to develop a practical method that supports elderly people to play a keyboard with a low threshold. The characteristics of elderly people vary. Some have reduced cognitive functioning, a reduction in skilled behavior, some degree of tremor, and/or lose their good eyesight. We will ask for the elderly people to play the keyboard instrument using the practice method in which a user sets his/her hands on the hands of the model performer. Then, we will discuss appropriate supports for them to use the practice method effectively.