Development of Chat System Added with Visualized Unconscious Non-verbal Information

Abstract

Face-to-face communications are performed by sending and receiving verbal and non-verbal information. And non-verbal information are sent and received consciously and unconsciously. In the face-to-face communication, this non-verbal information plays the important roles for smooth communication. In the case of text chat, we can send the some kind of non-verbal information, for example, the face marks, smiley and stamps to let the partner know our emotion and the true meaning of verbal information. However, it is difficult to treat the unconscious non-verbal information in text chat. Because of this, sometime we have a misunderstanding of text information. Therefore, we propose the text chat system which visualizes the unconscious non-verbal information of user. In the proposed system, the change of heart’s pulse wave of user is reflected in the background color of text chat. In this paper, the detail of proposed system and the result of system evaluation by sensory evaluation are described.

1 Introduction

The forms of real-time communication are broadly divided into face-to-face communication and non-face-to-face communication [1]. As shown in Fig. 1 on the left, in face-to-face communication users communicate effectively to understand each other by sending/receiving various kinds of non-verbal information including facial expressions, voice tones, gestures and nodding, and so on [2]. On contrary, in remote communication, the lacks of non-verbal information sometimes interfere with a communication. Especially in the text communication shown on the right in Fig. 1, many users attempt to express verbal information such as their feelings and intensions by adding non-verbal information including symbols, face marks and stamps consciously for smooth communication. However they hardly exchange unconscious non-verbal information. However, it is said that the exchange of unconscious nonverbal information such as face colors and body motions plays an important role in face-to-face communication.

Fig. 1.

Role of non-verbal information in face-to-face communication and text chat

Therefore in this study, we propose a chat system enables to add unconscious non-verbal information and develop a prototype aiming to support smoother text communication in visualizing unconscious non-verbal information. A change of user’s heart’s pulse interval is used as the added non-verbal information. A heart’s pulse interval is closely related to automatic nerve’s activity and usually used as an index to estimate emotional stress. In general it is considered that giving a user moderate stress, the heart’s pulse interval becomes short and the variability becomes little. Also it is considered that keeping a user in relaxed state, the heart’s pulse interval becomes long and the variability becomes big. Additionally as a blood flow affects a change of face color…etc., we believe that visualization of the heart’s pulse interval and visualizing such information to the text chat is reasonable.

In this study the influence of presenting the unconscious non-verbal information on users in text chat is investigated. We build the text chat system added unconscious non-verbal information obtained by visualizing heart’s pulse interval variation and perform the experiment using the proposed system.

As related study in this field, Watanabe and his colleagues developed and evaluated the voice chat system in avatar communication, in which user’s communication behaviors and breathing information was reflected to the avatar simultaneously. From the result of evaluation experiment, it was concluded that the system adding the breathing information improved a sense of presence and coexisting and that users found it preferable to the system without adding such information [3]. In our proposed system, the heart’s pulse interval variation was used as the presented information. Recently a list-watch type sensor allowing measurement/visualization of pulse waves is available on the market, which is easier than using general physiological indexes including breathing information.

2 Chat System Added Unconscious Non-verbal Information

In this chapter, the method to reflect a change of heart’s pulse interval as non-verbal information in text chat and the outline and configuration of the system are described.

2.1 Outline of Proposed System

The proposed text chat system was added non-verbal information sent out unconsciously. The user’s heart’s pulse interval was used as the non-verbal information to be added and the change was reflected into the color of balloon in the text chat. The heart’s pulse wave was monitored by an ear-phone type sensor and the sensed information was sent to the server where the heart’s pulse interval was measured. According to the temporal change of the heart’s pulse interval, the background color of the user’s text chat changed. Although the proposed system measured the heart pulse interval and controlled the text chat through the server, it seems that this system is able to be built as an application available on P2P for smart phone users.

2.2 System Configuration

Figure 2 shows the outline of the system. The user put on the earphone type sphygmograph made by Rohm Co., Ltd. The information of user’s heart’s pulse wave was sent to PC with Bluetooth and finally sent to the server with socket. In the server, the heart’s pulse interval was detected from the pulse wave information received and the balloon color of the text chat was determined by comparing the length of the pulse interval received previously. When the user sent out the message, the color of the message and the balloon were sent to the client.

Fig. 2.

System configuration.

2.3 Detection Method of Heart Pulse Interval from Pulse Wave

In this study a user’s heart’s pulse interval is used as unconscious non-verbal information [4]. The method for detection of the heart’s pulse interval is described as follows; the pulse wave information at rest was measured for 5 min and its average of the maximum and the minimum value was defined as threshold Th. Comparing the threshold Th and the pulse wave information, the time exceed the threshold Th (Time t_i) was once stored, and the difference between ti and t_i−1 is the heart’s pulse interval. Moreover the average of the heart’s pulse interval for 10 s was calculated. Comparing it to the previous average and the change was reflected to the balloon color in the chat system.

2.4 Representation Method of Change of Heart’s Pulse Wave Interval in Chat System

The user’s heart’s pulse wave information for 10 s is sent to the server. After the heart’s pulse interval is detected from that information, it is stored at the server. When the user sends a message, the proposed system executes comparison of the heart’s pulse interval at that moment and the one of 10 previous second. The color of the balloon is verging to red when the heart’s pulse interval becomes shorter and it is verging to blue when the heart’s pulse interval becomes longer. If any change is not detected in the heart’s pulse interval, the color of the balloon remained unchanged. The system has 9 colors stages changed by the interval length. The color determined is sent to the both users’ terminals along with the user’s message. The above process is conducted repeatedly every time the user sends a message and the color of the balloon is reflected (See Fig. 3).

Fig. 3.

Representation method of the change of the heart’s pulse interval.

3 Influence of Added Unconscious Non-verbal Information on User and Text Chat

In face-to-face communication, a face color, for example, is difficult to change consciously. However, sending/receiving this kind of unconscious non-verbal information plays an important role for smooth face-to-face communication. To investigate the influence of the added unconscious non-verbal information on a user’s state of mind and text chat, the following experiment was performed using the proposed system.

3.1 Experimental Purpose and Method

To investigate the influence of the added unconscious non-verbal information on a user’s state of mind and a text chat, the evaluation experiment with subjects was performed using the developed text chat system. 40 university students aged around 20 years were participated as the subjects. They worked in pairs during the experiment. The two types of experiments reflecting/without reflecting the change of the heart’s pulse interval in the balloon color were performed. The number of statements and the result of questionnaire answered by the subjects after each experiment were compared. The pulse wave information at rest was measured for 5 min. previously, and then the experiment was carried out for 10 min. Additionally considering an order effect, the order of the two type’s experiments was randomly shuffled per subjects.

3.2 Experimental Results

Figure 4 shows the average of the number of statement and the standard deviation. As the typing speed differs depending on the subjects, the number of statements under each experiment’s condition is standardized by the average of the number of statements sent by each subject. No statistic difference is found between the chat system with or without reflecting the change of the subject’s heart’s pulse interval in the balloon color. There is high possibility that the change of the color has no impact on the number of the statement. Figure 5 shows the result of the questionnaire on the matching degree of the emotional change of the subject and the balloon color change. Many subjects have no idea whether their emotions are coincident with the balloon color or not. Moreover, Fig. 6 shows the result of the questionnaire on the degree of reading the partner’s feeling by text chat under each experiment condition. In the case with reflecting the change of the heart’s pulse interval in the balloon color, more subjects give positive answer to the question comparing to the case without reflecting. Thus the significant difference between two cases is found. In the case with reflecting the change of the heart’s pulse interval in the balloon color, although the subjects are not aware of the coincidence between their feelings and the color reflected on their sides, they feel it possible to read the partner’s feelings from the color reflected on the partners’ sides using the same system. Further studies are needed to validate these results.

Fig. 4.

Average and standard deviation of number of statement.

Fig. 5.

Degree of coincidence between color of text balloon and feeling.

Fig. 6.

Degree of reading partner’s feeling by text chat.

Figure 7 shows the result of the questionnaire on the degree of consciousness of the heart’s pulse sensor. The either of the experiment conditions results in low consciousness. However the subjects’ consciousness increases more in the condition reflecting the balloon color than in the condition without reflecting. There are significant differences between the two conditions.

Fig. 7.

Degree of consciousness of the heart’s pulse sensor

Figure 8 shows the result of the questionnaire on the degree of reading the partner’s feeling from the background color change. Figure 9 shows the result of the questionnaire on which system can read the partner’s feeling change easily. Many subjects respond that they feel it easier to read the partners’ feelings when using the system reflecting the change of heart’s pulse interval in the balloon color. As the above result, it was found that many subjects have an intuitive feeling that they are able to read the change of the partners’ feelings by the balloon color change.

Fig. 8.

Degree of reading the partner’s feeling by background color’s change.

Fig. 9.

System of reading the partner’s feeling easily.

As shown in Fig. 5 previously, the matching degree between the change of the subject’s emotion and that of the balloon color is not high. Meanwhile, in Fig. 6, the result of the degree of reading the partner’s feeling is explored in details. Figure 10 presents the answers of the questionnaire shown in Fig. 5 which is filled out by each subject.

Fig. 10.

Dividing participants into 3 groups by degree of coincidence between color of balloon and emotion.

The subjects are divided into α and β groups. 14 subjects, who answer that the change of their emotions highly matches with the change of the balloon color, belongs to α group, and 7 subjects, who answer the matching degree is low, belongs to β group. The results of each group in each questionnaire are shown in as following. Figure 11 shows the average and the standard deviation of the number of the statement. Each group has no statistical difference between the cases with or without reflecting the change of the heart’s pulse interval in the balloon color. Therefore, along with the result of Fig. 4, the difference in the matching degree between the balloon colors has no influence on the number of the statement.

Fig. 11.

Average and standard deviation of number of statement.

Figure 12 shows the questionnaire result in each group, on the degree of reading the partner’s feeling by text chat under each experimental condition. In α group, many subjects respond that they are able to read the partners’ feelings easier in the case with reflecting the change of heart’s pulse interval in the balloon color than in the case without reflecting the change and the significant difference is found. In contrast, many subjects in β group have no idea whether they are able to read the partners’ feelings or not and no statistic difference is found. In short, it was found that the subjects in α group who respond their own feelings matches with the color change intuitively feel the partners’ feeling change from the balloon color change.

Fig. 12.

Degree of reading the partner’s feeling by text chat.

The questionnaire result in each group on the degree of consciousness of the heart’s pulse sensor under each experimental condition is shown as Fig. 13. It is found that either group is not aware of the heart’s pulse sensor in the case without reflecting the change of the heart’s pulse interval in the balloon color change. In addition, α group tends to be aware of detecting the heart’s pulse wave in the case with reflecting the change of the heart’s pulse interval in the balloon color change and the significant difference is found from the condition without reflecting the change.

Fig. 13.

Degree of consciousness of the sensor.

Figure 14 shows the questionnaire result in each group on the degree of reading partner’s feeling by the balloon color change. The group obtains higher evaluation than β group and has the significant difference between the both groups.

Fig. 14.

Degree of reading the partner’s feeling by balloon color changes.

Moreover, Fig. 15 shows the questionnaire result on the system enables to read the partner’s feeling change in each group. Comparing to the β group, the α group has more subjects who answer that they can read the partner’s feeling easier when using the system reflecting the change of the heart’s pulse interval in the balloon color and the -significant difference between each group is indicated.

Fig. 15.

Reading the partner’s feeling easily.

Figures 14 and 15 indicates that the group recognizing the coincidence between the change in their feelings and in the balloon color change perceives the change in the partners’ state of feelings by the balloon color change. Accordingly, it is shown that the proposed system is effective for this group.

4 Conclusions

This study aims to support a smooth text communication and develops a system visualizing the unconscious non-verbal information on a text chat. In particular, a user’s heart’s pulse interval was detected from a user’s pulse wave. When the interval length changes, that information was reflected in the color of the balloon on the text chat. Additionally, the experiment to investigate the influence of visualized the unconscious non-verbal information on the subjects was performed in this study.

The result of the communication experiment using the proposed system shows whether the change of subject’s heart’s pulse interval is reflected in the balloon color has no influence on the number of the statement by the subjects. According to the result of the questionnaire, all in all, many subjects have no idea whether their own emotion match with the balloon color or not. In contrast, many subjects respond that the change of the state of the partner’s feeling matches with the balloon color. Therefore the further study was conducted by sorting the subjects into two groups according to the matching degree of the change of their own feelings and the balloon color. As a consequence, it is shown that the group recognizing high matching degree perceives that they are able to read the partner’s feeling as the balloon color changes. Also it is shown that the consciousness of detecting the heart’s pulse wave increases as the balloon color changes. From these results, visualizing the unconscious non-verbal information on the text chat is very effective for the user who considers it as reasonable and it suggests that there might be high possibility to support the smooth communication.

A method for the control of the balloon color change will be necessary to be examined as future challenge. In addition, we are scheduled to examine the variability of the heart’s pulse interval, the body temperature change and the other unconscious non-verbal information.