Keywords

1 Introduction

In general, eyeblinks can be classified as voluntary, reflex, or spontaneous. A voluntary eyeblink occurs consciously, a reflex eyeblink occurs as a result of external factors such as sound and/or light stimuli, and a spontaneous eyeblink is one that occurs unconsciously [1]. If a system was able to distinguish when a user has blinked with a conscious desire to enter information, then we would be able to control a computer device. In other words, computer control using eyeblinks could be realized if a method that automatically distinguishes conscious eyeblinks from unconscious eyeblinks was available.

The results of psychology experiments have shown that the occurrence of eyeblinks is associated with cognitive status. Using this knowledge, a system that measures the state of exhaustion of drivers has been developed [2]. Further, studies have been conducted in an effort to determine whether it can be used as a communication support and assistance system for severely crippled persons such as amyotrophic lateral sclerosis (ALS) patients [3–5]. Systems using eyeblink as an input switch and otherwise combining it with eye gaze in an input interface to operate equipment have also been proposed [6–11]. However, in most systems, because eyeblinks occur at high speeds, accurate and dedicated equipment is required to measure them. In addition, these systems usually employ a fixed threshold or special operations.

Our aim is to develop an eyeblink input system that can be installed on conventional information devices, such as smartphones and smart glasses [12–14]. Using image analysis [10, 11], we previously obtained and examined shape feature parameters in an eyeblink waveform (i.e., the waveform representing the time evolution of the eyeblink process) and observed differences between conscious and unconscious eyeblinks among subjects [15]. In this paper, we propose a new automatic conscious eyeblinks differentiation method, and report on the results of evaluation experiments conducted using the proposed method and algorithm with 30 subjects.

2 Related Work

Conventional eyeblink input systems are classified into two basic types. The first type uses input based on pre-established time values (for example, when a user closes his/her eyes for more than 200 ms) [16, 17]. In this case, a dynamic threshold value is used for each type of eyeblink because eyeblinks show wide individual differences. A false input may occur if the threshold is fixed because the input time is user-dependent; a user might unconsciously produce considerably short or long eye movements. The second type of input system examines special eye movements, such as double eyeblinks and winks [18, 19]. However, these systems require the user to perform conscious, and occasionally complex, actions; therefore, users have to practice in order to be proficient at using these systems. In addition, the unusual eyeblinks required can cause user stress, especially when the systems are used over a long period [1, 9].

In an effort to overcome these problems, eyeblink input interfaces that incorporate more natural eyeblinks are being studied. However, a user who does not display a noticeable difference in shape feature parameters between voluntary and spontaneous eyeblinks must be conditioned and encouraged by such a system for it to accurately measure voluntary eyeblinks [7–10]. Conversely, the system proposed in this paper uses a messaging system—for example, it announces to a user, “you blinked correctly at the perceived signal”—to decrease user stress and to amplify the difference in the shape feature parameters. This system most closely approximates an actual eyeblink interface because it is expected that the user is conscious of the input when blinking, even if no user training has been conducted.

3 Characteristics of Eyeblinks Waveforms

We distinguish between conscious and unconscious eyeblinks by considering the fact that the duration of a conscious eyeblink is longer than that of an unconscious eyeblink [11, 16]. However, an eyeblink is a rapid motion that completes a series of operations on the order of a few hundred milliseconds; over and above that, individual differences are substantial. Consequently, because the time resolution of conventional video cameras is low, when measured with these cameras, significant differences in the eyeblink duration are not observed. Eyeblinks vary widely by individual, but in most cases, during a consious eyeblink, the eyelids close completely. In addition, variation in terms of the eyeblink waveform is relatively small in each individual. Therefore, we focused on the following parameters: closing-phase amplitude, opening-phase amplitude, and eyeblink duration, as discussed in a previous study [9]. Figure 1 shows a model of an eyeblink waveform in which the closing-phase amplitude Acl is defined as the height of the closing-phase starting point Ps to the minimum point Pmin. Pmin is defined as the point where the eye-opening area is smallest; that is, from the closing-phase end point Psb to the opening-phase starting point Peb. Similarly, the opening-phase amplitude Aop is defined as the height of the minimum point Pmin to the opening-phase end point Pe. Finally, the eyeblink duration Dur is defined as the field count from Ps to Pe.

Fig. 1.
figure 1

Model of eyeblink waveform.

Full size image

4 Automatic Measurement of an Eyeblink Waveform

If the time evolution of the eyeblink process could be accurately measured, it would be possible to express an eyeblink as a waveform. Individual eyeblinks must be measured and then analyzed for automatic differentiation of eyeblink types. The typical techniques used to sample eyeblink waveforms are electrooculogram (EOG) and image analysis. The EOG method involves placing an electrode on the skin near the eyeball. Eyeblink waveforms are then collected by recording changes in the cornea-retina potential. This technique was proposed for automatic detection of conscious eyeblinks until recently [20]. However, the EOG method requires a unique apparatus to process ocular potential, and the user must have an electrode attached to his/her skin. Therefore, the EOG method is unsuitable for a simple interface. Moreover, extraneous noise from a living body can cause interference. By contrast, image analysis examines pictures of eyeblinks captured by a video recorder. It has become popular because it requires no bodily contact and is manageable and adaptable. However, eye movements are difficult to capture with a video camera that has a standard aspect ratio (NTSC) because an eyeblink is a high-speed operation. Therefore, in this paper, we incorporate an algorithm used in previous research [10] that detects changes in eye aperture. The algorithm samples at 1/60 s using interlaced NTSC video images further divided into field images. Figure 2 shows the processing flow for detecting changes in the eye-aperture area.

Fig. 2.
figure 2

Overview of frame-splitting method and binarization.

Full size image

When image analysis is used, the first step is to analyze video images of the area surrounding the eye in order to assess changes in eye aperture using binarization based on flesh color. Figure 3 shows an example of changes that occur in the eye-aperture area. The data shown in Fig. 3 include changes in the eyeblink waveform. The next step applies smoothing differentiation between the split field area and the next split field. Coordinates that reveal the maximum area difference value and the minimum area difference value are then determined using a second differentiation.

Fig. 3.
figure 3

Changes in eye-opening area.

Full size image

However, this step in the analysis involves excessive noise resulting from small movements in the vicinity of the eye, such as from an eyelid. Therefore, we remove three coordinate classes of extreme value (maximum, minimum, and few-moving) using the k-means method. We determine the start and end of an eyeblink waveform using its maximum and minimum values because one eyeblink waveform contains only one maximum and one minimum value. Minimum values exist in the opening phase and maximum values exist in the closing phase. Data are obtained from one eyeblink waveform according to these factors. If the obtained maximum and minimum values are observed in succession as two points, the point closer to the field of temporal axes is used. An eyeblink start field is calculated by differentiating between field areas in the direction opposite to that of the temporal axes from the maximum value’s field. In this field, the threshold Th 1 becomes positive for the first time. By contrast, the eyeblink end field is calculated by the difference between the field areas in the forward direction of the temporal axes from the minimum value’s field. In this field, the threshold Th 1 becomes negative for the first time. The threshold Th 1 is then determined by the following equation:

$$ Th_{1} = f\left( n \right) - f\left( {n + 1} \right) $$

where n is the attention field and f(n) is the eye-opening area in the n field. Figure 4 shows an example of the detected eyeblink waveform.

Fig. 4.
figure 4

Example of an eyeblink waveform.

Full size image

An eyeblink waveform measured by means of image analysis can be applied to the model in Fig. 4. The eyeblink duration is represented as field numbers from the eyeblink starting point to the eyeblink end point. The eyeblink amplitude is represented as changes in the eye-aperture area. The point Pmin, at which an area is minimized, is determined based on the model (Fig. 1) in theory. However, it might not be determined by an actual measurement (Fig. 5). Therefore, Pmin is defined as the average of the eye-aperture field areas that are less than threshold Th 2 in one eyeblink waveform. Threshold Th 2 is determined from the following equation:

Fig. 5.
figure 5

Example of a difficult-to-decide minimum point.

Full size image
$$ Th_{2} = \frac{Amax - Amin}{10} + Amin $$

where Amax and Amin are the maximum and minimum, respectively, of the eye-opening area of the eyeblink waveform. In addition, the closing-phase amplitude is calculated based on the difference between the area of the eyeblink starting field and point Pmin. Similarly, the opening-phase amplitude is calculated based on the difference between the area of the eyeblink end field and point Pmin.

5 Automatically Differentiating Conscious and Unconscious Eyeblinks

In this section, we examine the differentiation of eyeblinks on the basis of the parameters of the extracted eyeblink waveform using the method outlined in Sect. 4. It has been reported that the duration of a conscious eyeblink is longer than that of an unconscious eyeblink [21]. However, in many cases, distinguishing between the two types of eyeblinks using this information is difficult because the difference in the duration of eyeblinks cannot be measured if the time resolution of the moving image is low. Therefore, the proposed method improves the distinction accuracy by combining the duration and amplitude. There are many cases in which differences between eyeblinks are not found because amplitude values are more sensitive to individual differences than duration values. However, we have already confirmed the following in preliminary experiments. Specifically, approximately one-half of all subjects in our experiment had a significant difference in both the duration and amplitude, and the other half had significant differences in either one of duration or amplitude. We also administered a t-test to the subjects using a 1 % standard deviation between conscious and unconscious parameters. And Table 2 shows the details of the results obtained.

A significant difference of 24 % (12 subjects) is evident in eyeblink duration. For eyeblink amplitude, the difference is 22 % (11 subjects). For both parameters, the significant difference is 46 % (23 subjects). Finally, no significant difference is apparent in 8 % of the subjects (4 subjects). In other words, a significant difference in shape feature parameters between voluntary and spontaneous eyeblinks is seen in a minimum of 92 % of the subjects. Moreover, the results of examination of individual parameters reveal the following. The total percentage of subjects who show a significant difference in eyeblink duration is 70 %. The total percentage of subjects who display significant differences in eyeblink amplitude is 68 %. Finally, the total percentage of subjects who show significant differences in both parameters is 46 %.

Figure 6 shows a histogram that summarizes the distribution of the average value of the eyeblink duration of the 50 subjects by eyeblink type. Conversely, the histogram in Fig. 7 summarizes the distribution of the average value of the eyeblink amplitudes of the 50 subjects by eyeblink type.

Fig. 6.
figure 6

Duration difference in each group [21].

Full size image
Fig. 7.
figure 7

Amplitude difference in each group [21].

Full size image

We perform automatic differentiation using a threshold to distinguish the larger differences among the measured feature parameters between conscious and unconscious eyeblinks in every subject. In the proposed method, a normalization process is first applied to each feature parameter based on the average value of conscious eyeblinks to decide the differentiation threshold of each subject. As shown in Table 1, the trends in the parameters of subjects can be classified into three groups: Groups A, B, and C. Group A comprises subjects who exhibit significant differences in eyeblink duration only. Group B comprises subjects who exhibit significant differences in eyeblink amplitude only. Group C comprises subjects who exhibit significant differences in both parameters. The method then selects the parameter difference of the larger side among the duration and amplitude after normalization. For instance, in a scenario where there is a particular difference in duration, the threshold, Th3, used to distinguish conscious eyeblinks, is determined from the following equation:

Table 1. Strengths and weaknesses of previous works.
Full size table
Table 2. Results of preliminary experiment [21].
Full size table
$$ Th_{3} = \frac{Tdv - Tds}{2} + Tds $$

where Tdv is the average duration of conscious eyeblinks and Tds is the average duration of unconscious eyeblinks. In this instance, an eyeblink is distinguished as a conscious eyeblink if the duration exceeds Th3. Further, an eyeblink is distinguished as unconscious if it falls below Th3.

On the other hand, in the case where there is a particular difference in amplitude, the threshold, Th4, used to distinguish conscious eyeblinks is determined from the following equation:

$$ Th_{4} = \frac{Adv - Ads}{2} + Ads $$

where Adv is the average amplitude of conscious eyeblinks and Ads is the average amplitude of unconscious eyeblinks. In this case, an eyeblink is distinguished as a conscious eyeblink if the duration exceeds Th4. Conversely, it is distinguished as unconscious if it falls below Th4.

The proposed method determines thresholds Th3 and Th4 via the above method during calibration, before actual measurements are conducted. Subsequently, it automatically distinguishes eyeblinks as conscious or unconscious based on the feature parameter.

6 Evaluation Experiments

In this section, we discuss the results obtained on employing the eyeblink waveform measurement and distinction algorithm outlined above and measuring the eyeblink waveform of 30 subjects (22 men and 8 women with ages in the range of 20–29 years; all without disabilities) to analyze the periodicity of the shape feature parameters of eyeblinks.

6.1 System Outline

The hardware comprising our experimental system included a Sony HDR-HC9 digital camcorder for obtaining eye images, and a personal computer for image and eyeblink waveform analysis. Although the camera could capture high-definition (HD) pictures, standard-definition (SD) pictures were used in the experiments. The system is intended to be mounted on wearable and smart devices. Furthermore, an experimental system was developed as a prototype.

Ordinary indoor lighting (incandescent lighting) was used when capturing moving images. A pair of light-emitting diodes (LEDs) was placed symmetrically on both sides of the camera and at a distance of approximately 60 cm directly in front of the face of the subject. The back of the subject’s head was lightly supported with a stabilizing device to prevent it from shaking. The video camera was placed in front of and below the subject’s head at a distance of approximately 20 cm. The camera was then used to magnify and obtain pictures of the area surrounding the subject’s left eye. Because the image format was set for SD video, the resolution was 720 × 480 pixels with a 16:9 aspect ratio and refresh rate of 30 fps (NTSC). These experiments were performed on the naked eye; therefore, eyeglasses were not allowed during filming.

6.2 Experimental Method

The subjects were given the following instructions during filming:

  • Pay attention to the silver dot mark located on the upper part of the camcorder. (The mark was placed at this location by us.)

  • When you hear the signal, “blink well” always.

  • You do not have to resist any unconscious urge to blink.

The “blink well” instruction was meant to increase the difference in the shape of the feature parameters between conscious and unconscious eyeblinks. In other words, the signal was a means of encouraging subjects to be strongly conscious of their voluntary blinks. The signal was sounded randomly at intervals of 4 to 10 s using a digital timer. Images were captured for an overall total of 90 s during the course of the experiment. The first 20 s was used for calibration. This experiment does not use a control group because it was a conscious property the eyeblink immediately after sounding.

After measuring the eyeblink video image, we measured and analyzed the individual eyeblink waveform using moving images. The calibration determined the distinction threshold by using 20 s at the beginning of the moving image based on the method described in Sect. 4 to decide the feature parameter to use as the distinction threshold by normalization and comparison in each subject. Differentiation of conscious and the unconscious eyeblinks was then performed in the subsequent 70 s of moving images, using the obtained Th3 and Th4 thresholds. At this point, the system distinguished only eyeblink waveforms that had been successfully detected automatically.

At the conclusion of the experiment, the subjects were asked to complete questionnaires and/or comment about their experiences during the experiment. The items in question were age, gender, sleep time during the previous night, health condition (five levels: one (bad) through five (good)), task difficulty (five levels: one (easy) through five (difficult)), confidence in achieving the task (five levels: one (low) through five (high)), and personal interpretation of “blink well.”

6.3 Real-Time Measurement Experiment

Table 3 provides data on the subjects that show a significant difference in the measurements between conscious and unconscious blinks. Representative results of the experiment in relation to measured conscious and unconscious blinks, including the average values of the durations of blinks, the closing-phase amplitude, and the opening-phase amplitude, are shown. The right side of the table shows the results after normalization and the selected feature parameter.

Table 3. Results of the extracted parameters.
Full size table

Using the amplitude ratio of the closing phase to the opening phase for parameters is complicated because the ratio of the closing-phase to the opening-phase amplitude was, in all cases, found to contain a minimum of one large parameter. Therefore, we redefined the average value of two amplitudes as the eyeblink amplitude. In Table 3, the tendency for variation in individual differences between conscious and unconscious eyeblinks is as follows.

Let us now analyze those subjects who either did not show significant differences or exhibited only some differences in shape feature parameters. The number of unconscious eyeblinks was found to be limited. Two reasons explain this. The first is the fact that few eyeblinks actually occurred, which may be because the subjects were under stress during the experiment. The second is that eyeblinks registered movements that were too small to be accurately detected. Therefore, this study might promote future research in eyeblink detection accuracy.

Table 4 provides the results of automatic distinction rate. Representative results of the experiment are displayed in relation to measured conscious and unconscious blinks. The table shows counts of detected conscious eyeblinks Vi and unconscious eyeblinks Si, distinction error of conscious eyeblinks Ev, and unconscious eyeblinks Es, distinction accuracy rate of conscious eyeblinks Cv and unconscious eyeblinks Cs, and total accuracy rate Ct. The accuracy rate values Cv, Cs, and Ct are determined from the following equations:

Table 4. Results of automatic distinction rate of conscious eyeblinks.
Full size table
$$ C_{v} = \frac{Vi - Ev}{Vi} \times 100 $$
$$ C_{s} = \frac{Si - Es}{Si} \times 100 $$
$$ C_{t} = \frac{{\left( {Vi + Si} \right) - (Ev + Es)}}{Vi + Si} \times 100 $$

These equations for accuracy rate are adopted from [11].

Using our proposed method, the average rate of successful differentiating of conscious eyeblink is approximately 72.7 % for the experimental sample of 30 subjects. While, the average rate of successful differentiating of unconscious eyeblink is approximately 90.3 %. Thus, the average accuracy rate of total is 83.6 %. In unconscious eyeblinks are high identification rate, however in conscious eyeblinks are lower as compared to the unconscious rate. At this point, we believe that this passed differentiating of conscious blink is not a major problem. If these passed differentiating occur, the input can be attempted again through an intentional repetition of the conscious eyeblink. Therefore, we think the accuracy rate of unconscious is more important than conscious rate. In addition, there are often subjects of only a low accuracy rate of either conscious or unconscious. Because we used a simple algorithm in this experiment (e.g. subject 1, 3, 5, and more…) intend to improve the accuracy of differentiating by using a combination of two parameters.

Following the experiments, we interviewed the subjects and discovered that some subjects did not perform eyeblinks consciously when signals were given because their unconsious eyeblinks occurred at the same rate. On the basis of the results of these interviews, we plan to revise future instructions to promote more clarity. In addition, the classification of eyeblink types can be improved based on those subjects who did not show significant differences.

7 Conclusion

In this paper, we proposed a method for automatic differentiation of conscious eyeblinks. A method that can automatically differentiate between conscious and unconscious eyeblinks is an important prerequisite for developing an input interface for eyeblinks. The results of the evaluation experiment conducted using the proposed method show that it is possible to automatically distinguish eyeblinks with higher accuracy than in previous studies if there is a small difference in the eyeblink duration. The proposed method shows that it is possible using a frame-splitting method even in environments that use a low time resolution video camera. The results of our evaluation experiment conducted with 30 different subjects indicate that the average accuracy is 83.6 %. We required to fix head lightly and to detach glasses from subjects. This is a problem at actual use. We believe that this problem can be solve by image processing using motion vector. Consequently, typical information devices will able to control using eyeblinks, only installing software based on proposed method.

In the future, we plan to develop a real-time computer input system based on proposed measuring system. We also plan to improve this method to increase the detection accuracy and investigate methods by which this system can be incorporated into mobile devices. And we want to validate racial and cultural difference influence to eyeblinks.