A Human-Computer Interface and an Analysis on the Drawing of Curves with a Face Tracker Mouse

Abstract

Recent research on interactive electronic systems, like computers, can improve the quality of life of many researchers, students, professors, etc. In the case of disabled people, technology helps them to engage more fully into the world. Our study aims to evaluate interfaces for curves drawing with movements of the face. This article discusses about motivations to build such software, how the software works, iterative development of the software, and user testing by people with and without disabilities.

1 Introduction

In general, we often refer to curves only as a set of points on the plane or in space. Nevertheless, the formal definition is given by the concept of function. In this case, we describe curves three-dimensionally in the space as polygonal curves. In other words, a polygonal curve is a finite sequence of line segments, called edges joined end to end. The endpoints of the edges are named vertexes. Let ${v_0, v_1, ..., v_n}$ denote the set of $n+1$ vertexes, and $e_0, e_1, ..., e_n$ designate a sequence of n edges, where $e_i = v_iv_{i+1}$. Here, we consider closed curves where the last endpoint is equal the first $v_n = v+0$.

It is possible to find several applications in many areas of knowledge like Mathematics, Physics, Engineering, Computing and, specially, in computer graphics [1–4]. Besides, the concept of curves allows us to define subsequent ideas of curvature, torsion, surface, and area, which are fundamental in any topics whose applications involve three-dimensional geometry. This fact allows further expansion to the applications of this work.

This present discussion is particularly devoted to Physically Challenged People (PCP) who cannot use their hands when working on a computer. Their participation in academia has been growing steadily, and as such their access to such programs enabling performing physical experiments in a virtual way is highly desirable. Overall, there is a few software has been specifically designed to attend these users’ needs of usability to these users. Exceptions include software for typing with the eyes and head by moving the gaze across a keyboard displayed on the computer screen [5]. However, software systems controlled by the these movements is not available for the most of the activities, for example, drawing. One of the biggest problems when designing on a computer is clicking and dragging to obtain the desired figure.

Despite not being so scientific, art is a form of communication. A possibility of using it relies on the digital art, a practice that uses digital technology as an essential part of the creative or presentation process. The images in the visual arts consist of points, lines, shapes, colors, textures that we call elements of visual language. There have recently been solutions with eye-gaze technology, which enables people without hand movements to draw with their eyes [6].

Since the beginning of the project, our focus has been on understanding the users needs and identifying the usability goals and the user’s experience. Furthermore, we present a graphical user interface (GUI) implemented to work with the HeadMouse ^® [7], which allows the computer mouse functions to be carried out through movements of the head and face by the person. The clicking and the movements of the cursor are performed by software, which identifies the users movements of head, eyes, and mouth via a web cam.

In terms of digital accessibility, we hope to contribute to people with disabilities as well as to works by many other researchers in the literature [8–11].

2 Methodology

The current objective is to investigate temporal requirement and performance, but the most important point is the experience of the users, such as: how they feel about their performance using a face tracker, paying attention to mental and physical requirements, effort and frustration. Our interface is programmed in MatLab ^® as seen at Fig. 1, with the commands being activated by a mouse click on virtual buttons via the face tracker. The main view of the interface and buttons are described as follows:

• 1. Draw: Start the point-picking routine by clicking the mouses left button in the correct location for a valid coordinate. The program shows the coordinate in the grid and stores it to the variable “pts”, plotting the points after each correct click. If the click is on a wrong location, then nothing is done. A click with the mouses right button finishes the sub-routine and stops the clock. During this operation no other action is allowed. The next Draw command will start at the last point and the clock is resumed.

• 2. Read: Open a file window searching for a .mat file previously stored with the “pts” data. After choosing the file, the grid points are updated.

• 3. End Curve: This command closes the curve by connecting the last point taken with the first one. The number of picked points is incremented. Attention is need as it doesnt have any restrictions to get more points.

• 4. Undo: Clean the last point and decrement the actual “Current Number of Points” at the information panel. It is possible to “Undo” all the points.

• 5. Redo: Redraw the next point if there is a next point. If more than 1 point has been undone, but not all of them has been re-picked, then “Redo” will redraw the previously points until the “Current Number of Points” reaches the “Maximum Number of Points”.

• 6. Save next: Stores the “pts” data at a .mat file, saves the actual view as a .png file and creates a .tex file with these data. Each file has some prefix to identify it, which is identified by the extension “file_save_name”. This .mat file is exactly the one to be read at the “Read” button.

• 7. Restart: Refresh the variable “pts”, the clock, and the Open/Closed curve option.

• 8. Exit: Close the interface.

The Save next function has a very important task, it enables a fast data analysis because variable “pts” has all points and the time used to select them, so there is no need to make manual notes during the test and on the other hand the .tex file organizes it together with the set of figures for every test, therefore a simple .tex base file calls every single .tex file from the tests by using an input function from . It is simple to insert all of the same tests because their names change only by the number from the “Save Counter”, and show the results. In addition to providing a useful tool for saving data to users, it facilitated the collection of data in the test sessions. Figure 2 shows the interface while choosing a point. Buttons stays disabled until the right button is clicked, which warrants that the function gets the points needed. Figure 3 presents the possibility to rotate the view and check the picked points, it can be stored as it is by clicking the “Save next” button.

Fig. 1.

Interface view

Fig. 2.

Collecting points on screen

Fig. 3.

Rotation of view and check selected points

Fig. 4.

Information panel

Figure 4 is the information panel, exhibiting the time when the last point was picked, the current point to be picked and the maximum point number. This is especially useful to know how many times a “Undo” commands have been made and therefore how many “Redo” can be done forwards.

The interface is designed to prevent the user from committing unintentional errors. With this proposal, the drawing region is enabled only when the “Draw” button is pressed. In addition, the user has access again to other options when the curve drawing ends.

For a scheme of a richer characterization of how people with physical disabilities are adopting devices for interaction, we asked: Which devices are being used for on a daily basis? Which adaptations are users making to improve accessibility?

We noticed some non-commercial adaptions created by the participants, such as the use of a pen in the mouth to guide the mouse and the use of cheek movements to press buttons. Although there are many advances in technology, accessibility challenges still exist.

2.1 Users Characteristics

Typically, each of the 16 users participated in at least one session. For the rest of this article, we will use the word “user” to refer to the primary subject. Our study includes the following demographic breakdowns:

• Gender: $56.25\,\%$ male and $43.75\,\%$ female;

• Age Group: $6.25\,\%$ teenagers $(13 \sim 17 yrs)$ $93.75\,\%$ adults $(18 \sim 64 yrs)$.

We collected data from 4 mobility impaired users including three painters and two web designers (one participant being both) and 12 able-bodied participants involving undergraduate students. Of these, one was undergraduate student, one was master student, eight were doctoral students and one was post-doctoral student. The areas of study were the following: Geology, Applied Mathematics, Bioinformatics, Statistic and Differential Geometry.

Among the participants who have disabilities, there were two women with the Charcot-Marie-tooth disease (CMT is a hereditary motor and sensory neuropathy affecting both motor and sensory nerves, with weakness and muscle atrophy affecting arms and hands), a man who became quadriplegic after car accident, and two people with atrophy in the upper limbs. Two participants were virtually tested with a remote control software according to our evaluation criteria.

2.2 Survey Respondents

Twelve participants responded to our survey. We wanted to investigate about the experience in drawing using a computer, the needs, benefits and difficulties doing this action.

Twelve participants answered our survey. We wanted to investigate the users experience of drawing through a computer along with needs, benefits and difficulties regarding this action. All participants use the computer more than once a week. But, just $37.5\,\%$ were expert computer users and only 25 % were programmers.

In the group of mobility impaired users there also was one undergraduate student, one master student in a French language program, one doctoral student in applied mathematics, one programmer, and seven participants who use software to draw.

Of these, three are mobility impaired users and they are able to use professional applications. They reported that the most effortful task was to learn commands to get started with these programs, followed by the difficult to control the conventional mouse.

The others told that they do not use software to draw because they know no software that can meet their needs, such as plotting graphics, in an easy way easy without having to spend time training it.

2.3 Interaction Styles

The concept of interaction styles refers to all the ways by which an user can interact with a computer. In our study, we had considered the followings: conventional mouse/mouse-pad by hand (Fig. 8), chin (Fig. 5), thumbs, feet (Fig. 7); trackball by chin; face tracker (Fig. 6). An experimental restriction was that the participant needed to be seated. Because of the condition of being lying in bed, a young man had difficulties in positioning the camera in order to get good calibration, thus he did not participate in the research.

Fig. 5.

Trackball by chin

Fig. 6.

Virtual mouse with face tracker

Only two participants had experience in the field. One did it by curiosity, whereas the other had already used an eye and facel tracking device at the same university. One participant said to have used some kind of tracker, not just facial, for a game. In Fig. 8, the participant is a painter and he told that he would use our interface in his digital painting classes. Interaction with the fingers was by far the most common direct interaction method for people with and without disabilities. A boy and one woman use their toes to point and it is noteworthy that this boy and other girl used thumbs to interact. We observed that motor impairments did not interfered with their ability to perform. Our attentiveness to the surrounding issues of human-computer interaction is in the sense to ensure system functionality and usability, providing effective user interaction support, and enhancing a pleasant user experience. Thus, the user performance is so important as the user experience. In this sense our present work is also devoted to impaired people that cannot use the upper limbs. Their participation in academia has been growing steadily and their access to programs that allow the access to draw in each interaction style is crucial.

Fig. 7.

Conventional mouse by foot

Fig. 8.

Mousepad by hand

3 Discussion

The experiment was divided into three sessions of about 40 min, with each participant working individually.

In the first session, the users created eight curves freely with three four, five, six, seven, eight, nine, and ten points, respectively (Fig. 9). The time of each action was estimated and it is showed in Fig. 10.

The average time was 53.3862 s. In general, all users were satisfied with their performance and told that the interface was easy to use and the learnability of the commands was not difficult to understand because of the buttons and the grid. Performance and estimated time were similar between impaired users and able-bodied participants. One boy who has problem in his arms tried to manipulate the mouse with the two thumbs initially, but he had a lot of difficulties and then he tried it with his right foot, which was the easiest way to him.

When asked if they had ever experienced other method of interaction besides the conventional mouse and keyboard, only four of them had used touch screen technology in their personal computer. The colored lines on the graph represent the performance of the participants in the sessions, which measured the time spent in relation to the amount of points in the curves.

In the first session (Fig. 11), the users made the first attempt of moving the mouse with head movements. The participants should redraw the first and second curves created in the free-drawing session. The time increased by 90 s. Being out of the comfort zone can be pointed as a possible cause, since it is arduous to perform a new task in the beginning. Other reason is related to the users personality: some participants were worried because they could not click where they wanted and this influenced their results. They aimed to have the same performance as the first attempt with the methods they were used to.

Fig. 9.

Free draw

Fig. 10.

Repeating first and second curve

After the user detected and located the virtual mouse by means of the head mouse, the click actions can be emulated. With regard to the interface, the targets may be small, leftover, unexpected clicks generated by involuntary movements resulting from lack of practice in controlling the face. In order to alleviate such a problem, a routine was established to attract the click to the desired target even if the click is done with $0.5^\circ$ of distance.

In the second session, the participants should redraw five opened curves with three (2), four (2) and five points selected from the database captured in the first session. The average time was reduced by 30 s. Some users reported during the validation experiments that they did not feel so stressed in the second session as in the first one because they knew what they should do and their performance was better. By comparing head movements to natural movements, it was found that the users had similar performance, as can be seen Fig. 13.

However, the members being tested committed minor errors when starting the curve as points were clicked wrongly or accidentally. The participants felt more satisfied with controlling better the head. In the following two pictures we compare a wrong curve (Fig. 12) to a correct one (Fig. 11). This participant could have used the “Undo” button, but she had to use it twice and so she preferred not get bored and tired doing it.

Fig. 11.

Correct example

Fig. 12.

Wrong curve

In the third session, the participants were expected to draw closed curves selected from the database captured in the first session. The experiment was divided into two parts:

• (i) It was required to join the last point to the first one, thus closing the curve.

• (ii) It was necessary to press the “End Curve” button to close the curve.

The average time between both parts was increased by 50 s, even for a lower amount of points (Fig. 14). The same deviation was observed between the first and second sessions. One possible reason for these results was the fatigue caused during the development in the first part of the session. However some participants who had given up presented a better performance compared to the first one.

Fig. 13.

Performance of second session

Fig. 14.

Performance of third session

In this study, in addition to evaluating the performance of input interfaces for drawing curves with face tracking mouse, we were interested in understanding the users’ experience, more specifically the “work load” experienced by the participants. Notwithstanding, some concepts regarding mental demand or effort are difficult to measure. Some comments cited by the participants helped identify changing needs in the physical components, which improved the users experience. For example, if we could enlarge the target area, it would be much more convenient to click on a button. Other suggestion was to replace the direct click on the grid for access via a virtual keyboard with large buttons.

The issues related to each users perception are shown in Table 1. They were requested to describe the feelings experienced while performing the task with respect to mental demand, physical demand, temporal demand, own performance, effort and frustration. The performance ranged from “very good” to “very bad”, whereas the other five scales from “very low” to “very high”. We can estimate the experienced work load described by the participants in the first and last sessions.

Table 1. Results of session 1 and 3

Mental requirement has to do with remembering to click, which enables the region by pushing the draw button. Then, in order to have access to virtual buttons again it is needed to simulate the conventional mouses right button with the functionality of a face tracker. After the third time, it became a natural process for the participants.

The use of a different tool can have caused stress and fatigue because the users wanted to avoid making mistakes. Maybe personality traits, such as those of scientists and artists, make the participants to strive to be perfect at the tasks and they end up forgetting the time. In fact, no error occurred in the first attempt with the head mouse. As a consequence, as shown in the time bar chart, the time spent was longer than the expected.

Due to the slow process of moving the cursor and clicking on a small target to achieve good performance, some participants were angry and insecure, besides the tiredness. Although there is no error, the majority of the participants felt some kind of disappointment due to thoughts that they could do better. However, almost all participants (two had given up - one with impairment and an able-bodied one) reported that it was a challenging situation and they would eventually become an expert.

In general, the feeling of frustration is associated with the time spent to perform the task. There is no difference of opinion between the users with or without disabilities. If the participants thought it was taking too much time in comparison to the time spent in their usual interaction style, frustration would be higher and performance rated as poor. The controlling of the face tracking mouse was improved with the inclusion of a button for closing the loop and an attractor for the target, which changed the situation, as can be seen in Table 1.

The mental activity, including the physical demand, became so simple that sometimes the participant had to be reminded about what should be done.

After two sessions, the users found the best way to control the facial movements to achieve the desired performance. This fact reduced both time spent and frustration significantly.

All participants had improved their performance by using less effort, which can be seen at their choice in the fourth and fifth items of the Table. Most of them managed to reach performance rates similar to those in the form of interaction they were used to, either by using conventional mouse or individual adaptations.

The users noticed that the combination of our interface with HeadMouse ^® is useful when they have awareness about their own hands or when they are continually busy with other tasks. Then, they were interested in improving their performance.

The frustration was lower when a button was included to end a curve. Clicking on a button is one standard form of interaction, and for this reason they can feel comfortable and less stressed. Other consideration is about the button size, which was larger than the target area.

Time demand, physical demand and efforts decreased. Also, the users were satisfied with their performance. Some comments indicate that the interface allowed them to create prototypes rapidly as well as examples of curves and polygons. However, error rates increased too.

We highlight the examples of Figs. 15 and 16, which had the highest error rates because of the curve begins and ends with the same color.

Fig. 15.

Correct closed curve

Fig. 16.

Wrong closed curve

We highlight the examples of Figs. 15 and 16, which had the highest error rates because of the curve begins and ends with the same color. We did not expect such a mistake because the previous session was clear, with change in color sequence being normal on the edge every click. When closing the curve, the user noticed it was wrong because of the difference in the color sequence. This fact was interesting for some participants who play a memory game, where the player has to follow an aleatory sequence. It was suggested to create a game using the idea of this development to curves associated with a face tracker, something like the 80’s game called Simon Says or Genius in Brazil. Additionally, there is a significant area for development in accessibility, ranging from custom-made interaction styles, for people with limited mobility through support for indirect interaction methods, thus allowing personalization of interactions that can be adapted over time.

4 Conclusion

One of our goals by using a high-level language such as MatLab ^® , was to reduce as much as possible the verbosity and provide rapid prototypes to understand issues about interaction styles and users experience. As a consequence, we have also provided better maintenance, usability and understanding of the method to draw curves with a face tracking mouse. One of the best results of this research was to find that if equal conditions are provided for people with or without physical disabilities, their performances are equivalent. There are also essential advantages in using interaction methods, such as prevention from repetitive strain injuries and progressive degenerative diseases (e.g. CMT). It is better to create long-term skills, as seen in the first session, since a new method of interaction can be stressful and cause the participant to the drop out from the trial. Further studies are necessary to study the possibility of enlarging the target point and creating a virtual keyboard to access the coordinates in the grid. We have also intended to create an indicator to show where the curve begins. By applying these concepts, we aim to generalize the idea to surfaces in the space. Also, we can implement the game as cited above. Another possible work is to obtain access via an eye tracker where the user can make less effort to perform actions more quickly, since the eye is the organ moving faster. Therefore, we could associate it with the face tracker for a finer adjustment. Developing these technologies is not just a matter of wanting to help others, but also a question of giving tools to people with reduced mobility so that they can fulfill their potential. And for people without physical problems, these technologies can provide alternative methods of interest.