Keywords

1 Introduction

Training and exercise programs, under the guidance of skilled therapists and trainers, have become important tools during the rehabilitation process. Traditionally, these programs have been difficult to perform at home due to the need for a trainer to observe and provide feedback on body compliance during the performance of the exercises. Non-compliance caused by impaired proprioception can result in serious injuries and slow the progression of motor learning if left uncorrected during exercise performance. As a result, research is actively seeking new automated methods for the detection and intervention of non-compliance in unsupervised environments. There are two main sub-problems that together comprise the problem of non-compliance prevention:

  • Anticipation: how does a system anticipate the occurrence of non-compliance?

  • Intervention: how does a system intervene to correct the behavior of the individual to prevent prolonged non-compliance?

In this project, we introduce an anticipatory system to detect and intervene before non-compliance or compensation occurs. Although prior work has looked at detection of body states during exercise, it is important to explore anticipation since any time spent in a non-compliant state could harm the individual. Once it is determined that the user may be at risk, the system provides multimodal feedback through auditory, visual and haptic cues to correct the harmful behavior before it results in injury.

As an initial application, the area of exercise rehabilitation for cerebral palsy is used to provide a context to demonstrate the effectiveness of the system in anticipating and correcting body position. In this paper, we present the SmartGym, an intelligent Total Gym Pro that monitors an individual’s performance and provides feedback through haptic, auditory and visual cues. An overview of related work in Sect. 2 shows the need for an effective tool for anticipation, rather than detection, of potentially dangerous body states, which we explore in our design of the SmartGym. In Sect. 3, we differentiate anticipation from prediction and give an overview on how this is achieved within the system. In Sect. 4 we describe the details of our implementation and justify our design decisions in the feedback that the system provides. In Sect. 4 we evaluate our initial prototype for both usability and effectiveness in a case study involving an adult with cerebral palsy who has developed a hemiparetic lower extremity and her physical trainer. We deploy the system within the subject’s regularly scheduled training and present feedback to test the effectiveness of replicating the trainer’s feedback. We conclude in Sect. 5 with directions for future work.

2 Related Work

Because of the importance and impact of at-home exercise during rehabilitation, much work has been done in the area of remote rehab through sensors to provide the performance feedback that a trainer would physically provide to individuals in clinic.

2.1 Microsoft Kinect for Remote Rehabilitative Exercise

The Microsoft Kinect (Fig. 1) has proven to be an important tool in these applications since it is an effective tool at gathering motion data and is much cheaper than traditional solutions. Prior work has demonstrated the effectiveness of this sensor in this domain by showing its ability to measure the position and trajectory of each joint, the working envelope of each body member, the average velocity, and a measure of the user’s fatigue after an exercise sequence [1].

Fig. 1.
figure 1

Microsoft Kinect sensor

Full size image

In Physio@Home, the authors explore the use of a Kinect camera to prevent re-injury during at-home rehabilitative exercises [2]. The system takes a multi-camera approach in order to provide the individual with visual feedback to guide them in the performance of finer-grained physio exercises. The user is shown wedges that indicate the direction and angle that the specific movement requires and are expected to perform those movements while keeping their arm within the wedge. Similarly, other approaches have looked at using the Kinect to affect balance ability in injured athletes [3]. In this approach, the authors used a Kinect intervention to enhance balance ability and compared this to traditional physiotherapy. The proposed intervention provided feedback through a gaming interface that increased in difficulty over a 10-week period.

2.2 Wii Balance Board for Remote Rehabilitative Exercise

The Wii Balance board (Fig. 2) has also proven to be a useful and cost-effective solution for obtaining accurate measurements of foot pressure and center of balance [4,5,6]. In evaluating the accuracy of the board, research has shown that it is best used for relative measures using the same device and should be used for low-resolution measurements [4].

Fig. 2.
figure 2

Wii Balance board

Full size image

In [5], the authors explore the use of the Wii Balance Board as a gaming interface to improve balance for children with cerebral palsy. The study found a significant improvement based on the Trunk Control Motor Scale in the subjects’ trunk control and balance. Similar results were found in studies that looked at the effectiveness of a Wii Balance Board intervention for balance improvement in patients with acquired brain injury as well as individuals with Parkinson’s Disease [6, 7].

2.3 Anticipatory Computing in Health Applications

Although much of the work in the area of health intervention has leaned more towards predictive rather than anticipatory, Pejovic and Musolesi have proposed the potential for applications of anticipatory computing within the emerging field of digital behavior change interventions [8]. The authors have referenced UbiFit [9] and BeWell [10] as two applications that have taken very rudimentary steps towards the inclusion of anticipation in mobile health applications as well as provided potential architectures for future applications in this domain [11]. UbiFit is a personal health application that was designed to monitor user’s weekly activity and provide subtle feedback if he or she has not had enough activity. The app shows a garden that thrives if the user is meeting activity goals and remains barren if the user has remained inactive for too long. BeWell is a mobile application that monitors a user’s health along three dimensions: sleep, physical activity and social interaction. Much like UbiFit, this application provides intelligent feedback to the user to promote better health through an ambient display of an aquatic background that becomes more active the healthier you are.

2.4 Limitations of Prior Work

The major limitation of prior work has been that it has used these sensors in applications that solely provide feedback regarding past or current performance of exercise tasks. In traditional behavior change interventions, the therapist relies on self-reported experiences and progress to guide the intervention. In digital behavior change interventions, the interventions are personalized in real time, responding to the immediate user’s context. In anticipatory digital behavior change interventions, however, interventions are modified based on the model of the predicted context [8]. Because the concept of anticipation within behavior change applications is so new, few applications exist that are truly anticipatory and thus the literature is lacking. This is important since injury can very easily occur the moment an individual’s body position becomes non-compliant. Being able to anticipate this past prediction takes a step towards a more preventative approach to re-injury in the rehabilitation domain. SmartGym looks to build upon existing work by predicting a user’s future context and intervening to prevent unsafe states.

3 Proposed Approach

The proposed approach has two main objectives:

  1. 1.

    Anticipate when a user is going to become non-compliant before injury occurs

  2. 2.

    Provide multimodal feedback to an individual so that they can correct their body in real-time

3.1 Anticipation of Non-compliance

Robert Rosen defines an anticipatory system as one that “contains an internal, predictive model of itself and its environment, which allows it to change state at an instant in accord with the model’s predictions pertaining to a later instant” [12]. It is important to first differentiate prediction from anticipation since the two are often incorrectly used interchangeably. Predictive applications are those that simply build predictions of the user’s current or future context. Anticipation is set in the domain of action that is based on the predictions of future context in order to impact the future to the benefit of the user [13].

To achieve the goal of anticipation, the system breaks down the problem of non-compliance by looking at sensing events that occur before risk of injury. In determining the risk of an accidental fall from the Total Gym machine, the system looks at center of balance and postural stability as both of these are key attributes that worsen gradually throughout the individual’s exercise routine. In considering center of balance, the SmartGym measures pressure placed by each foot against the landing plate of the Total Gym Pro and warns the user when compensation occurring between the affected limb and unaffected limb is sensed. Compensation can lead to early fatigue and can also lead to the individual being imbalanced before the start of a movement leading to injury during landing. Secondly, the system considers postural stability by looking at body alignment and warning the user if he or she is leaning to one side while lying down on the machine. By sensing these two indicators, the system can anticipate that the user is going to enter a non-compliant state and provide feedback to correct this before injury.

3.2 Feedback and Actuation

In order to provide feedback to the individual, three modalities are used: auditory, visual, and haptic. The three feedback channels were chosen to map to the feedback that is given by the instructor during supervised exercise. The instructor begins by verbally informing the individual to correct posture or balance. They will then visually show the correct body positioning themselves so that the individual can mirror this. Lastly, they will physically touch the limb or area of the body that is out of position to shift it back into place. This methodology is emulated in the SmartGym since the system begins by making an auditory tone, then displaying a visual feed of their body superimposed on top of what the correct posture should be, and finally vibrating the Total Gym board to inform the user of the direction that they need to shift (Fig. 3).

Fig. 3.
figure 3

SmartGym system design

Full size image

4 Implementation

The SmartGym has been built and integrated with the Total Gym Pro (a commercially available piece of exercise equipment). The system consists of a Wii Balance Board, a laptop with Bluetooth communication, a Microsoft Kinect, an array of pancake motors, an Arduino Uno and a webcam. The laptop serves as the central link between all the devices and uses information from each of the sensors to anticipate when the user is going to enter a non-compliant state.

The Balance Board is positioned on top of the landing platform of the exercise equipment and is used to detect pressure applied by the feet. Each of the user’s feet has two pressure sensors under it to measure the difference between pressures put on the ball of the foot versus pressure put on the heel. This pressure is used to determine balance throughout the exercise motions. The system is able to detect if the user is compensating by using one foot more than the other or, similarly, if the user is compensating by prioritizing the ball or heel of one foot. A webcam is also placed facing the individual’s feet to give a heads-up display indicating where the user’s feet are positioned on the Balance Board to avoid false positives because of incorrect alignment of the feet. This is an important factor given that impaired proprioception is a common side-effect within our target user population.

The Kinect is positioned above the exercise equipment and faces down toward the user to measure joint and skeletal position. This measure gives insight into postural stability during movement and warns the user when his or her body exits the threshold for compliance. Measurements are collected for the shoulders, elbows, wrists, hands, hips, knees and feet. Before starting the exercise, the user is asked to hold a neutral state in which all joints are lined correctly for the upcoming motion. The system captures these positions and uses them to create a bounding box around each of the joints that determines the threshold for compliance. This threshold is set as a variable within the system and can be adjusted by the trainer to become smaller as the individual progresses within their exercise routines. All information from these sensors is streamed to the laptop, which handles all of the processing and gives the auditory and visual feedback.

Two arrays of six pancake vibration motors were embedded onto the Total Gym Pro’s padded board and placed horizontally spanning the outside edges of the board as shown in Fig. 4. Each of these motors has a pressure sensor connected to it which will detect if the user is beginning to lean or slide to one side of the board before or during the exercise. These motors and pressure sensors were connected to an Arduino Uno which served as the logic unit for this feedback mechanism. If the pressure sensors pass a certain threshold, the haptic motors are activated and are used to guide the user back to a neutral state where they are balanced on the board. Twelve motors were used in order to fully span the area on the board that a user’s body could encompass. This allows the device to detect shifts from the shoulders all the way down to the user’s hips. The motors will continue to vibrate until the user shifts his or her body to rectify their postural stability. Given the information that is gathered through the Kinect and presented on the visual interface, this information can sometimes be redundant, but serves to offer the user multiple modalities of feedback that they can choose between. Occasionally, a user’s vision might be pre-occupied with an alternate task and thus haptics can be an effective method of providing knowledge of performance.

Fig. 4.
figure 4

Vibration motor placement

Full size image

The system was designed for use by an individual with cerebral palsy during rehabilitative exercise. Specifically, we look at an exercise where the individual starts with their feet against the landing plate, as shown in Fig. 5, and presses off to slide up and then lands back on the plate. The key components of this exercise are to ensure that the individual has both knees and feet together throughout the entire exercise and to make sure that they are positioned on the center of the exercise equipment in order to avoid falling off or injuring themselves. The trainer’s role is to monitor the individual throughout this exercise and to provide feedback whenever they are breaking these safety guidelines. We have mapped the interface and feedback to the same protocol that the trainer uses in instructing the individual to correct body positioning as shown in Table 1.

Fig. 5.
figure 5

Squat exercise

Full size image
Table 1. Feedback protocol
Full size table

For the purposes of this system, we are only considering individuals with moderate, hemiparetic cerebral palsy as these individuals can often gain better motor control through rehabilitative exercises. However, this design can also be used in many other domains where hemiparesis can occur such as in stroke since the same principles of body compliance and compensation play major roles.

4.1 Interface Design

The graphical interface has been designed to give feedback on several metrics at one time. As shown in Fig. 6, the pressure information is displayed to the user as a heat map to show what areas of the foot might be getting too much or too little pressure. The center of gravity is shown as a circle on top of this display that shifts with the user’s pressure changes. When the circle is in the center of the interface where the target zone is shown, it switches to green to inform the user that they are centered and may proceed with the exercise. Another section of the interface shows the joint positions of the user that is being captured by the Kinect throughout the exercise. Joints that are outside of the threshold for compliance show up as red and a tone is also played until the desired body position is achieved. Haptic motors embedded within the total gym system also inform the user when they are starting to slip off of the Total Gym System so that the user can correct his or her posture.

Fig. 6.
figure 6

Interface design

Full size image

5 Conclusion and Future Work

The SmartGym shows great promise in offering an at home solution for individuals to carry out their rehabilitative exercise programs. Through the detection of center of balance and postural stability as indicators for non-compliance, the system looks to reduce risk of injury while the user is not under the supervision of a trained professional. Future work will look to augment the current anticipation of falls and balance issues with other detrimental cues that could lead to injury during exercise. As a first step, we plan to consider cues for muscle fatigue, as these can also be important factors that cause involuntary non-compliance or compensation. The system will also undergo a much more rigorous evaluation as to the long-term effectiveness of an anticipatory application within this domain. We look to have a subject complete a 6-week exercise program using the system for their at-home training to compare outcomes between traditional physical therapy and the proposed anticipatory approach.