Automatic Exercise Assistance for the Elderly Using Real-Time Adaptation to Performance and Affect

Abstract

This work presents the design of a system and methodology for reducing risk of locomotive syndrome among the elderly through the delivery of real-time at-home exercise assistance via intensity modulation of a worn soft exoskeleton. An Adaptive Neural Network (ANN) is proposed for the prediction of locomotive risk based on squat exercise performance. A preliminary pilot evaluation was conducted to determine how well these two performance metrics relate by training the ANN to predict test scores among three standard tests for locomotive risk with features from joint tracking data. The promising initial results of this evaluation are presented with discussions for future implementation of affective classification and a combined adaptation strategy.

1 Introduction

Locomotive Syndrome, a disability characterized by the degeneration of locomotive capability, affects a significant portion of the worldwide population and leads to reduced life expectancy, mobility and an individual’s capability to independently complete activities of daily living (ADL) [21, 31]. One of the primary contributors leading to locomotive syndrome progression is a lack of regular lower extremity exercise [20]. Exercises like the squat and lunge, when performed regularly and accompanied with proper diet and lifestyle changes, have been proven to reduce the risk for this disorder and to improve mobility and general well-being in the elderly [7, 20, 25]. Yet for many in this population, lack of a guided training environment can make it tedious to complete the exercises independently in the home [17].

To address these issues, the authors propose the development of an automated environment to provide adaptive, guided resistance training for completion of at-home squat exercises to improve quadricep function. In this case, the two-leg standing squat exercise was chosen as a target exercise task for such a system because it has proven to be highly beneficial in improving the lower extremity muscle strength necessary to improve locomotion [8, 10] and is a fairly prevalent exercise within this population. To provide adaptive resistance, a wearable, soft exoskeletal suit has been developed consisting of 10 pneumatic gel muscles (PGMs) distributed along the hip, knees and ankles which, when actuated, contract in a similar manner to human muscles to provide resistance during the standing (extension) phase of the squat. While the wearer is guided by a coach to complete squats in a virtual reality (VR) gym environment, these force-generating muscles are automatically controlled in real-time by adaptation software that uses an artificial neural network (ANN) to assess locomotive risk based on squat performance, and in future work, a second ANN that classifies emotion based on real-time physiological data.

It is argued that, by being observant of an individual’s locomotive risk and emotional state in a manner similar to real coaches and physical trainers, this system can provide the appropriate level of resistance to build quadricep strength while completing routine squat exercise. To enable an automated system to perform this assessment in real-time, it is first necessary to examine the relationship between an individual’s performance in the squat task and that individual’s level of locomotive risk, which can then be utilized to assign the appropriate level of actuation to the PGMs along that individual’s legs. This relationship is explored and evaluated in this work by preliminary training of a neural network using 13 healthy subjects in the age range of 20–35. Results of the training indicate the network’s predictive accuracy for this age group in the three different tests of locomotive risk and lower extremity muscle function designed by the Japanese Orthopaedic Association (JOA) [15]. Future work is described that would validate these results by expanding the training set to include the target population of elderly users, and methods for classification of affective data, combined classification, and adaptation of PGMs based on this classification are presented.

2 Related Work

Several fields of research are encompassed by the proposed system for exercise assistance, including the application of ANNs to exercise performance evaluation and classification, the quantification and parameterization of squat performance based on real-time-measurable indicators, the quantification and assessment of risk for locomotive syndrome, and the classification of affective state in real-time. The relevant findings within these fields are summarized and their application toward the proposed system is discussed.

2.1 Neural Networks in Motion Performance

ANNs have proven invaluable and highly effective when applied toward the automated assessment and classification of exercise performance in several specific cases. Oniga and Suto successfully applied ANNs toward the recognition of a variety of daily activities as well as arm and body posture in the assistance of independence of sick and elderly individuals in their daily lives [23]. Cary, Postolache and Girão carried out pose and gesture recognition using ANNS and a Kinect camera to assist therapists in monitoring health information of patients for use in physiotherapy assessments [2]. Recent work has also emphasized the use of ANNs in predicting competitive performance in athletic tasks such as swimming [26].

These findings suggest that the application of these classifiers toward the assessment of locomotive risk based on squat performance may help to determine how well the two relate. In this case, the relationship can be examined by observing the accuracy attainable through iterative learning of a simple feedforward artificial neural network when the parameters of squat performance are provided as input and parameters of locomotive risk are provided as output. Hence, this preliminary evaluation is the focus of this study. Once a suitable means for real-time assessment by means of Artificial Intelligence has been determined, its application toward the control of assistive and rehabilitative robotics for exercise, particularly in the control of exercise intensity, is well-noted in recent review [22].

2.2 Parameters of Squat Performance

Recent work in squat performance classification and assessment is used as the basis for the selection of input features to the ANN. A primary restriction of the design of this system is that the system’s method of tracking squat motion data should be as noninvasive and cost-effective as possible [18], which lends itself to the use of external joint tracking via the usage of a depth camera as in [2]. Hence, kinematic information that can be derived from real-time data on the joints of the lower extremity (hips, knees, trunk, spine, etc.) is of particular applicability. Based on the findings in Escamilla’s observation of knee biomechanics within the squat exercise [10], the angular displacement (range of motion, from standing to squatting) of the knees and hips, in additional to the lateral motion (or shakiness) of the knees, are several useful indicators that can be externally tracked. Furthermore, an estimation of the Center of Mass (CoM) of a subject has been the focus of several studies observing balance during squat tasks and other exercises [12, 14].

Finally, based on sit-to-stand testing as a common clinical evaluation technique [30], the time required to stand during a squat exercise and the number of squats completed within a time limit are also relevant features of performance. These indicators can be quantified only if a system can recognize the various phases of a squat task and, by extension, recognize a completed squat by a user, provided only with real-time joint tracking information on that user. Consequently, a simple method of differentiating between the phases of a squat is presented as a part of the design of this system.

2.3 Locomotive Risk Quantification

Classification of locomotive syndrome in individuals, particularly within the elderly population, is a recent accomplishment of the JOA, who has developed a 3-assessment method known as the Short Test Battery for Locomotive Syndrome (STBLS) [15] which can be completed independently by an individual to determine his or her locomotive risk. Each test provides a score relating to several different factors of individual performance and experience that relate directly to that individual’s lower-extremity strength.

In the first test of the STBLS, known as the Stand-Up test, the subject must stand from the lowest height possible out of four seated heights of 40 cm, 30 cm, 20 cm, and 10 cm, using one leg or both legs, and to hold the standing position for 3 s or longer. This test typically begins with the easiest scenario of standing with both legs from a 40 cm height, then has the subject progressively lower the standing height, and if it is possible to stand from all heights using both legs, switch to using one leg with the same strategy. A score is assigned to the subject based on the lowest height from which he or she was able to stand and whether or not the subject was able to do so with both legs or one leg. A score of 1 through 4 denotes the ability to stand with both legs from a height of 40 cm, 30 cm, 20 cm, and 10 cm, respectively, while a score of 5–8 represents the ability to stand with one leg from the same respective range of heights. A score of 0 denotes the inability to stand and remain balanced for 3 or more seconds in any of these scenarios. The strength of several muscle groups in the lower extremity, especially the quadricep, and the subject’s balance in the one-leg scenario, are all determined by the scoring of this assessment. It is perhaps the most closely related as an assessment task to the squat exercise.

In the second test, known as the Two-Step test, the subject begins by standing with both feet aligned, and must then take two strides forward, moving as far forward as possible with each stride while avoiding falling or losing balance. The distance from the subject’s starting point to the ending point of the second stride is measured in centimeters, and then normalized by dividing by the subject’s height in centimeters. This ratio represents the subject’s score in this test. The score is invalidated if the user falls or loses balance during the strides. This assessment tests the gait stability, balance and lower extremity musculoskeletal strength of the subject.

The third test of the JOA STBLS is known as the Geriatric Locomotive Function Scale of 25 questions, or GLFS-25. It is a 25-question assessment that asks the subject a range of questions related to the level of pain or difficulty experienced in completing a variety of daily tasks, walking and other activities over the last month. Responses to each question are given as a value from 0 to 4 with 0 indicating the least pain, difficulty or discomfort and 4 indicating the greatest. These scores are totalled over all 25 responses to give a final score for the subject. In this case, a lower score indicates a healthier subject with a lower risk for locomotive syndrome. It provides a measure of the subject’s mobility and motor ability as well as the effects of these factors on social participation.

To reach a final decision on a subject’s clinical risk of locomotive syndrome, the JOA has also established clinical decision boundaries that relate the three scores above to a single locomotive “risk level” describing the subject [31]. Risk level 2, the highest level, is determined when the subjects scores 2 or less on the Stand-Up test, or less than 1.1 on the Two-Step test, or gets a GLFS-25 score of 16 or higher. Risk level 1 is attributed to individuals who meet none of the conditions of risk level 2, but score 4 or less on the Stand-Up test, less than 1.3 on the Two-Step test, or 7 or higher on the GLFS-25. Risk level 0 is assigned to healthy subjects who score higher than 4 on the Stand-Up test while also achieving a Two-Step score of 1.3 or higher and a GLFS-25 score lower than 7. Thus, using the STBLS, three levels of locomotive risk classification are possible for a subject.

2.4 Affective Classification

Although it has been made clear that adaptive coaching agents can greatly benefit from responsiveness to user performance [11, 27], adaptation to affect is significantly less explored, as well as how affective adaptation should manifest itself from the perspective of an autonomous virtual trainer. Adjustment to affect is particularly important for the elderly population as recent studies have shown that it can be at least as effective as skill-based adaptation in exergames for this population [1, 16].

Fortunately, recent work has supported the use of biometric signals (Heart Rate Variability or HRV, Galvanic Skin Response (GSR), and Skin Temperature (ST)) for real-time affect determination from the perspective of stress (negative affect) [29] and flow (positive affect) [3, 6]. Given that real trainers respond and adapt training to emotional output [13, 29], it is proposed in this work that an autonomous system should consider affective data in determining the amount of support to provide the user.

To facilitate this adaptation strategy, it is necessary to define how stress and flow are derived from the signals of HRV, GSR and ST, under the assumption that these two signals can be measured in real-time without interfering with exercise. Real-time stress detection has been achieved by Cho et al. in [4] from biosignals related to heart rate, skin response and skin temperature using a rapidly-trained kernel-based extreme learning machine (K-ELM) classifier across 5 distinct classes: baseline, mild stress, moderate stress, severe stress and recovery. One challenge with the approach is the invasiveness placed by the high quantity of finger-worn sensors on the ease of setup and exercise with the system. To alleviate this, the approach of Ciabattoni et al. [5] using a smartwatch may be considered. In this case, stress is reduced to a 2-class decision problem using features of Respiratory Rate (RR) frequencies and means, Galvanic Skin Response (GSR) mean and standard deviation, and Body Temperature (BT) mean and maximum. The features are used in the training of a 1-NN with Euclidean distance metric, and achieve a classification accuracy of almost 90% for stress classification indicating a potential tradeoff between sensor intrusiveness and classification accuracy.

The derivation of “flow”, or a measure of the amount of focused engagement of the subject with the interface, is a far less validated approach. While there is plenty of evidence to support a strong effect of flow-state on positive learning experience, the methods of deriving it in real-time require further validation, particularly when such methods rely on non-intrusive sensing. Some studies have indicated a correlation between signals related to heart rate and skin response and flow-state [9]. Martinez et al.’s work [24] found that features extracted from heart rate and GSR signals such as average and minimum heart rate can serve as good indicators in the creation of an ANN designed to classify flow state in the standard 3-class set (boredom, flow, frustration). However, it is important to note that heart rate can increase naturally as a result of exercise. This presents the challenge of deriving a method to isolate the variation in mean and minimum heart rate induced as a result of emotional output from the component of this variation attributed to exercising.

3 Methodology

To address the challenges posed above, an at-home exercise environment has been developed to provide automated physical resistance and virtual guidance during the squat exercise. Components of this system include a wearable soft exoskeleton for the lower extremity for resistive strength training at the hip, knees and feet, a VR training gym with a virtual coach avatar to lead the subject in the completion of squats, a depth camera for real-time joint tracking, a feedforward ANN to evaluate the relationship between squat performance and locomotive risk, and a mechanism for real-time configuration of the pneumatic muscles on the soft exoskeleton based on the outputs of this neural network and the outputs of an emotional state classifier (to be implemented in future work) for real-time automatic squat exercise assistance. The design and configuration of each of these components are described as follows:

3.1 Squat Assistance Exoskeleton

A primary goal of at-home squat exercise is to build strength in the quadriceps and other muscle groups in the lower extremity, particularly in the elderly population wherein the musculoskeletal system can begin to degenerate. As such, a soft exoskeletal suit was designed as a wearable device for at-home strength training with a focus on the squat task. The suit, pictured in Fig. 1, consists of a series of soft pneumatic gel muscles introduced in previous work [28] attached at three key sections along the hips and legs to worn velcro bands. These PGMS are actuated by air flow through a network of tubes connecting to a single pressurized canister and a central controller, located on a worn belt, which are responsible for regulating air pressure to the valves and actuating the muscles. In this case, actuation refers to the contraction of a PGM, which causes it to shrink at both ends and, as a result, exert a pulling force at both ends to which it is attached. The same pumping mechanism can be used to expand the muscle, relaxing its pulling force at either end.

Fig. 1.

First prototype of the soft exoskeleton for squat exercise. Pneumatic gel muscles are synchronized with a belt-worn controller which communicates wirelessly over Bluetooth with the training software.

Fig. 2.

Illustration of several squat performance parameters and the placement of PGMs across the lower extremity on the exoskeleton. Dotted lines represent the angles and motion values for knee angular displacement, hip angular displacement, and knee lateral motion. Locations of the 10 PGMs are shown in red. (Color figure online)

In total, 10 PGMs are distributed across the exoskeleton. Two are used to connect the upper thighs to the pelvis along the outside, while four are used to connect the thighs to the calves along the outside and inside, and four are used to connect the ankles to the outer and inner shin. The locations of these PGMS are illustrated in Fig. 2. Upon receiving a signal from the controlling software, the controller on the suit can actuate or relax the corresponding PGMs on the exoskeleton. When actuated while the subject is standing back up from a squatted position, these PGMS provide resistive force that increases the intensity of the task. For squatting down, this force becomes supportive, as it helps pull the subject into a squatted position. This supportive force, however, is unused in this system, as it is unnecessary and can even be dangerous if it moves the user beyond the appropriate range for a squat.

3.2 Virtual Reality Coach Software

To help guide the user through the completion of routine squat exercise in the home, a VR gym environment including a virtual coach avatar and player avatar have been developed in the Unity 3D platform. To interact with the environment, the user wears an Oculus VR headset and starts the application, at which point he or she is immediately placed in the VR gym space facing the virtual coach. A screenshot of this environment is provided in Fig. 3. The user’s joints are tracked in this environment using the Intel RealSense D435 depth camera and NuiTrack joint tracking library. A Microsoft Kinect V2 camera was used in an earlier prototype, but was bulkier by comparison and tracked a large amount of unnecessary data for this application.

Fig. 3.

Screenshot of the VR Gym environment, developed in the Unity 3D engine. The virtual coach avatar on the left leads the player through the squat task by demonstrating the next step of the squat, while the player avatar on the right (optionally hidden) reflects the player’s movement as he or she follows the coach.

Fig. 4.

Overview of the various phases of squat exercise as determined by the system. C represents the calibration phase while 0–3 represent the fully upright, squatting, fully squatted, and standing phases of the exercise, respectively.

In the first few seconds of gameplay, the user is asked to stand upright without moving so that the system can calibrate the position of the joints for tracking. During this time, the light beside the virtual coach, pictured in Fig. 3, displays red. Once this light turns green, the coach begins guiding the subject by moving into a squatted position. The user is then asked to follow along. As the user moves down into a squat, the light turns yellow to indicate that the system detects that the user is in the squatting phase. Once the user has reached a squatted position and stopped moving for at least one second, the light once again turns green and the virtual coach leads by returning to a standing position. Once again, the user follows and the light turns yellow as the user is standing up. Once the user is fully upright for at least one second, the process continues and the light turns green once again.

By observing the vertical movement of the user’s spine base joint, the system is able to recognize the various phases of squat activity and the virtual coach leads the user accordingly. The entire process of the squat, as detected by the system, is shown in Fig. 4. An optional second avatar, shown to the right of the coach avatar in Fig. 3, copies the subject’s motion rather than leading, giving the subject a second external point of reference to compare his or her motion to that of the virtual coach.

3.3 Joint Tracking for Squat Performance Data

While the subject completes squat exercises with the system, joint angle data is read from the depth camera via the Nuitrack library to provide joint angle and joint displacement data of the subject at each frame (approx. 60 frames/second). Based on the findings in Sect. 2.2, the following 9 values are selected as features for squat performance:

Input. Squat performance is represented as a vector of values {$i_1$, $i_2$, $\dots$, $i_9$}, one per squat. The current configuration of this input vector is the following:

Input values $i_1$, $i_2$, $i_3$, and $i_4$ correspond to left knee angular displacement, right knee angular displacement, left hip angular displacement, and right hip angular displacement, respectively. These values are obtained by observing the corresponding joint angles at the fully upright and fully squatted positions of the subject on each squat, and finding the differences between each pair of joint angle values. Raw values fall within the range [0, 180] and are normalized to the range [0.00, 1.00].

Input values $i_5$ and $i_6$ correspond to the left and right knee lateral motion average. These values are obtained by observing the joint displacement values of the left and right knee along the frontal plane, sampled once per frame at roughly 60 frames per second and averaged over a single squat. Essentially, these values represent the shakiness of the knees or the load placed on the knees during exercise. Absolute value is used as we are interested in the magnitude of motion and not direction. The average of these values will lie in the range [0.00, 3.50] whose maximum represents the physical limitation of knee lateral motion expressed as a coordinate value within Unity coordinate space. This value is normalized to the range [0.00, 1.00].

Input value $i_7$ corresponds to the Center of Mass stability along the frontal plane. This value is roughly estimated by observing the vertical and horizontal differences between the left and right hip and knee joints to estimate a frontal plane CoM location, and measuring the displacement of that point over a single squat. A value of 1.0 represents perfect stability while 0.0 represents motion of 3.50 or higher as in the previous case with knee lateral motion.

Finally, input values $i_8$ and $i_9$ correspond to the time required, in seconds (including milliseconds) to move from fully squatted position to fully upright position, otherwise referred to in this work as “standing time” or “extension time”, and the number of repetitions completed by the subject.

3.4 ANN for Locomotive Risk Based on Squat Performance

A feedforward ANN has been developed which accepts as input the squat performance features derived in Sect. 3.3 and attempts to predict the individual’s scores on the three tests of the JOA STBLS described in Sect. 2.3. The structure of this neural network is shown in Fig. 5. The nine inputs for squat performance are supplemented with a bias node to create 10 total inputs and passed to the ANN with a single hidden layer of size 6 and an output layer of size 3. A leaky rectified linear unit (LReLU) activation function is utilized in this case to minimize the impact of vanishing gradient challenge on the learning process of the network:

$$\begin{aligned} f(x_i)=max(0, x_i) + 0.01 * min(0, x_i) \end{aligned}$$

(1)

Outputs of the network are a 3-tuple $O = \{O_1, O_2, O_3\}$ and represent the three scores obtained by completing the JOA STBLS as detailed in Sect. 2.3.

Fig. 5.

Feedforward ANN used to predict locomotive risk as three scores (sit-to-stand, two-stride, 25-question assessment) representing the three tests of the JOA STBLS. Bias nodes, included in the input and hidden layer, are not shown in the diagram. Connections represent the weighted, activated output of the previous layer as it is passed as input into the next layer.

3.5 Real-Time Adaptation

To configure the pneumatic muscles based on the output of the ANN, the range of possible configurations and support levels must first be defined formally. For all pneumatic muscles present in the soft exoskeleton, the supportive capability of these muscles is represented as their strength of actuation discretized in the integer classification range [0 ...n], where n is the number of distinct actuation levels derived by dividing the maximum force of actuation by a value representing the minimum change in force required for the average human to perceive a difference in the force felt in the lower extremity:

$$\begin{aligned} n = F_{max} / \delta F_{pmin} \end{aligned}$$

(2)

Hence, a range of actuation forces $S = [s_0 \dots s_n]$ can be defined wherein $S_i = i * \delta F_{pmin}$ in order to complete the discretization of pneumatic support into mappable values.

Squat Performance. Next, it is necessary to define the dimensional reduction function $f(O) = f(O_1, O_2, O_3)$ which outputs a singe value in the range [0 ...m], where m is the number of distinct classifications of performance.

Fortunately, this dimensional reduction strategy is already derived by the JOA using the clinical decision boundaries in [31]:

$$\begin{aligned} f(O_1, O_2, O_3) = {\left\{ \begin{array}{ll} 0 &{} \quad O_1>4 \text{ and } O_2>=1.3 \text{ and } O_3<7\\ 1 &{} \quad O_1<=4 \text{ or } O_2<1.3 \text{ or } O_3>=7\\ 2 &{} \quad O_1<=2 \text{ or } O_2<1.1 \text{ or } O_3>=16 \end{array}\right. } \end{aligned}$$

(3)

The output of this function represents the corresponding stage of locomotive syndrome and the value ranges resulting in each output represent the clinical decision limits of these tests as proposed by the JOA. Hence, the outputs of the neural network can be reduced to a single value in the integer range $[0 \dots 2]$ in which case $m = 3$.

At this point, it is possible to then map $f(O)$ to $S$ as follows:

$$\begin{aligned} g(f(O), S) = {\left\{ \begin{array}{ll} s_0 &{} \quad f(O) = 0\\ s_{n/2} &{} \quad f(O) = 1\\ s_n &{} \quad f(O) = 2 \end{array}\right. } \end{aligned}$$

(4)

where $n = F_{max} / \delta F_{nmin}$.

For even values of n, we simply discard the highest force value $s_n$ from S to revert to an even-valued n. In the case where $n = 1$, the single value in S is mapped to all outputs $f(O)$. For $n = 2$, the smaller value is assigned to scores of 0 and 1 while the larger value is mapped to the risk score of 2.

This provides a coarse mapping wherein 3 evenly distributed actuation values are selected from the range of distinct values available to adapt the system solely to the three-class “risk factor” of an individual as predicted by the ANN. However, this mapping alone does not take advantage of the dynamic nature of squat performance indicators, which can provide more detailed information about an individual’s current skill level than the JOA assessment can alone. Perhaps one of the most powerful indicators of dynamic performance, or the performance of a specific squat attempt relative to an individual’s standard, is the input value $i_7$ representing the lateral stability of the individual’s CoM during a squat attempt.

Using the CoM value, one can derive an offset value z that is used to determine a more accurate value for $g(f(O),S)$, and can allow the output of this function to enable more dynamic adaptation between squat attempts. Assuming a Gaussian distribution of CoM values, z can be determined using the following function:

$$\begin{aligned} z = {\left\{ \begin{array}{ll} 0 &{} \quad n < 5\\ i_7 * floor((n-3)/2) &{} \quad n >= 5 \end{array}\right. } \end{aligned}$$

(5)

This then allows us to define $g(f(O),S)$ as follows:

$$\begin{aligned} g(f(O), S) = {\left\{ \begin{array}{ll} s_{0+z} &{} \quad f(O) = 0\\ s_{n/2+z} &{} \quad f(O) = 1\\ s_n &{} \quad f(O) = 2 \end{array}\right. } \end{aligned}$$

(6)

Alternatively, we can represent the output values $\{O_1, O_2, \dots , O_t\}$ (where t represents the number of squats performed) as 3D points within a centroid clustering space, where the value n from above represents the number of classes. To configure the classification, the first n values of output are obtained from the user after $t = n$ squat attempts and form the set Y. During this time, the system defaults to the maximum support level (or minimum resistance level) of $s_n$ to ensure no risk to safety. The values of Y are then sorted as the first n centroids of the clustering space based on their distance from the origin (0, 0, 0). Should there be less than n distinct output values among this set, then the number of classes are reduced to match the number of distinct values among this set.

From this point forward, the Nearest Centroid Classification [19] method is used to assign a support configuration to each future squat attempt. If the values $\{O_1, O_2, \dots , O_t\}$ are highly similar, the dynamic adjustment strategy proposed earlier can be utilized to offset the classification and remap within the full range of S. Then, we can represent the final actuation value associated with an output O of the ANN as follows:

$$\begin{aligned} g(O,S) = s_{c+z} \end{aligned}$$

(7)

$$\begin{aligned} c = argmin_{i \in Y}||O_t - O_i|| \end{aligned}$$

(8)

where z is defined as above.

Affect. This work proposes the classification of stress and flow-state using ANNs in real-time. Based on the work presented in Sect. 2.4, stress can be classified in the range {low stress, moderate stress, high stress} using the input features {HRV mean/std, GSR mean/std, deriv. GSR mean/std, BT min/max/avg.}, and flow-state can be classified in the range {boredom, flow, frustration} using the input features {HR mean/std/min, GSR mean/std}.

We then represent affective state as a two-tuple $AS = {STR, FS}$, where STR is stress level of integer index in the range [0 ...2] and FS is flow state of integer index in the range [−1 ...1]. If the output of AS is represented as a point in two-dimensional space, a secondary goal of the coaching system’s support then becomes to reduce the distance of this output from the optimal value (0, 0), which represents the lowest output of stress and highest output of engagement/flow.

We can define a function that reduces AS to a single dimensional output reflecting the above goal, assuming that stress can be detected as a result of both boredom and frustration:

$$\begin{aligned} f(AS)=STR*FS \end{aligned}$$

(9)

This output is given the range $[-2 \dots 2]$ and reflects both the degree of adjustment necessary and the direction of that adjustment. Now we can define the output actuation value $s_t = g(f(AS))$ given the previous actuation value $s_p$ as follows:

$$\begin{aligned} g(f(AS)) = s_i \end{aligned}$$

(10)

$$\begin{aligned} i = clamp[0 \dots n](p + \frac{f(AS)}{2} * n) \end{aligned}$$

(11)

Combined Adaptation. It is of particular interest in this work to combine the adaptation based on performance and that based on affect into a single system so that the system can respond to both indicators. This combined approach has not been explored in previous work, which focuses on one type of dynamic difficulty adjustment or the other. Given two channels of individual response to a virtual exercise task (performance and affect), and a measure for each channel of the degree of deviation in the player’s response from the optimal value for that individual, it is proposed that a system can provide the most effective dynamic difficulty adjustment (DDA) by adapting to the channel with the higher degree of deviation.

This implies that, as an adaptation strategy, given the values $g(f(O),S)=s_a$ and $g(f(AS),S)=s_b$, the system can choose between the two using the following strategy:

$$\begin{aligned} g(f(O),f(AS),S) = {\left\{ \begin{array}{ll} g(f(O),S) &{} \quad f(O) > ||f(AS)||\\ g(f(AS),S) &{} \quad f(O) < ||f(AS)||\\ s_c (c = \frac{a+b}{2}) &{} \quad f(O) = ||f(AS)|| \end{array}\right. } \end{aligned}$$

(12)

4 Evaluation: Neural Network Training

To determine whether the ANN described above is capable of converging to a sufficiently reliable model for predicting locomotive risk based on squat performance, a preliminary pilot study was conducted wherein 13 members of the research team acted as subjects to train the ANN. The goal of this training was to determine to what error values the network converges in its prediction of each of the three locomotive assessment scores for the subject group as an approximate preliminary measure of the strength of the input feature set in relating to these assessments. In simpler terms, the potential relationship between squat performance and locomotive risk was explored through observation of post-training prediction results of the ANN.

4.1 Procedure

Thirteen healthy subjects (11 male and 2 female), in the age range 20–35 with no motor impairment, participated in this preliminary study. Each subject began by completing each of the three assessments of the JOA STBLS. The subject’s three scores are written to a log file, and serve as real values against which the results of the ANN’s prediction for that subject can be compared to determine error and perform gradient descent. After completing all three assessments, each subject then wears the Oculus VR headset and is asked to perform one minute of squat exercise in the virtual gym under the guidance of the virtual coach.

The exoskeleton is unused in this preliminary study as it is used to determine baseline performance for each subject. Once one minute of squat exercise is complete, the features of squat performance are collected and stored in a log file for each subject. This data is then passed as input into the ANN and used to generate three predicted score outputs. The error between these predictions and the subject’s actual STBLS score is used to adjust the weights on the network. This process is repeated 100,000 times with the subject’s performance data from the single exercise session.

4.2 Results and Discussion

Results of the ANN’s predictive capability after training on the data of the 13 subjects is shown in Figs. 6, 7 and 8. For the GLFS-25, the ANN performed at an average error of 1.268 across all 13 subjects as shown in Fig. 6. This result is expected, as the GLFS-25 assesses factors such as pain, activity completion and mobility over a one-month period that are potentially not well-represented by execution of the squat task. In some cases, as in Subject 11’s case, this error is high enough to misclassify the subject’s locomotive risk level, which should be considered in future reconfiguration and pruning of the network. For the Standing test, the ANN converged to an error rate of 0.387. The remaining error may reflect the fact that the standing test is a measure of balance as much as strength, and therefore includes a one-leg standing task, which is not included in the standard squat task.

Fig. 6.

Results of the ANN’s prediction of 25Q score compared to actual score on 13 subjects after training.

Fig. 7.

Results of the ANN’s prediction of Sit-Stand score compared to actual score on 13 subjects after training.

Fig. 8.

Results of the ANN’s prediction of Two-Stride score compared to actual score on 13 subjects after training.

Fig. 9.

Results of the ANN’s prediction of Sit-Stand, Two-Stride, and 25-Q score compared to actual score (after training) on a single new subject with which the ANN was not previously trained.

Finally, the 2-Stride score prediction was significantly more accurate than the other two scores, converging to an average error of 0.063 over all subjects. While this may seem unusual since the 2-Stride test is a walking task and measure of gait stability, it can be noted that many of the features selected as inputs into the NN, such as angular displacement of the knees and hip and CoM displacement, are also useful as indicators of gait posture and can therefore relate well to this measure. These initial results were verified by testing the ANN on a 14th subject whose data had not been used to train it. The subject completed the same procedure as the training subjects, and the results of the ANN’s output for that subject are given in Fig. 9. Error rates of 0.268, 1.131 and 0.010 were obtained for the system’s prediction of the subject’s Standing test score, GLFS-25 score and Two-Step test score, respectively.

4.3 Limitations and Future Work

Several limitations were present in this initial study. The first is in the variability of scoring among subjects. Since all subjects were young, healthy subjects, they all scored fairly highly on all three assessments, with very little variablity among their assessment scores. This resulted in a poor training dataset, so the scores needed to be normalized and rescaled based on the best and worst performance in each test among the group. This scaling is reflected in the vertical axis of Figs. 6 through 8. For example, the range of performance in GLFS-25 was rescaled to 0–9. Furthermore, this sample set is not representative of the elderly population, for whom this system and these standardized assessments were designed. Hence, future work will train the ANN on data from users in this population to determine whether the results may extend to this population.

5 Conclusions

Having successfully determined the stronger and weaker relationships between squat performance and locomotive risk through the training of an ANN, the next step is to reconfigure the network and introduce new input features that incorporate balance and pain/discomfort to improve predictive capability. Once this is achieved, as proposed in this work, a dynamic adjustment mechanism can be implemented wherein the supportive strength of the pneumatic muscles on the wearable suit are adjusted on different levels based on the system’s assessment of the user’s risk. To evaluate this system, biometric data such as heart rate and galvanic skin response will be recorded to form an estimate of the affective response (negative-positive) of these subjects during exercise and compared with self-reported responses to post-exercise surveys to determine whether the implementation of adaptive support increases the quality of a subject’s exercise experience during the squat task. The targeted population for this study will be elderly subjects, particularly those with locomotive difficulty, with the ultimate goal of improving physical and emotional response to at-home squat exercise.