Keywords

1 Introduction

Over the last 10 years, functional training exercises became a popular method to improve muscular and cardiovascular fitness, and many studies, such as [5] and [9], point out the benefits of this type of training regimen - while also suggesting that functional training practitioners have increased risk of foot, knee and lower back pain and related injuries. Nevertheless, works such as [17] and [14] point out a twofold increase of the prevalence of knee pain injuries, whereas works such as [8] show a threefold increase of the prevalence of lower back pain and injuries. Foot and Knee pain have been associated with numerous orthopedic pathologies and injuries of the lower limbs. From high intensity functional training (HIFT), such as CrossFit, to Extreme Conditioning Programs, lower limb injuries correlate highly with unevenly distributed plantar pressure and knee positioning during long-term physical practice and can lead to severe orthopedic injuries if the movement pattern is not amended. Therefore, the monitoring of foot plantar pressure distribution and the spatial and temporal characteristics of foot and knee positioning abnormalities is of utmost importance for injury prevention. Common methods employed by physical educators for the diagnosis of plantar pressure and knee positioning abnormalities are based on human observational approach, predominantly subjective, which is neither reliable nor scalable to a large group of individuals.

Human Activity Recognition (HAR) research on posture and movement information has seen an intense growth during the last ten years, drawing attention of fields such as mobile healthcare and ergonomics. Researchers investigate the recognition of human movement patterns and behaviors to better understand our actions and their context, to detect misguided action and to help people perform better in their daily life or professional activities. Traditionally, the equipment used to track and process these movements patterns were invasive, expensive and unsuited for outdoor experiments, but the development of Internet of Things (IoT) wearable technologies allowed researchers to investigate HAR related questions without the constrains of a laboratory environment.

This work presents a study for foot and knee movement and posture analysis of individuals to help them assess and correct their movements during training, aiming to mitigate the probability of repetitive strain injuries (RSI). Our solution employs an IoT wearable device with a novel sensor array and a machine learning HAR activity classifier for movement pattern recognition. The wearable device comprises three components: an US men’s size 8 insole that houses the plantar pressure sensors, an external protective case that houses the microcontroller and the foot sensors and a knee band that houses the knee sensors. It is equipped with a set of 16 sensors and can collect detailed foot and knee movement and posture information every 20 ms, providing a feature-rich stream of data for local processing at the microcontroller or remote processing at an application server. The machine learning HAR activity classifier employs an individually tailored decision tree algorithm, when processing data locally, or a Random Forest, when processing data at the server, to recognize, within a set of 13 activities, whether the movement pattern is correct or inaccurate. The experimental results show that the HAR classifier achieves an overall accuracy of 87.08% for local processing. After classification, the IoT wearable device uses a component that provides haptic feedback to the user to warn whenever an inaccurate movement pattern is performed.

We conducted an experiment to build our machine learning HAR activity classifier, employing twelve volunteers carefully selected for their diverse characteristics, since a common limitation of the surveyed works that is the prevalence of homogeneous participants. We employed participants from both genders and diverse age categories, some of which with disabilities, such as class II obesity, severe knee injury and blindness. Each participant performed 3 sessions of 30 min of functional training supervised by a certified physical educator, following a routine consisting of the 13 activities of our activity model. We conducted a second experiment with the same participants to validate our HAR classifier, consisting of a 30-min session followed by a semi-structured interview to evaluate usability and user experience questions. Our experiment demonstrates the feasibility of our real-time feedback system and the supporting role it can perform for early intervention and prevention of orthopedic injuries, by providing a mobile, easy to use and affordable long-term monitoring of individuals.

2 Literature Review

This section presents a literature review about wearable-based research projects using feet and knee information for the detection of exercise execution errors. We conducted the literature review in four steps: (i) definition of a research question and its sub questions, (ii) formulation of a search query string, (iii) definition of exclusion criteria and (iv) completion of a quantitative and qualitative data analysis. The research question posed in this work is: What are the wearable-based research projects conducted in recognition of exercise execution to help prevent or treat sport-related injuries? This research question was broken down into four sub-questions:

  • How the incorrect execution of exercises was addressed?

  • How the activity model was structured?

  • What activity classifier was built?

  • What hardware and software setting types, quantities and locations of the sensors were assembled?

The surveyed works may be grouped into three distinct research categories: (i) user feedback and performance assessment, (ii) activity recognition and (iii) prevention or treatment of injuries and diseases.

2.1 User Feedback and Performance Assessment

The study presented in [13] uses off-the-shelf devices, such as the Microsoft Kinect, to evaluate posture and analyze users movement in order to help them assess and correct their movements during a CrossFit training session. Although the experiment focused on only one exercise within a restricted context, the study suggests that the proposed application provides a coach-like feedback useful in the absence of such an expert. The works proposed by [6] and [2] use upper and lower body accelerometer data to provide, respectively, feedback to swimmers and rowers. In [6], a sensor system is used to extract lap times and stroke counts for each lap of the pool, achieving good overall results. However, in [2], even though the experiment was successfully conducted by providing users with immediate feedback, no improvements were found for performance-related parameters. One other work focused on rowers collects femur and lower back kinematic data through an accelerometer-based body sensor network (BSN) [10]. The proposed system was used, alongside optical tracking, to distinguish between good and poor rowing techniques. Also, in [4], an experiment was conducted with professional swimmers to help improve their performance. The researchers used acceleration sensors to monitor relevant swimming parameters for a continuous performance evaluation, while also offering visual, audio and haptic feedback to swimmers. After testing four feedback modes, visual signs were chosen for their fastest reaction time. In the ongoing work presented by [43], an experiment was conducted with professional skiers aiming at improving the trainer-athlete relationship. Through the usage of wearable sensors and visualization software, participants were able to share their observations and impressions. Another work aiming at benefiting athletes with immediate feedback during training is presented by [19]. The researchers propose feedback systems for rowing, table tennis and biathlon professional athletes, providing detailed hardware information.

2.2 Activity Recognition

Many works, such as [21, 23, 25, 35] and [22], rely on plantar FSR pressure sensors to classify user activity according to a previously elaborated activity model. Other works, such as [28, 42] and [23], rely on inertial motion units (IMUs) located on user’s feet for that purpose. Sensor fusion - FSRs and IMUs - is employed by works such as [32, 49, 53] and [39], achieving good overall results. Only a few of the surveyed works used sensors other than ground contact force (GCF) sensors and IMUs, such as infrared sensors [36] or capacitive sensing technology [29] and [41]. Henceforth, the main difference being these works is the machine learning algorithms applied and the context of the experiments. Differently, [54] proposes a novel approach using ultrasonic sensors in a wearable platform to monitor lower limb movements and patterns. Although it is an ongoing work, the results show that ultrasonic systems may be successfully used for gait analysis in running and jogging. The researchers in [16] use in-shoe Ground Contact Force (GCF) sensors to evaluate patients with postural instability making use of a posturography. The wearable-based sensor technology used in their experiment allowed for static and dynamic posturography in clinical and home environments an important assessment for orthopedic diagnosis. There is evidence showing a strong correlation between dynamic in-shoe sensor data acquisition and static pressure plate data acquisition.

The work in [15] presents an extensive research using wearable sensors to understand the best signal processing, sensors and classification methods for classifying health-enhancing physical activities such as walking, running and cycling. It discusses how each activity is best characterized and what sensors should be used to recognize them. Likewise, in the work proposed by [7], activity recognition using wearable sensors provides lifestyle feedback regarding health-enhancing physical activities. However, contrary to the previous study, this one focuses on non-lab activities and sports in unsupervised settings. Considering the proposed activity model, results showed that using both unsupervised and supervised data for machine learning yields similar results to using only supervised data. The work in [37] proposes an optimal sensor set for gait identification of patients with dropped foot for clinical evaluation. This sensor set consists of three IMUs capable of six degrees of freedom, placed onto users thigh, shank and foot of the impaired leg. The study identifies the best sensor orientations and attachment positions, whilst accurately identifying gait events. Finally, it suggests that the system could be used to analyze other walking conditions during daily activities.

Some other shoe-based wireless sensor platforms, such as the SmartStep [34], were used by many different healthcare-related works. In [33], the former platform was used to develop an Android application to capture data from the wearable device and provide real time recognition of a small set of activities. In [48] and [50], the SmartShoe platform is used for energy expenditure estimation after the classification of the activities performed by the user, and in [51] it is further used to predict body weight. The same platform is then used by [27] and [26] to identify activity levels and steps in people with stroke.

Many of the surveyed studies that were conducted in the recognition of activities were related to healthcare and well-being, such as (i) the research presented in [38], that aims at recognizing caregiver’s patient handling activities (PHA) and movement activities to help prevent overexertion injuries, (ii) the work presented in [55], that measures activity in people with stroke, (iii) the work presented in [24], that recognizes activities and postures to provide behavioral feedback to patients recovering from a stroke, and (iv) the research proposed by [44], in which researchers present a pair of shoes that offer low-cost balance monitoring outside of laboratory environments and uses features identified by geriatric motion study experts. This lightweight smart shoes platform is based on the MicroLEAP wireless sensor platform [18], and uses an IMU and FSR pressure sensors embedded inside each insole for data acquisition.

2.3 Prevention and Treatment of Injuries and Diseases

The prevention and treatment of injuries and diseases is the most prevalent theme of research found in this literature review. The research presented in [11] and [12] recognizes that the first step to reduce caregivers risk of overexertion injuries is to identify patient handling activities [PHA]. It proposes an eTextile fabric with 48 plantar pressure sensors and an IMU capable of nine degrees of freedom motion sensing to detect user activity and identify awkward postures that might lead to injuries. Despite the complexities of the interaction between users and loads (e.g., patients and instruments), both studies show promising results.

In works such as [1] and [3], researchers propose an accelerometer-based wearable framework for recognizing athletes activities in outdoor training environments. They aim at identifying (i) potential injury or performance determining factors, (ii) users in the early stages of a developing injury and (iii) a predisposition to injury based on movement patterns. In this work, the researchers performed an experiment to monitor thigh and shank movement and posture during jogging, achieving good overall results. In [44], the researchers present a pair of shoes that offer low-cost balance monitoring non-lab environments. The shoes use features identified by geriatric motion study experts to monitor balance and predict fall risk, demonstrating the feasibility of a model of instability assessment. They are based on a wireless sensor platform using IMU and FSR pressure sensors embedded inside each insole for data acquisition.

3 Building the HAR Classifier

On this Section, we describe the stages followed to develop the HAR classifier: (i) prototyping the wearable IoT Device, (ii) conducting the experiment to acquire use data, (iii) processing the acquired data, (iv) extracting and selecting features, (v) building models and (vi) validating the selected model.

3.1 Wearable IoT Device Prototype

On this section, we present the wearable device prototype used to collect functional training data. To allow for the reproduction of this research, we provide detailed hardware information - types, quantities and models for each component.

Aiming at a user base as diverse as possible, we followed the prototyping principles discussed in [46] and [47]. To reach this goal, we (i) developed a low power consumption easy-to-use wearable IoT device with only a power button and (i) that required only one functional foot to collect and analyse user data. This allowed for extended operation and ease operation during the experiments. The use of only one foot to collect user data was discussed in [46] and does no incur any significant loss of accuracy, reduces prototyping costs and broadens our user base. The wearable device comprises three components: (i) an US men’s size 8.5 insole that houses the plantar pressure sensors, (ii) an external protective case that houses the microcontroller and the foot sensor array and (iii) an external protective case that houses the knee sensor array.

The insole employs six GCF sensors following the literature’s recommendations for placing discussed in [45, 52] and [40], in addition to the lessons learned from the prototypes presented in [46] and [47]. We used the FSR 402, by Interlink Electronics - a PTF (Polymer Thick Film) device that exhibits a decrease in resistance as the force applied to its active surface increases - and Amphenol FCI Clincher Connectors to avoid melting or distorting the silver traces of each sensor. As in [47], to create a variable voltage for the microcontroller’s Analog to Digital Converter (ADC) inputs of each sensor, we embedded six 10 \(\mathrm{k}\Omega \,{\frac{1}{4}}\,\mathrm{W}\) static resistors inside the insole next to them.

The main component of the foot external protective case is the Electron, a 3G-enabled microcontroller from Particle.io that collects data from the foot sensor array and transmits the data to the remote database. For the accelerometer, gyroscope and magnetometer sensors responsible for monitoring the feet posture, we used the SparkFun 9DoF IMU LSM9DS1 Breakout, a system-in-package component that houses a 3-axis accelerometer, 3-axis gyroscope, and 3-axis magnetometer sensor array that is capable of digital communication with the Electron microcontroller. The barometer selected for the experiment is the MPL3115A2, by Freescale Semiconductor, a low power, high-precision altitude and pressure. For the range finder sensor we used two of the RFD77402 3D ToF (Time of Flight) by Simblee, a low-cost accurate sensor that allows millimeter readings up to two meters. The RFD77402 uses an infrared VCSEL (Vertical Cavity Surface Emitting Laser) module to measure the amount of time the emitted light takes to bounce off a target.

The Electron microcontroller and its board, along with the MPL3115A2 and RFD77402 sensors mentioned above, were positioned in the ABS 3D printed external protective case. The prototype is powered by a 2,200 mAh lithium ion battery pack by Sparkfun Electronics, allowing for an easier, faster replacement and improved usability. We did not take any measures to address the sensor drift over time of this prototype, as discussed in [47].

The knee external protective case houses a second SparkFun 9DoF IMU LSM9DS1 Breakout and is connected to the foot external protective case by a flexible cable.

The software model used in this work is based on the model proposed in [46] and [47], comprising two components: (i) the embedded software running on the microcontroller, responsible for acquiring, structuring and transmitting raw sensor data over 3G to the application server, and (ii) the application server itself, responsible for processing and logging the streamed data to Firebase NOSQL database - tasks not suitable for the embedded microcontroller due to its hardware limitations. The authors made available the complete and commented source code of the application server in [46].

3.2 The First Experiment

The first experiment, aimed at building the HAR classifier, was conducted with twelve volunteers carefully selected for their diverse characteristics. One prevailing limitation of the surveyed the works is the employment of homogeneous participants in their experiments. This study tries to circumvent this problem with the participation of people with disabilities and mature adults. Table 1 below summarizes participants information.

Table 1. Experiment 1 participant profiles.
Full size table

Since the insole and shoes are US men’s size 8.5, we selected participants in the 7.5 to 9.0 shoe size range. We collected 18 h of activity data - 90 min of feet and knee posture and movement data from each volunteer. The number of subjects in our study is not that different from the surveyed works others, considering the median of 6 found in the literature. However, the number of samples in our study - over 1.2 million - is significantly higher than the median number of samples - around 50,000 - found on the surveyed works.

The activity model we developed for the experiment comprises 13 activities: walking straight (2 km/h), slow jogging (6 km/h), hopping, ascending stairs, descending stairs, standing, Basic Squat, Sumo Squat, Squat Hold, Basic Step Up, Front Lunge, Side Lunge and sitting. The experiment was conducted in 3 distinct 30-min sessions, where participants performed a set of the planned activities supervised by a certified physical education professional. Following the advice of the physical educator, no participant performed more than 1 session per day and sessions were spaced by at least 48 h. All sessions were performed in a personal training studio.

At each session, participants performed 6 cycles of 5-min routines. In the first session, participants were required to interweave walking, jogging, hopping, ascending stairs, descending stairs, standing and the execution of the Basic Step Up functional exercise. In the second session, participants were required to interweave walking, standing, sitting and the execution of the Basic Squat, Sumo Squat and Squat Hold functional exercises. In the third session, participants were required to interweave walking, standing, sitting and the execution of the Front Lunge and Side Lunge functional exercises. Subjects were free to perform the activities - the wearable prototype did not restrict in any sense their movement. All routines were developed jointly with the certified physical education professional, who provided clear instructions of how the activities were to be performed during the experiment. We focused on the time that each participant should spend performing the proposed activities, rather than the number of repetitions, as the physical conditioning of each participant varied significantly. Table 2 below details the performed activities:

Table 2. Experiment session routines.
Full size table

Since the classifier proposed in [46] performed well, achieving 93.34% overall accuracy, we were confident to extend the original activity model of 6 activities to the current activity model of 13 activities for this experiment. The results showed a drop of 6.26% in the overall accuracy, when compared to the first activity model. We accompanied all participants during the sessions to monitor the wearable IoT device data collection and for logging any unusual occurrence. Hypoallergenic socks were used to avoid skin allergies.

3.3 Data Acquisition

During the data acquisition stage, a stream of unprocessed sensor signals is built from the combination of the insole’s sensors and the knee’s sensors, and it is stored in the microcontroller in JSON format. This raw data combines two accelerometers, two gyroscopes, two magnetometers, six FSR sensors, altitude, pressure and two range finder sensor signals, resulting in 28-feature set entries to the dataset. We used a 20 Hz sampling rate to recognize between similar activities and subtle variations in execution style [31]. The JSON formatted data was periodically sent to the application server in small packages of 200 KB to reduce energy and data usage.

3.4 Data Processing, Feature Extraction and Selection

A data processing pipeline of two steps, similar to the one proposed in [46] and [47], was employed. No experiment data is discarded, given that the prototype starts collecting feet movement and posture information immediately after it is powered. First, the dataset is labelled for supervised learning, and activity class information is appended to each entry according to the activity performed in the experiment. Finally, all sensor data is normalized to make their scales equivalent for the model building.

In the feature extraction stage, we used descriptive statistics - standard deviation, variance, minimum, maximum and average values - to generate derived features from each of the 28 original features:

  • Six FSR sensor readings;

  • Six gyroscope axis data;

  • Six magnetometer axis data;

  • Six accelerometer axis data;

  • Six accelerometer axis data;

  • Altitude reading;

  • Pressure reading; and,

  • Two range finder sensors reading.

Moreover, (i) the cumulative difference between samples for each feature and (ii) the Euler angles of pitch, roll and yaw are also used to generate additional derived features, for a total of over 200 features for selection.

As in [46] and [47], we employed Hall’s algorithm [30] based on correlation for feature selection, using its default “Best Fit” backtracking greedy strategy configuration. In total, 22 features were utilized to build the classifier: 2 axis of the foot gyroscope, 2 axis of the foot magnetometer, 1 axis of the foot accelerometer, 4 FSRs, 2 Euler angles of the foot, 3 axis of the knee gyroscope, 2 axis of the knee magnetometer, 2 axis of the knee accelerometer and the maximum and minimum of the two range finder sensors. Based on the surveyed related works, we already expected to see accelerometer, gyroscope, magnetometer, Euler angles and FSR features showing high correlation and being used in the building of the classifier. However, as in [36, 46] and [47], the range finder sensors were successfully employed and improved the average accuracy of our results. Unlike the classifier proposed in [46] the altimeter was not employed by the classifier.

3.5 Classification and Validation

We experimented different strategies to build the HAR classifier model, as in [46] and [47], and once more the Random Forest Algorithm achieved better results - with an overall classification accuracy of about 87%. To validate the model, we applied the Leave-one-out Cross Validation in attempt to increase the robustness of the model - since this method guarantees that both training and test splits does not share any example data. The individual validation results for each of the 12 examples were: 81.11%, 79.26%, 94.33%, 92.21%, 84.18%, 82.72%, 87.51%, 90.12%, 96.44%, 74.71%, 89.49% and 92.90%.

We experimented several time window sizes to build the classifier and decided to use a 3-s window based on our model validation results. Although [20] recommends window sizes within the 0.25 s–0.50 s range for single activity recognition, our activity model consists of complex single activities, and smaller window sizes did not achieve good overall classification accuracy.

4 Validating the HAR Classifier

On this Section, we describe the stages followed to validate the HAR classifier: performing a second experiment and a semi-structured interview to evaluate usability and user experience.

4.1 The Second Experiment

During the second experiment we employed 9 of the 12 original first experiment volunteers - three of them were not available at the time it was conducted. Table 3 below summarizes participants information.

Table 3. Experiment 1 participant profiles.
Full size table

The goal of the second experiment was to validate the HAR classifier and its capability to assess if the movement pattern of a particular functional training exercise was performed correctly by the user. To achieve that goal, we used the same wearable IoT device employed to build the model with two modifications: (i) we added a SparkFun Haptic Motor Driver with the DRV2605L by Texas Instruments, together with a vibration motor, and (ii) we altered the embedded code to send packages of the collected data to the application server for data processing and waiting for a return code that indicated if the activity was performed correctly. The second experiment was conducted in a 30-min session, where participants performed a set of the planned activities supervised by the physical educator responsible for the first experiment. This session was performed at the same personal training studio used for the first experiment. No data was collected during the second experiment.

At the session, each participant performed 6 cycles of 5-min routines. The routines consisted of all 13 activities of our activity model, and were also developed jointly with the certified physical education professional, who did not provide instructions of how the activities were to be performed during the experiment. A few participants that were performing correctly every activity of the proposed routines were asked to alter the movement pattern to replicate the most common execution errors found during a functional exercise practice. This was done for no more than 3 repetitions for each activity - since the repetitions were performed without extra weights on foam mat floor tiles, the physical educator ensured that no harm would befall those participants. Once more, we focused on the time that each participant should spend performing the proposed activities, rather than the number of repetitions, as the physical conditioning of each participant still varied significantly. Table 4 below details the performed activities:

Table 4. Experiment session routines.
Full size table

During each cycle, participants performed only one of the 6 specific functional exercises of our activity model. We configured the server application to assess only the matching functional exercise each cycle, sending a return code of 0 if the movement pattern was performed correctly and 1 otherwise. A return code of 1 activated the vibration motor, warning the user of the movement pattern performed. After each session, we analyzed the application server log and the physical educator notes - the classifier detected between 6 and 7 incorrect executions out of every 10, per participant.

4.2 The Interview

After the second experiment, we performed a semi-structured interview to evaluate usability and user experience. We interviewed all 9 participants and the physical educator. Below, the questions that were asked:

  • Did the device or the haptic feedback hinder in any way the execution of the activities?

  • Do you perceive the haptic feedback as a help to perform the proposed activities?

  • Do you think that the physical assessment platform proposed is necessary or adequate for your regular physical training program?

The participants did not consider that the device hindered the execution of the activities, since it is lightweight and the flexible cable connecting the knee band and the foot external protective case was well positioned and of adequate length. However, 3 of the 9 respondents reported that they were over overzealous with their movements to prevent any damage to the wearable IoT device, so in their perception the device hindered their focus. Of those 3 respondents, 2 reported that the impaired focus may have been the cause of some execution mistakes. The participants knew that there was only one prototype available and that the research group did not receive any grant to fund the research. The haptic feedback was perceived as a help factor to perform exercises more correctly by 8 of the 9 respondents, although 6 of those respondents pointed out that a human feedback is more helpful since it enables them to understand the reason why an execution was performed incorrectly. All seven physically active participants train without the individual supervision of a physical educator and agreed that (i) a physical assessment platform is necessary and that (ii) the proposed platform, after adjustments necessary for production, is adequate for their training needs. The physical educator did not perceive the device or the haptic feedback to hinder in any way the execution of the proposed activities or the other common functional exercise activities he is used to supervise. The haptic feedback was perceived as a very helpful feature, even considering the real-life accuracy below 70%, since the key factor regarding RSI prevention is the detection of repeated mistakes, not the detection of a single poor execution. In addition to that, (i) a typical functional exercise class equipped with a physical assessment platform could be supervised by only one physical educator, able to make better use of his time helping the users instead of monitoring every aspect of the execution and (ii) some aspects of the execution, such as plantar pressure distribution, are not easily assessed by an observational approach.

5 Conclusion

This work proposes a platform for physical evaluation of functional exercise activities with real-time feedback to help reduce injury risk. We conducted two experiments: (i) the first with 12 volunteers, to build a HAR classifier of a 13-classes activity model based on foot and knee movement and posture information and (ii) the second with 9 volunteers, to validate the physical evaluation model and investigate usability issues of the proposed wearable IoT device. The platform was considered helpful by the experiment participants and the supervising physical educator.

The main contributions are:

  • A comprehensive literature review about wearable-based HAR research using feet and knee movement and posture information for the detection of exercise execution errors;

  • A platform and a model that can be used to assess the quality of execution of different lower limb functional exercises; and,

  • A wearable IoT device blueprint with a comprehensive and novel sensor fusion selection that can be used for lower limb HAR.

Currently, we are evolving the proposed wearable IoT device prototype and broadening the activity model of the HAR classifier to use it in two studies. The first study aims at assessing collective engagement in Physical Education and Sports programs at the high school education level. The second study is an extension to this work and aims at reducing back and shoulder injury risk during functional exercise sessions. Our goal is to progressively provide real time feedback of the whole athlete’s body during the practice of functional exercises.