Keywords

1 Introduction

Neurodegenerative diseases are incurable and afflict humanity. A timely diagnosis could help the physician to slow down the progress of the disease thus improving the quality of the rest of life. To date, the diagnosis of these diseases is long and expensive, and understanding the first symptoms is a complex activity. Neurodegenerative diseases result in a behavioral, cognitive and execution functionalities degradation. Some Computer Aided Diagnosis (CAD) tools have been proposed based on behavioral biometrics and from a pure pattern recognition perspective [1] as for example handwriting [2, 3] by inspecting various aspects of the neuromuscular system [4], but also velocity-based models [5]. A more comprehensive review is treated in [1] and [6]. Motions problems are related to these diseases ant to their severity. Changes in behavioral biometrics can be a prominent biomarker. In fact, human movements are complex activities which involve cognitive, kinesthetic and perceptual-motor components [6]: their evolution can be adopted for diseases evaluation by using several kind of inertial sensors [7] with the aim to perform real-time gait monitoring for different kind of pathologies, such as Parksinson’s disease [8].

Among the other biometrics, a crucial role is played by gait. This activity is fundamental in the life of humans and it is carried out from the first years of life, although it requires the use of many resources of the nervous system to bring it to completion. In fact, neurodegenerative diseases tend to destroy various parts of the nervous system depending on which disease is contracted, bringing, in addition, various disorders manifested while walking. Of particular interest is the application commercially available inertial sensors to the gait analysis [9]. Those sensors are very effective, as shown in [10].

In this work, starting with the existing NDDGD [11, 12] neurodegenerative diseases gait dataset, several classification and regression algorithms have been trained and tested with an inter-patient separation scheme and later integrated within a larger automatic diagnosis system which make use of videos in input or real-time streaming from cameras for predicting the neurodegenerative disease, if present, and its stage. The proposed system is capable of predicting among 3 neurodegenerative diseases and differentiate among the severity (stage) level of the disease, if found. This is an early exploratory analysis on inter-patient separation scheme used for neurodegenerative disease prediction. This separation scheme is very popular on practical computer aided diagnosis system, where the models are trained and tested over disease patterns of different people, and thus enable the application of these systems outside the mere laboratory research, with concrete applications in hospitals and specialized centers.

Thus, this system, is a valuable tool for training physicians and neurologists who need to compare their results with the results obtained by our system for assessing the right disease and its stage.

The use of common cameras for the 2D pose estimation and features engineering is twofold: its cheap compared to depth cameras or 3D tracking wearable sensors, can easily be deployed in every hospital and even in houses and it doesn’t require the patient to wear anything, thus avoiding problems linked to forgetfulness and rejection.

The paper is organized as follows. Section 2 contains a brief description of the treated diseases as well an introduction on gait analysis, Sect. 3 describes the dataset used and related works, Sect. 4 discusses the architecture of the proposed system, Sect. 5 discusses the algorithms used and results. Section 6 contains conclusions and future work.

2 Diseases and Gait Cycle Analysis

Neurodegenerative diseases show long neuronal cells leak that turns in some physical disorders during walking. Some of most known and serious neurodegenerative diseases are Parkinson’s (PD), amyotrophic lateral sclerosis disease (ALS), Alzheimer’s (AD), Huntington Chorea (HUN) and Dementias (DD).

In PD patients, slowness of automatic movements is detected as well as balance. The main physical motor symptoms are rigidity with hypertonia, bradykinesia and, in a deep stage, lack of balance and akinesia [13].

ALS is a disorder with no etiology, and its symptom is an evolutionary muscular atrophy, decrease in strength as well as phonation and chewing disorders [14].

In AD the neuronal lack causes serious damages in terms of short-term memory comprehension capability as well as normal life. Alzheimer’s disease is characterized by uncoordinated movements, erect standing and walking, facial grimaces, dysarthria, dysphagia, alteration of breathing and hyperkinesia accentuated by emotions [15].

DD represent a group of typical, but not exclusive, neurodegenerative diseases of old age, with irreversible loss or reduction of intellectual abilities. However, the disease itself can be found only when clinical evidences appear [16].

The HUN is a hereditary disease in which cognitive and motor skills are particularly compromised. The first clues are mood changes, memory loss, dementia, difficulty in walking, language and swallowing, depression and, in last stages, suicide [16].

In this work, according to [17], the gait cycle is considered to be constituted by the following eight phases:

  • Initial contact (IC): when the foot touches the ground;

  • Loading response (LR): when the other foot is lifted for the swinging;

  • Mid Stance (MS): the swinging foot exceeds the foot that acts as a lever;

  • Terminal stance (TS): the right foot’s heel moves vertically until the left foot touches the ground;

  • Pre-swing (PS): now the left foot acts as a lever allowing the right foot to walk in;

  • Initial Swing: the hip, knee, and ankle are flexed to begin advancement of the limb forward and create clearance of the foot over the ground.

  • Mid-swing (MS): the left leg’s tibia is vertical so that right leg can overcome it;

  • Terminal swing (TS): the progress of the limbs is completed when the right leg moves in front of the left thigh and the right foot touches the ground, going back to the IC phase.

3 Dataset and Related Works

3.1 Dataset

There is not a large amount of publicly available dataset related to gait and neuro-muscular diseases. The most used is, probably, the Gait Dynamics in Neuro-Degenerative Disease Data Base (NDDGD) [11, 12]. It includes 15 patients with Parkinson’s disease, 20 patients with Huntington’s disease, and 13 patients with amyotrophic lateral sclerosis. In addition, 16 healthy control subjects are also included. The raw data were obtained using force-sensitive resistors, with the output roughly proportional to the force under the foot. The dataset contains the features listed in Table 1:

Table 1. Features of NDDGD dataset
Full size table

The dataset also includes clinical information for each subject, including age, gender, height, weight, walking speed, and a measure of disease severity or duration. For the subjects with Parkinson’s disease, this is the Hohn and Yahr score is reported.

Gait recognition has been performed in several ways: with infrared sensors [18] or from b/w images by extracting the silhouette [19], with inertial sensors, UWB sensors and much more.

3.2 Related Works

Zheng et al. [20] investigated three supervised learning algorithms (SVM, Kstar and Random Forest) with a reduced set of features for the aim of ALS, HD and PD classification. Feature selection has been also performed by authors of [21]. In this case Recursive Feature Elimination (RFE) was adopted to select the top 5 features, successively Random Forest and Bagging CART were adopted to obtain, respectively, 96.93% and 97.43% of accuracy.

Ye et al. [22] observed that patients’ gait dynamic is non-linear, so that they proposed an Adaptive Neuro-Fuzzy Inference System (ANFIS) able to combine neural network adaptive capabilities and the fuzzy logic approach. Also in this case a reduced set of features is adopted: left stride interval, right stride interval, left stance interval, right stance interval, and double support interval. Tests are performed discriminating each patient group from the HC within a binary task. Accuracy ranges from 90 to 94%. At the same time, standard classifiers have been adopted observing comparable results to those obtained by authors.

In [23] authors used different feature selection methods gaining a final accuracy of 93% on a similar schema proposed in [22]. In [24] authors used Gaussian radial basis function and SVM to predict Parkinson’s disease patterns from human gait with accuracy of 83.3%.

In [25] authors developed a FPGA which is capable of correctly recognize health and unhealthy patterns from gait analysis and perform classification with accuracy of 93.8%, 89.1%, 94% and 93.3%, respectively for ALS, HD, PD, and healthy person.

Differently from the previously mentioned works, here an inter-patient data scheme is used [26], more specifically gaits of different people are used to predict the health status of others. More specifically, patients for training and testing are chosen randomly, as it will be described later in Sect. 5, but each patient and his/her signals will be exclusively used in training or testing, not both.

4 System Architecture

4.1 Classification and Regression Models

In the proposed schema there are 4 different models: the first is a multi-class classification model which predicts if the instance to evaluate belongs to ALS, HUN, PD or is a HC. As it will be shown later, this model predicts probabilities for an instance to fall into one of the four classes. Than for each class, except for HC, a regression model is used.

The regression model was previously trained separately on its subset of instances belonging to the same class, separated in training set and test set respecting the inter-patient scheme previously described. It is used to predict the severity/duration of the disease.

The multi-class classification model outputs a probability of the instance belonging to a specific class. If the class is a disease, the system specifies the top two disease (ranking probabilities), otherwise the system just outputs that the instance belongs to a healthy person. If the classified instance is evaluated to belong to a disease class, the system uses the previously trained regression model for that disease and use it to predict the severity or duration (in months) of the disease. At the end the system outputs the top two classes found and their severity/duration if applicable.

4.2 Video Real-Time Classification and Severity Prediction

The 4 models developed at previous step are then integrated into a bigger system that allows for real-time video neuro-degenerative disease classification and severity prediction. The following steps are performed [27]:

  • Frame extraction: video input is acquired from the recording device in real-time and frames are extracted for further processing.

  • Skeleton calculation: a skeleton extraction process is performed, i.e. the individual’s “skeleton” is obtained for the acquisition of the most important points, such as the hips, knees, feet, shoulders and head. The angle of the head is then obtained to differentiate the right side from the left.

  • Gait feature extraction: the data previously extracted from the image are analyzed to extract the gait features.

  • Gait classification and severity prediction: the results of the previous phase are inserted in the classifier that will return the class to which the analyzed data belongs. The classifier will be able to distinguish a healthy gait from a patient and give information on the disease that has been identified in that dataset.

Skeleton Calculation with Pose Estimation

OpenPose [28] has been used for pose estimation. The algorithm here used was designed to find the key points of a person (shoulders, head, arms, hips, knees and ankles) in 2D images [27]. The algorithm learns jointly, both the positions of the parts of the body and their association through two branches of the same forecasting process. The model was trained on over 25,000 images of the MPII database [29] (multi person database).

The model receives a color image of any size as input and produces, as output, a list with the 2D coordinates in terms of pixels of the key points of the person in the image.

Gait Feature Extraction

The coordinates and time stamp within each frame, for the right and left ankle, are used to estimate all the features present in the NDGGD Database and shown in Table 1.

Gait feature extraction is performed in two steps: elimination of position errors, identification and feature engineering. Since the analyzed patients, with a neurodegenerative disease, all showed a shuffling walk, that is the behavior to crawl feet on the ground without lifting them, only the x-coordinate have been considered.

The pose estimation process introduces some errors. For example, in the stance phase, in which the foot is stopped on the ground, the x-axis data of the shin should return with a series of identical values. However, there may be variations in the series that are due to the swing phase (the one in which the foot is moving). Moreover, the right ankle is frequently mixed with the left ankle. To remove these errors, a threshold time was set at 0.15 s to consider a stance phase (roughly 4 consecutive frames). Values affected by the described errors are forced to those of the series to which they are considered to belong. Technically a forward fill procedure was applied.

In this way, stances phases are identified as a series of identical values. Successively, swing phases are those between two phase of stance.

The phases of double support are the phases in which both steps are in stance and finally the stride is the sum of a stance and the next swing. Time duration of the aforementioned phases are calculated by using the timestamp of each frame. The result of this process is a new unlabeled instance with same features of NDDGD database as in Table 1.

All features have been normalized with Z-Score (using standard deviation and mean computed on training set) and fed into the classification engine for disease prediction and its severity/duration.

5 Performance Comparison

The training/testing splitting process of the dataset has been realized according to a probability value Pr taken from a uniform distribution D at random. If such value Pr < 0.35, the entire file corresponding to all the instances computed from that particular person, are entered in the test set, otherwise they enter in the training set. The same training set/test set split is than used for training the regression models of each disease class.

The dataset is almost balanced if each disease class is compared against the control class, but in case of multi-class classification, a class balancing is required. For class balancing, LICIC [30] has been used with linear kernel and a components ratio of 0.6.

The data is normalized with z-score and all tests are executed with 10-fold cross validation technique.

Table 2 reports prediction accuracies. As can be easily observed, multi-class classification on the inter-patient scheme on this dataset, is not a trivial problem.

Table 2. Multi-class classification accuracies with LICIC balancing
Full size table

All models’ parameters have been selected with grid search technique and the parameters with better cross validation accuracies have been selected.

All the algorithms used have accuracies < 0.5. Figure 1 shows that the classification algorithm makes errors in all the classes, which means that, for the inter-patient scheme, the data distribution over classes in the training set is not representative of the real data distribution. This result is related to the fact that time features are not very representative of the particular walk pattern of a certain disease, since almost all diseases share similar time features patterns.

Fig. 1.
figure 1

Confusion matrix for Neural Network with LICIC

Full size image

Successively a binary classification has been considered by testing each disease vs the control class. In this case, Neural Networks with two hidden layers, with respectively 8 and 4 neurons, has been used.

Table 3 shows the various accuracies obtained when performing binary classification. In each classification task, the training set has been balanced with LICIC. The results show that the system achieves almost 80% of accuracy when predicting ALS vs healthy control people, follows Parkinson VS healthy control people with about 71% of accuracy and finally Huntington vs healthy control people with 69% of accuracy.

Table 3. Accuracies for Neural Network over different tasks
Full size table

Table 4 shows the RMSE errors when performing the prediction of severity/duration.

Table 4. RMSE errors with best performing algorithm with respect to disease class.
Full size table

As it is possible to note, an accurate prediction of the duration (in months) for the ALS produces an error of about 15 months. Parkinson severity is quite appreciable, HUN severity, being a sum, has an appreciable deviation of 4.26 HUN score. Values are not extremely accurate, but, keeping into consideration the deviations, will help the physician with the diagnosis.

To recapitulate, feed forward neural network with two hidden layers with 8 and 4 neurons per hidden layer, is the architecture that provides the best generalization accuracy when performing binary classification as shown in Table 3. Instead for the severity/duration prediction, each task has its own best model: for Parkinson severity prediction, a feedforward neural network with one single hidden layer and 8 neurons achieved the lowest RMSE error. For ALS duration and Huntington disease, the random forest regressor with 10 trees and a max depth of 4 is the model who achieved the lowest RMSE in both tasks.

Real-Time Video Disease Classification

Because of the limitations in multi-class classification, during the real-time test phase with video cameras, it has been decided to show the first two most confident classified diseases with their respective severity/duration prediction. The multi-class classification algorithm selected, the Neural Network, instead of outputting a single class, it outputs the probabilities that the testing instance belongs to each class (PD, HUN, ALS; CO), the first two classes with highest ratio (sorted in descending order) are shown and for each class, the severity/duration value is predicted and showed.

The multi-class classification keeping the first two most probable classes showed a cross validation accuracy of 0.9620 and an overall accuracy of 0.7637 by using the same Neural Network as specified previously, as the same inter-patient scheme. The accuracies over all classes can be seen in the confusion matrix represented in Fig. 2.

Fig. 2.
figure 2

Confusion matrix for Neural Network with LICIC and two most probable classes

Full size image

By using the limited amount of videos available on internet showing Parkinsonian patients and Alzheimer’s patients gait, the system was capable of correctly recognizing the right disease, within the top two most probable classes, in the majority of gaits.

6 Conclusions and Future Work

A real-time system for neurodegenerative diseases classification and severity/duration prediction has been here presented. When using the inter-patient dataset separation scheme, suitable for medical purposes, features present in NDDGD dataset are not representative of the particular pattern able to accurately discriminate a specific neurodegenerative disease from others in a multi-class classification scenario. It has been observed that almost all neurodegenerative diseases exhibit similar temporal features patterns. For this reason, class belonging probabilities have been computed to show the two most probable classes, with their respective severity/duration maturity (and a confidence level: the RMSE). In a future work, other features as well as stability medals [31] and zoning techniques [32] will be evaluated in an inter-patient separation scheme.