Some Approaches to Recognition of Sign Language Dynamic Expressions with Kinect

Abstract

The paper considers recognition of isolated Polish Sign Language words observed by Kinect. A whole word model approach with nearest neighbour classifier applying dynamic time warping (DTW) technique is compared with an approach using models of subunits, i.e. some elements smaller than words, resembling phonemes in spoken expressions. Such smaller models are obtained using data-driven procedure involving division of time series representing words from a training set into subsequences which form homogeneous groups. Symbols are assigned to these groups and then gestures receive symbolic representations (transcriptions). Such transcriptions are classified using nearest neighbour approach based on edit distance. Two sets of features have been used: one based on Kinect’s skeletal images and the other using simplified description of hands extracted as skin coloured regions. Ten-fold cross-validation tests of classifiers using data representing signed Polish words were performed. Subunit based approach proved to be superior, in particular in the case when only one learning example was available. Features taking into account description of hands led to better recognition results in comparison with those based on skeleton.