Sliding Covariance Matrix: Co-learning Spatiotemporal Geometry Feature for Skeleton Based Action Recognition

Abstract

The covariance matrix is a generic feature representation in vision applications. It can accurately and efficiently capture geometric features of Riemannian manifold especially in the condition of data size is medium-scaled. When the covariance matrix is applied in describing skeleton data, how to represent spatial and temporal relations of skeleton joints, meanwhile ensuring the matrix is nonsingular is a challenging problem. In this work, we first propose a sliding window-based frame appending model acquiring a nonsingular covariance matrix descriptor for all skeleton frames. Then, sliding covariance matrixes for all sliding windows are sequentially fed to the modified Long Short-Term Memory (LSTM) network for extracting the spatiotemporal characteristics and action recognition. The proposed method is verified by the experiments on five medium-sized skeleton datasets and the results show that the proposed method improves the accuracy by 6%–20% compared to the state-of-the-art models. Meanwhile, the experiment results clarify that when the data size is not so large, our proposed method can describe spatiotemporal characters of skeleton data more accurately and efficiently than deep network methods.