Tensor Decomposition and Non-linear Manifold Modeling for 3D Head Pose Estimation

Abstract

Head pose estimation is a challenging computer vision problem with important applications in different scenarios such as human–computer interaction or face recognition. In this paper, we present a 3D head pose estimation algorithm based on non-linear manifold learning. A key feature of the proposed approach is that it allows modeling the underlying 3D manifold that results from the combination of rotation angles. To do so, we use tensor decomposition to generate separate subspaces for each variation factor and show that each of them has a clear structure that can be modeled with cosine functions from a unique shared parameter per angle. Such representation provides a deep understanding of data behavior. We show that the proposed framework can be applied to a wide variety of input features and can be used for different purposes. Firstly, we test our system on a publicly available database, which consists of 2D images and we show that the cosine functions can be used to synthesize rotated versions from an object from which we see only a 2D image at a specific angle. Further, we perform 3D head pose estimation experiments using other two types of features: automatic landmarks and histogram-based 3D descriptors. We evaluate our approach on two publicly available databases, and demonstrate that angle estimations can be performed by optimizing the combination of these cosine functions to achieve state-of-the-art performance.

Appendix

For the objective function of Eq. 15, partial derivatives should be computed with respect to variables $\alpha _j^{(*)}, \beta _j^{(*)}, \gamma _j^{(*)}, \varphi _j^{(*)}$. Let us rewrite this equation as:

$$\begin{aligned}&\mathop {\mathrm{argmin}}\limits _{\alpha _j^{(*)}, \beta _j^{(*)}, \gamma _j^{(*)}, \varphi _j^{(*)}}{ \Vert u^{(*)}_{ij} - f_j( \omega _i^{(*)} ) \Vert }\\&= \mathop {\mathrm{argmin}}\limits _{\alpha _j^{(*)}, \beta _j^{(*)}, \gamma _j^{(*)}, \varphi _j^{(*)}} {E(\alpha _j^{(*)}, \beta _j^{(*)}, \gamma _j^{(*)}, \varphi _j^{(*)})} \end{aligned}$$

Where error function E can be written in element form as:

$$\begin{aligned}&E(\alpha _j^{(*)}, \beta _j^{(*)}, \gamma _j^{(*)}, \varphi _j^{(*)})\\&= \frac{1}{2} \sum _i{\big ( u^{(*)}_{ij} - ( \alpha _j^{(*)} \, \cos {( \beta _j^{(*)} \omega _i^{(*)} + \gamma _j^{(*)})} + \varphi _j^{(*)}) \big )}^2 \end{aligned}$$

Thus, partial derivatives of the function E become:

$$\begin{aligned} \frac{\partial {E}}{\partial {\alpha _j^{(*)}}}= & {} \sum _i \big ( \cos {( \beta _j^{(*)} \omega _i^{(*)} + \gamma _j^{(*)})}\\&\cdot (\alpha _j^{(*)}\cos {( \beta _j^{(*)} \omega _i^{(*)} + \gamma _j^{(*)})} - u^{(*)}_{ij} + \varphi _j^{(*)})\big )\\ \frac{\partial {E}}{\partial {\beta _j^{(*)}}}= & {} \alpha _j^{(*)}\sum _i \big ( \omega _i^{(*)} \sin {(\beta _j^{(*)}\omega _i^{(*)}+\gamma _j^{(*)})}\\&\cdot (u^{(*)}_{ij} - \alpha _j^{(*)} \cos {(\beta _j^{(*)}\omega _i^{(*)} + \gamma _j^{(*)}) - \varphi _j^{(*)}})\big )\\ \frac{\partial {E}}{\partial {\gamma _j^{(*)}}}= & {} \alpha _j^{(*)}\sum _i \big ( \sin {(\beta _j^{(*)}\omega _i^{(*)}+\gamma _j^{(*)})}\\&\cdot (u^{(*)}_{ij} - \alpha _j^{(*)}\cos {(\beta _j^{(*)}\omega _i^{(*)}+\gamma _j^{(*)})} -\varphi _j^{(*)})\big )\\ \frac{\partial {E}}{\partial {\varphi _j^{(*)}}}= & {} \sum _i \big ( \alpha _j^{(*)}\cdot \cos {(\beta _j^{(*)}\omega _i^{(*)}+\gamma _j^{(*)})}\\&+ \varphi _j^{(*)} + u^{(*)}_{ij} \big ) \end{aligned}$$

Similarly to the previous case, the error function in Eq. 12 can be written as:

$$\begin{aligned} E(\omega ^{(y)}, \omega ^{(p)}, \omega ^{(r)}, \mathbf u ^{(id)}) = \frac{1}{2} \sum _n{\big ( \mathbf x _n - \hat{\mathbf{x }}_n \big )}^2 \end{aligned}$$

where $\mathbf x _n$ using Eq. 14 is:

$$\begin{aligned} \mathbf x _n= & {} \sum _i \sum _j \sum _k \sum _l\big ( \mathscr {W}_{ijkln}\cdot f_i^{(y)}( \omega ^{(y)} ) \cdot f_j^{(p)}( \omega ^{(p)} )\\&\cdot f_k^{(r)}( \omega ^{(r)} ) \cdot {u}^{(id)}_l \big ) \end{aligned}$$

Now we can compute partial derivatives with respect to variables $\omega ^{(y)}, \omega ^{(p)}, \omega ^{(r)}$ and each l-th element in vector $\mathbf u ^{(id)}$:

$$\begin{aligned} \frac{\partial {E}}{\partial {\omega ^{(y)}}}= & {} -\sum _n \big ( (\mathbf x _n - \hat{\mathbf{x }}_n ) \cdot \frac{\partial {\hat{\mathbf{x }}_n}}{\partial {\omega ^{(y)}}} \big );\\ \frac{\partial {\hat{\mathbf{x }}_n}}{\partial {\omega ^{(y)}}}= & {} -\sum _i \sum _j \sum _k \sum _l\big ( \mathscr {W}_{ijkln}\cdot \alpha ^{(y)}_i \sin (\beta ^{(y)}_i\omega ^{(y)}+\gamma ^{(y)}_i)\beta ^{(y)}_i\\&\cdot f_j^{(p)}( \omega ^{(p)} ) \cdot f_k^{(r)}( \omega ^{(r)} )\cdot {u}^{(id)}_l \big ) ;\\ \frac{\partial {E}}{\partial {\omega ^{(p)}}}= & {} -\sum _n \big ( (\mathbf x _n - \hat{\mathbf{x }}_n ) \cdot \frac{\partial {\hat{\mathbf{x }}_n}}{\partial {\omega ^{(p)}}} \big );\\ \frac{\partial {\hat{\mathbf{x }}_n}}{\partial {\omega ^{(p)}}}= & {} -\sum _i \sum _j \sum _k \sum _l\big ( \mathscr {W}_{ijkln}\cdot f_i^{(y)}( \omega ^{(y)} )\\&\cdot (\alpha ^{(p)}_j \sin (\beta ^{(p)}_j\omega ^{(p)}+\gamma ^{(p)}_j)\beta ^{(p)}_j)\cdot f_k^{(r)}( \omega ^{(r)} )\cdot {u}^{(id)}_l \big );\\ \frac{\partial {E}}{\partial {\omega ^{(r)}}}= & {} -\sum _n \big ( (\mathbf x _n - \hat{\mathbf{x }}_n ) \cdot \frac{\partial {\hat{\mathbf{x }}_n}}{\partial {\omega ^{(r)}}} \big );\\ \frac{\partial {\hat{\mathbf{x }}_n}}{\partial {\omega ^{(r)}}}= & {} -\sum _i \sum _j \sum _k \sum _l\big ( \mathscr {W}_{ijkln}\cdot f_i^{(y)}( \omega ^{(y)} )\\&\cdot f_j^{(p)}( \omega ^{(p)} ) \cdot (\alpha ^{(r)}_k \sin (\beta ^{(r)}_k\omega ^{(r)}+\gamma ^{(r)}_k)\beta ^{(r)}_k) \cdot {u}^{(id)}_l \big );\\ \frac{\partial {E}}{\partial \mathbf{u ^{(id)}_l}}= & {} -\sum _n \big ( (\mathbf x _n - \hat{\mathbf{x }}_n ) \cdot \frac{\partial {\hat{\mathbf{x }}_n}}{\partial \mathbf{u ^{(id)}_l}} \big ); \\ \frac{\partial {\hat{\mathbf{x }}_n}}{\partial \mathbf{u ^{(id)}_l}}= & {} \sum _i \sum _j \sum _k \big ( \mathscr {W}_{ijkln}\cdot f_i^{(y)}( \omega ^{(y)} )\cdot f_j^{(p)}( \omega ^{(p)} ) \cdot f_k^{(r)}( \omega ^{(r)} )\big ) \end{aligned}$$