One-Shot-Learning Gesture Recognition Using HOG-HOF Features

Abstract

The purpose of this paper is to describe one-shot-learning gesture recognition systems developed on the ChaLearn Gesture Dataset (ChaLearn 2011). We use RGB and depth images and combine appearance (Histograms of Oriented Gradients) and motion descriptors (Histogram of Optical Flow) for parallel temporal segmentation and recognition. The Quadratic-Chi distance family is used to measure differences between histograms to capture cross-bin relationships. We also propose a new algorithm for trimming videos—to remove all the unimportant frames from videos. We present two methods that use a combination of HOG-HOF descriptors together with variants of a Dynamic Time Warping technique. Both methods outperform other published methods and help narrow the gap between human performance and algorithms on this task. The code is publicly available in the MLOSS repository.

Editors: Isabelle Guyon, Vassilis Athitsos, and Sergio Escalera.

Appendices

Appendix A

In this appendix, we analyse the computational complexity of our methods.

Let us first describe the computational complexity of the building blocks of our algorithms. Let r, c be the resolution of our videos. For this data set we have $r = 240, c = 320$. Let P denote number of pixels ($P = rc$). Computing both HOG and HOF features requires performing a fixed number of iterations for every pixel. Creating histograms in spatial cells requires a fixed number of operations with respect to the size of these cells. Thus the complexity of computing HOG and HOF descriptors for one example requires $\mathcal {O}(P)$ operations. Let m be the number of pixels used in the median filter for every pixel. Since computing the median requires ordering, the complexity of filtering an image requires $\mathcal {O}(P m \log m)$ operations. In total, for both SM and MM, the whole training on a batch of N frames in total requires $\mathcal {O}(N P m \log (m))$ operations.

Before evaluating a new video of F frames, we have to compute the representations of the frames, which is done in $\mathcal {O}(F P m \log m)$ operations. In both methods we then perform a Viterbi search. In MM this is divided into several searches, but the total complexity stays the same. The most time consuming part is computing the Quadratic-Chi distances (Sect. 12.5.3) between all FN pairs of frames from the new video and model. Computing the distance needs sum over elements over sparse $H \times H$ matrix (H being the size of the histograms used) described in Algorithm 2. The number of non-zero elements is linear in H. Thus, the overall complexity of evaluating a new video is $\mathcal {O} (N P m \log (m) + N F H).$

To summarize, the running time of our methods is linear in the number of training frames, number of frames of a new video, number of pixels of a single frame, and size of histogram (number of spatial cell times number of orientation bins). Dependence on size of the filtering region for every pixel is linearithmic since it requires sorting.

Appendix B

In this Appendix, we provide MATLAB algorithm for creating similarity matrix used in the Quadratic-Chi distance described in Sect. 12.5.3. We have histograms of $h \times w$ spatial cells, and p orientation bins in each of the spatial bins. The size of the final matrix is $H \times H$, where $H = hwp$.