Multimodal video classification with stacked contractive autoencoders

Highlights

• A two-stage framework for multimodal video classification is proposed.

• The model is built based on stacked contractive autoencoders.

• The first stage is single modal pre-training.

• The second stage is multimodal fine-tuning.

• The objective functions are optimized by stochastic gradient descent.

Abstract

In this paper we propose a multimodal feature learning mechanism based on deep networks (i.e., stacked contractive autoencoders) for video classification. Considering the three modalities in video, i.e., image, audio and text, we first build one Stacked Contractive Autoencoder (SCAE) for each single modality, whose outputs will be joint together and fed into another Multimodal Stacked Contractive Autoencoder (MSCAE). The first stage preserves intra-modality semantic relations and the second stage discovers inter-modality semantic correlations. Experiments on real world dataset demonstrate that the proposed approach achieves better performance compared with the state-of-the-art methods.

Introduction

With rapid progress of storage devices, Internet and social network, a large amount of video data are generated. How to index and search for these videos effectively is an increasingly active research issue in the multimedia community. To bridge the semantic gap between low-level features and high-level semantics, automatic video annotation and classification have emerged as important techniques for efficient video retrieval [1], [2], [3].

Typical approaches to accomplishing video annotation and classification are to apply machine learning methods only using image features from keyframes of video clips. As a matter of fact, video consists of three modalities, namely image, audio and text. Image features of keyframes just express visual aspects, whereas auditory and textual features are equivalently significant for video semantics understanding. A great deal of research has been focused on utilizing multimodal features for better understanding of video semantics [4], [5]. Thus multimodal integration in video may compensate the limitations of learning from any single modality.

There are also many other multimodal learning strategies. One group focuses on multi-modal or cross-modal retrieval that learn to map the high-dimensional heterogeneous features into a common low-dimensional latent space [6], [7], [8], [9]. Another group is composed of graph-based models, which generate geometric descriptors from multi-channel or multi-sensor to improve image or video analysis [10], [11], [12], [13], [14], [15], [16]. However, these methods are discriminative by supervised setting, which require a large amount of labeled data and waste abundant unlabeled data. Collecting labeled data is time-consuming and labor intensive. Thus discovering good representations of data that make it easier to extract useful information when building classifiers with only unsupervised learning has become a big challenge.

Recently, deep learning methods have tremendously attracted researchers interests. The breakthrough in deep learning was initiated by Hinton and quickly followed up in the same year [17], [18], [19], and many more later. A central idea, referred to as greedy layerwise unsupervised pre-training, was to learn a hierarchy of features one level at a time, using unsupervised feature learning to learn a new transformation at each level to be composed with the previously learned transformations; essentially, each iteration of unsupervised feature learning adds one layer of weights to a deep neural network. Finally, the set of layers could be combined to initialize a deep supervised predictor [20]. Methods that have been considered include Deep Belief Networks (DBNs) [17] with Restricted Boltzmann Machine (RBM), autoencoders [18], Convolutional Neural Network (CNN)[19], and so on.

Deep learning has been successfully applied to unsupervised feature learning not only for single modality, but also for multiple modalities [21], [22], [23], [24]. However, these approaches just learn deep networks with two modalities, e.g., image-text or audio-image pairs. This paper explores useful feature representation by fusing three modalities of video into a joint representation that reflects the intrinsic semantics that the video data corresponds to. Specifically, we first build one Stacked Contractive Autoencoders (SCAE) for each single modality, whose outputs will be joint together and fed into another Multimodal Stacked Contractive Autoencoders (MSCAE). The Contractive Autoencoder (CAE) is a representation-learning algorithm which captures local manifold structure and has the potential for non-local generalization [25]. It is much more appropriate for multimedia data, such as video and image, which are intrinsically on low-dimensional manifolds. The proposed method has two stages. The first stage preserves intra-modality semantic relations and the following stage discovers inter-modality semantic correlations. Compared with existing supervised learners, our method requires minimum amount of prior knowledge of the training data.

Moreover, the power of deep architecture used in the proposed algorithm has its own advantages than other shallow methods for video semantic analysis. For example, [26] used one hidden layer to learn an intermediate representation from video features. However, the single layer learning is limited, while deep network is able to discover more abstract and higher level semantic features for multimedia data. In addition, authors [27], [28] have studied the fusion and adaption multiple features for Multimedia Event Detection (MED). But their work mostly focuses on various image features, ignoring the other modalities in video.

The remainder of this paper is organized as follows. We first provide some background to build our model in Section 2. The proposed framework is introduced in Section 3. Section 4 reports the experimental analysis. Finally, we summarize the conclusion and future work in Section 5.