Semantic Structures for Video Data Indexing

Abstract

Video indexing based on contents annotations can fully explore semantic information of video data. However, the most difficult and time-consuming process in annotation-based indexing is to identify appropriate video intervals for various semantic contents manually. Thus, automatic discovering video intervals from video data will be helpful for the indexing work. For this purpose, we propose “semantic structures” of video data and a mechanism for discovering semantic structures. The basic concept of our approach is to (1) discover consecutive sequences of shots from video data, each of which represents a consistent action or situation, and (2) index each of the discovered video intervals based on its semantics. A semantic structure is a collection of discovered video intervals that are classified into three categories: “unchanged” (i.e. actors or backgrounds are unchanged throughout the interval), “gradually changing” (i.e. actors or backgrounds are changing shot by shot) and “multiplexing” (i.e. individual actors or backgrounds are appearing by turns). The mechanism discovers these types of video intervals by comparing and contrasting similarity between each shot, and indexes each of discovered intervals by using indexing algorithms prepared for each type. We show how well our approach works for identifying video intervals with some experimental results.