Recent developments in human motion analysis

Abstract

Visual analysis of human motion is currently one of the most active research topics in computer vision. This strong interest is driven by a wide spectrum of promising applications in many areas such as virtual reality, smart surveillance, perceptual interface, etc. Human motion analysis concerns the detection, tracking and recognition of people, and more generally, the understanding of human behaviors, from image sequences involving humans. This paper provides a comprehensive survey of research on computer-vision-based human motion analysis. The emphasis is on three major issues involved in a general human motion analysis system, namely human detection, tracking and activity understanding. Various methods for each issue are discussed in order to examine the state of the art. Finally, some research challenges and future directions are discussed.

Introduction

As one of the most active research areas in computer vision, visual analysis of human motion attempts to detect, track and identify people, and more generally, to interpret human behaviors, from image sequences involving humans. Human motion analysis has attracted great interests from computer vision researchers due to its promising applications in many areas such as visual surveillance, perceptual user interface, content-based image storage and retrieval, video conferencing, athletic performance analysis, virtual reality, etc.

Human motion analysis has been investigated under several large research projects worldwide. For example, Defense Advanced Research Projects Agency (DARPA) funded a multi-institution project on Video Surveillance and Monitoring (VSAM) [1], whose purpose was to develop an automatic video understanding technology that enabled a single human operator to monitor activities over complex areas such as battlefields and civilian scenes. The real-time visual surveillance system W⁴ [2] employed a combination of shape analysis and tracking, and constructed the models of people's appearances to make itself capable of detecting and tracking multiple people as well as monitoring their activities even in the presence of occlusions in an outdoor environment. Researchers in the UK have also done much research on the tracking of vehicles and people and the recognition of their interactions [3]. In addition, companies such as IBM and Microsoft are also investing on research on human motion analysis [4], [5].

In recent years, human motion analysis has been featured in a number of leading international journals such as International Journal of Computer Vision (IJCV), Computer Vision and Image Understanding (CVIU), IEEE Transactions on Pattern Recognition and Machine Intelligence (PAMI), and Image and Vision Computing (IVC), as well as prestigious international conferences and workshops such as International Conference on Computer Vision (ICCV), IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), European Conference on Computer Vision (ECCV), Workshop on Applications of Computer Vision (WACV), and IEEE International Workshop on Visual Surveillance (IWVS).

All the above activities have demonstrated a great and growing interest in human motion analysis from the pattern recognition and computer vision community. The primary purpose of this paper is thus to review the recent developments in this exciting research area, especially the progress since previous such reviews.

Human motion analysis has a wide range of potential applications such as smart surveillance, advanced user interface, motion-based diagnosis, to name a few [6].

The strong need of smart surveillance systems [7], [8] stems from those security-sensitive areas such as banks, department stores, parking lots, and borders. Surveillance cameras are already prevalent in commercial establishments, while camera outputs are usually recorded in tapes or stored in video archives. These video data is currently used only “after the fact” as a forensic tool, losing its primary benefit as an active real-time media. What is needed is the real-time analysis of surveillance data to alert security officers to a burglary in progress, or to a suspicious individual wandering around in the parking lot. Nowadays, the tracking and recognition techniques of face [9], [10], [11], [12] and gait [13], [14], [15], [16] have been strongly motivated for the purpose of access control. As well as the obvious security applications, smart surveillance has also been proposed to measure traffic flow, monitor pedestrian congestion in public spaces [17], [18], compile consumer demographics in shopping malls, etc.

Another important application domain is advanced user interfaces in which human motion analysis is usually used to provide control and command. Generally speaking, communication among people is mainly realized by speech. Therefore, speech understanding has already been widely used in early human–machine interfaces. However, it is subject to the restrictions from environmental noise and distance. Vision is very useful to complement speech recognition and natural language understanding for more natural and intelligent communication between human and machines. That is to say, more detailed cues can be obtained by gestures, body poses, facial expressions, etc. [19], [20], [21], [22]. Hence, future machines must be able to independently sense the surrounding environment, e.g., detecting human presence and interpreting human behavior. Other applications in the user interface domain include sign-language translation, gesture driven controls, and signaling in high-noise environment such as factories and airports [23].

It is particularly useful to segment various body parts of human in an image, track the movement of joints over an image sequence, and recover the underlying 3-D body structure for the analysis and training of athletic performance. With the development of digital libraries, interpreting video sequences automatically using content-based indexing will save tremendous human efforts in sorting and retrieving images or video in a huge database. Traditional gait analysis [24], [25], [26] aims at providing medical diagnosis and treatment support, while human gait can also be used as a new biometric feature for personal identification [13], [14], [15], [16]. Some other applications of vision-based motion analysis lie in personalized training systems for various sports, medical diagnostics of orthopedic patients, choreography of dance and ballet, etc.

In addition, human motion analysis shows its importance in other related areas. For instance, typical applications in virtual reality include chat-rooms, games, virtual studios, character animations, teleconferencing, etc. As far as computer games [27] are concerned, they have been very prevalent in entertainment. Maybe people are surprised at the realism of virtual humans and simulated actions in computer games. In fact, this benefits greatly from computer graphics dealing with devising realistic models of human bodies and the synthesis of human movement based on knowledge of the acquisition of human body model, the retrieval of body pose, human behavior analysis, etc. Also, it is obvious that model-based image coding (e.g., only encoding the pose of the tracked face in images in more detail than the uninterested background in a videophone setting) will bring about very low bit-rate video compression for more effective image storage and transmission.

The importance and popularity of human motion analysis has led to several previous surveys. Each such survey is discussed in the following in order to put the current review in context.

The earliest relevant review was probably due to Aggarwal et al. [28]. It covered various methods used in articulated and elastic non-rigid motion prior to 1994. As for articulated motion, the approaches with or without a prior shape models were described.

Cedars and Shah [29] presented an overview of methods for motion extraction prior to 1995, in which human motion analysis was illustrated as action recognition, recognition of body parts and body configuration estimation.

Aggarwal and Cai gave another survey of human motion analysis [30], which covered the work prior to 1997. Their latest review [31] covering 69 publications was an extension of their workshop paper [30]. The paper provided an overview of various tasks involved in motion analysis of human body prior to 1998. The focuses were on three major areas related to interpreting human motion: (a) motion analysis involving human body parts, (b) tracking moving human from a single view or multiple camera perspectives, and (c) recognizing human activities from image sequences.

A similar survey by Gavrila [6] described the work in human motion analysis prior to 1998. Its emphasis was on discussing various methodologies that were grouped into 2-D approaches with or without explicit shape models and 3-D approaches. It concluded with two main future directions in 3-D tracking and action recognition.

Recently, a relevant study by Pentland [32] centered on person identification, surveillance/monitoring, 3-D methods, and smart rooms/perceptual user interfaces to review the state of the art of “looking at people”. The paper was not intended to survey the current work on human motion analysis, but touched on several interesting topics in human motion analysis and its applications.

The latest survey of computer-vision-based human motion capture was presented by Moeslund and Granum [33]. Its focus was on a general overview based on the taxonomy of system functionalities, viz., initialization, tracking, pose estimation and recognition. It covered the achievements from 1980 into the first half of 2000. In addition, a number of general assumptions used in this research field were identified and suggestions for future research directions were offered.

The growing interest in human motion analysis has led to significant progress in recent years, especially on high-level vision issues such as human activity and behavior understanding. This paper will provide a comprehensive survey of work on human motion analysis from 1989 onwards. Approximately 70% of the references discussed in this paper are found after 1996. In contrast to the previous reviews, the current review focuses on the most recent developments, especially on intermediate-level and high-level vision issues.

To discuss the topic more conveniently, various surveys usually select different taxonomies to group individual papers depending on their purposes. Unlike previous reviews, we will focus on a more general overview on the overall process of a human motion analysis system shown in Fig. 1. Three major tasks in the process of human motion analysis (namely human detection, human tracking and human behavior understanding) will be of particular concern. Although they do have some overlap (e.g., the use of motion detection during tracking), this general classification provides a good framework for discussion throughout this survey.

The majority of past works in human motion analysis are accomplished within tracking and action recognition. Similar in principle to earlier reviews, we will make more detailed introductions to both processes. We also introduce relevant reviews on motion segmentation used in human detection, and behavior semantic description used in human activity interpretation. Compared with previous reviews, we include more comprehensive discussions on research challenges and future open directions in the domain of vision-based human motion analysis.

Instead of detailed summaries of individual publications, our emphasis is on discussing various methods for different tasks involved in a general human motion analysis system. Each issue will be accordingly divided into sub-processes or categories of various methods to examine the state of the art, and only the principles of each group of methods are described in this paper.

Unlike previous reviews, this paper is clearly organized in a hierarchical manner from low-level vision, intermediate-level vision to high-level vision according to the general framework of human motion analysis. This, we believe, will help the readers, especially newcomers to this area, not only to obtain an understanding of the state of the art in human motion analysis but also to appreciate the major components of a general human motion analysis system and the inter-component links.

In summary, the primary purpose and contributions of this paper are as follows (when compared with the existing survey papers on human motion analysis):

(1) This paper aims to provide a comprehensive survey of the most recent developments in vision-based human motion analysis. It covers the latest research ranging mainly from 1997 to 2001. It thus contains many new references not found in previous surveys.

(2) Unlike previous reviews, this paper is organized in a hierarchical manner (from low-level vision, intermediate-level vision, to high-level vision) according to a general framework of human motion analysis systems.

(3) Unlike other reviews, this paper selects a taxonomy based on functionalities including detection, tracking and behavior understanding within human motion analysis systems.

(4) This paper focuses more on overall methods and general characteristics involved in the above three issues (functionalities), so each issue is accordingly divided into sub-processes and categories of approaches so as to provide more detailed discussions.

(5) In contrast to past surveys, we provide detailed introduction to motion segmentation and object classification (an important basis for human motion analysis systems) and semantic description of behaviors (an interesting direction which has recently received increasing attentions).

(6) We also provide more detailed discussions on research challenges and future research directions in human motion analysis than any other earlier reviews.

The remainder of this paper is organized as follows. Section 2 reviews the work on human detection including motion segmentation and moving object classification. Section 3 covers human tracking, which is divided into four categories of methods: model-based, region-based, active-contour-based and feature-based. The paper then extends the discussion to the recognition and description of human activities in image sequences in Section 4. Section 5 analyzes some challenges and presents some possible directions for future research at length. Section 6 concludes this paper.