Anomalous entities detection and localization in pedestrian flows

Abstract

We propose a novel Gaussian kernel based integration model (GKIM) for anomalous entities detection and localization in pedestrian flows. The GKIM integrates spatio-temporal features for efficient and robust motion representation to capture the distinctive and meaningful information about the anomalous entities. We next propose a block based detection framework by training a recurrent conditional random field (R-CRF) using the GKIM features. The trained R-CRF model is then used to detect and localize the anomalous entities during the online testing stage. We conduct comprehensive experiments on three benchmark datasets and compare the performance of the proposed method with the state-of-the-art anomalous entities detection methods. Our experiments show that the proposed GKIM outperforms the compared methods in terms of equal error rate (EER) and detection rate (DR) in both frame-level and pixel-level comparisons. The frame-level analysis detects the presence of an anomalous entity in a frame regardless of its location. The pixel-level analysis localizes the anomalous entity in term of its pixels.

Introduction

The population growth and traffic congestion in urban areas are rapidly increasing which makes the safety of pedestrians a key concern [1]. Therefore, it is important to analyze the pedestrian flows to facilitate smart video surveillance for ensuring pedestrian safety. For this purpose, an important task to ensure pedestrian safety is the detection and localization of anomalous entities in the pedestrian flows which can be used to warn against the possible risks.

The term anomalous entity may have different interpretations varying significantly depending on the given context. Therefore, learning a generic model for anomaly detection is a challenging problem. In this paper, an anomalous entity is considered to be a moving object exhibiting motion patterns that do not conform to the expected behavior in the pedestrian flow and may warrant special attention. Such entities present infrequent behavior compared with the normal prevalent behaviors. This definition is not limited to specific entity types (a running pedestrian or a cyclist) or scenes (sparse or dense). Similar definitions are presented by a number of papers that addressed the problem in recent years [2], [3], [4], [5], [6], [7]. We consider that anomalous entities are rare in the pedestrian flow and they are different from the majority. The motivation of our paper is to build a generic model for the detection and localization of anomalous entities in pedestrian flows. We consider each moving object as part of the pedestrian flows and non-moving objects or groups of pedestrians as a background.

A number of computer vision methods for video surveillance [2], [3], [4], [5], [7], [8] have previously addressed anomalous entities detection. Most of these methods assume that the pedestrian flows are very consistent in motion. In fact, this assumption is not realistic since the pedestrian flows may be scattered and sparse. For example, the frequency and crowdedness of the pedestrians at a certain location may be higher in the official hours and lower during the weekends and later hours. Furthermore, the problem of detecting and locating anomalous entities in the pedestrian flows is very challenging due to the appearance variations of individual entities, temporal variations, and view angle changes.

To address the above mentioned challenges, we present an efficient method for anomalous entities detection and localization in pedestrian flows that does not rely on the assumption of pedestrian flow consistency. We propose a novel Gaussian kernel based integration model (GKIM). Our GKIM is based on a Gaussian kernel based integration of local difference binary patterns (LDBP) [9] and nested motion descriptor (NMD) [10]. We consider the LDBP since it is characterized by the compact representation of spatial information while the NMD is employed to encode temporal information. The distinguishing feature of NMD is the representation of motion information without requiring an explicit optical flow estimate. Furthermore, we exploit both LDBP and NMD simultaneously to integrate their strengths into a unified model. We propose a Gaussian kernel based approach to integrate the spatio-temporal features by transforming the trace and the determinant of our feature Jacobian matrix into a distinctive space. Therefore, the GKIM represents high quality description of anomalous entities in term of most distinctive information. GKIM models the evolving relative spatial relationships and captures a specific nuance of the underlying motion considering temporal variations. Due to the aforementioned properties, our proposed GKIM is independent of the scattered, sparse, and dense nature of the pedestrian flows.

The complete flow of our proposed method is shown in Fig. 1. In order to detect and localize anomalous entities, we divide each video frame into blocks of equal size where the spatio-temporal features for each block are extracted. To this end, the features are used as a-priori for recurrent conditional random field (R-CRF) [11] training which detect and localize anomalous entities during the testing stage. We propose to use the R-CRF since it can deal efficiently with the label bias problem [11] by integrating the traditional conditional random field (CRF) [12] and recurrent neural networks (RNN) [13]. The main contributions of this paper are:

• We propose a novel model called GKIM for anomalous entities detection and localization. One of the major attraction of the GKIM is its capability to model anomalous entities distinctively in pedestrian flows representing different degrees of scatteredness and sparseness. In the GKIM, we propose Jacobian matrix and Gaussian based kernel integration to transform spatio-temporal features into discriminant and unique representation. To the best of our knowledge, we are the first to explore R-CRF for entities classification in pedestrian flows.

• We extensively evaluate the proposed method on three standard datasets and compared to 11 state-of-the art methods. Our results show that the proposed method significantly outperforms all 10 state-of-the-art methods.

• We categorize state-of-the-art methods and present a comprehensive survey in this area in the next section.

To assess the proposed GKIM model, we perform extensive experiments on three benchmark datasets and compare the results with 11 state-of-the-art methods: the mixture of dynamic texture (MDT) [8], the mixture of optical flow (MPPCA) [14], the social force (SF) [15], the multiple location monitors (MLM) [16], the clustering and sparse coding (CSC) [4], the holistic features (HF) [5], hierarchical feature representation (HFR) [17], the pedestrian energy map (PEM) [18], the statistical histograms model (SHM) [6], the change detection model (CDM) [7], and the spatial-aware motion reconstruction (SMR)[19]. Our results show that GKIM achieves superior anomalous entities detection. Moreover, our proposed GKIM outperforms the compared methods in both frame-level and pixel-level analysis in terms of equal error rate (EER) and detection rate (DR). The frame-level analysis detects the presence of an anomalous entity in a frame regardless of its location. The pixel-level analysis localizes the anomalous entity in term of its pixels.

The rest of the paper is organized as follows. In Section 2, an overview of related work is provided. The proposed method for the detection and localization of anomalous entities is presented in Section 3. Experimental results on the benchmark datasets are shown in Section 4 and the conclusion is presented in Section 5.