Multiview fusion for activity recognition using deep neural networks

Rahul Kavi; Vinod Kulathumani; Fnu Rohit; Vlad Kecojevic

doi:10.1117/1.JEI.25.4.043010

18 July 2016 Multiview fusion for activity recognition using deep neural networks

Rahul Kavi, Vinod Kulathumani, Fnu Rohit, Vlad Kecojevic

Author Affiliations +

Journal of Electronic Imaging, Vol. 25, Issue 4, 043010 (July 2016). https://doi.org/10.1117/1.JEI.25.4.043010

Abstract

Convolutional neural networks (ConvNets) coupled with long short term memory (LSTM) networks have been recently shown to be effective for video classification as they combine the automatic feature extraction capabilities of a neural network with additional memory in the temporal domain. This paper shows how multiview fusion can be applied to such a ConvNet LSTM architecture. Two different fusion techniques are presented. The system is first evaluated in the context of a driver activity recognition system using data collected in a multicamera driving simulator. These results show significant improvement in accuracy with multiview fusion and also show that deep learning performs better than a traditional approach using spatiotemporal features even without requiring any background subtraction. The system is also validated on another publicly available multiview action recognition dataset that has 12 action classes and 8 camera views.

Citation Download Citation

Rahul Kavi, Vinod Kulathumani, Fnu Rohit, and Vlad Kecojevic "Multiview fusion for activity recognition using deep neural networks," Journal of Electronic Imaging 25(4), 043010 (18 July 2016). https://doi.org/10.1117/1.JEI.25.4.043010

Published: 18 July 2016

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available