Action segmentation and understanding in RGB videos with convolutional neural networks

Abstract

In this work, we propose three techniques for accelerating a modern action recognition pipeline. For reaching this point, we carried out an extensive study about the current state of action recognition. First, the traditional action recognition methods based on handcrafted features where reviewed, along with the ones focused on using machine learning as well as those which use deep learning techniques. Valuable insights were extracted from them, as well as the difficulties the task of action recognition carries out. Subsequently, we explored numerous video datasets available for properly train deep models in the task of understanding human actions. Then, several video action recognition works from the past three years where thoroughly studied. Coming back to the three proposed techniques, we first selected two of the reviewed deep learning works. Specifically, (1) Temporal Segment Networks (TSN), a Convolutional Neural Network (CNN) framework that makes use of a small number of video frames for obtaining robust predictions which have allowed to win the first place in the 2016 ActivityNet challenge; (2) MotionNet, a network that is capable of inferring optical flow from Red Green and Blue (RGB) frames by means of simple and transposed convolutions. As the third proposal we selected NVIDIA Video Loader (NVVL), a new Graphics Processing Unit (GPU) video decoding software developed by NVIDIA. Useful for efficiently reducing the data transfer and storage bottleneck of video deep learning applications. Finally, we combine the first and third technologies. For this, we trained the RGB stream of the TSN network in videos loaded with NVVL on a subset of daily actions from the University of Central Florida 101 (UCF101) dataset. Thus proving their validity for their integration into a deep learning action recognition system that is both faster and effective.