Natural Language Description of Videos for Smart Surveillance

Aniqa Dilawari,Zahoor-Ur Rehman,Yunyoung Nam,Yasser D Al-Otaibi,Atta-Ur Rahman,Muhammad Usman Ghani Khan

doi:10.3390/app11093730

Abstract

After the September 11 attacks, security and surveillance measures have changed across the globe. Now, surveillance cameras are installed almost everywhere to monitor video footage. Though quite handy, these cameras produce videos in a massive size and volume. The major challenge faced by security agencies is the effort of analyzing the surveillance video data collected and generated daily. Problems related to these videos are twofold: (1) understanding the contents of video streams, and (2) conversion of the video contents to condensed formats, such as textual interpretations and summaries, to save storage space. In this paper, we have proposed a video description framework on a surveillance dataset. This framework is based on the multitask learning of high-level features (HLFs) using a convolutional neural network (CNN) and natural language generation (NLG) through bidirectional recurrent networks. For each specific task, a parallel pipeline is derived from the base visual geometry group (VGG)-16 model. Tasks include scene recognition, action recognition, object recognition and human face specific feature recognition. Experimental results on the TRECViD, UET Video Surveillance (UETVS) and AGRIINTRUSION datasets depict that the model outperforms state-of-the-art methods by a METEOR (Metric for Evaluation of Translation with Explicit ORdering) score of 33.9%, 34.3%, and 31.2%, respectively. Our results show that our framework has distinct advantages over traditional rule-based models for the recognition and generation of natural language descriptions.

Highlights

There is an exponential increase in digital multimedia, resulting in the generation of enormous amounts of video data
This section details the results of the TRECViD, UET Video Surveillance (UETVS) and AGRIINTRUSION datasets
We compare the results of this multitask learning-based framework with two baseline models—an long short-term memory (LSTM) semantic compositional network (SCN-LSTM) and multimofal stochastic recurrent neural network (RNN) (MS-RNN)—and video descriptions using deep neural networks

Summary

Introduction

There is an exponential increase in digital multimedia, resulting in the generation of enormous amounts of video data. The growing rate of multimedia content uploaded on the Internet involves automatic interpretation and description of the videos for the retrieval of important information This can be useful in surveillance, security, human–computer interaction, robotic intelligence and even helps visually impaired people. Among these applications, an automatic description of videos in a natural language is gaining interest, where we give a video to the deep learning framework that converts it into one or multiple sentences. More complex rules were applied in [5] that contained a relatively large vocabulary to generate sentences These approaches require monotonous work when the data is huge. The results of these approaches were lacking with large datasets like Microsoft Common Objects in Context (MS COCO) [8]

Methods

Results

Conclusion