MTRFN: Multiscale Temporal Receptive Field Network for Compressed Video Action Recognition at Edge Servers

Abstract

With the wide deployment of Internet of Things monitoring terminals, a tremendous number of videos are accumulated continuously. Big data processing and analysis-based action recognition has an increasingly important role in making cities simpler, better, and smarter. The traditional cloud server-centered analysis mode has to spend extra time transmitting vast video data terminals to remote cloud servers, which is always violated in real implementation. Edge servers with limited caching and computation capacities near the monitoring terminals enable implementation. However, due to the dependency on training data and the high complexity of extracting information and network architecture, existing image domain-based methods cannot be implemented at edge servers. Moreover, recognizing actions with different durations is still challenging. Due to these issues, we extend the traditional image domain to the compressed domain to efficiently extract the information of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$I$ </tex-math></inline-formula> frames and physical knowledge motion vectors (MVs), which can reflect the multiscale temporal feature just by partial decoding. To recognize the actions with different durations, a multiscale temporal receptive field network (MTRFN), including short-term and long-term branches, is proposed to simultaneously capture the action’s instant change based on the extracted MVs, the long temporal feature between adjacent <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$I$ </tex-math></inline-formula> frames, and the interaction between them. The results show that our algorithm can achieve a better balance between accuracy and computational complexity.