Semantical video coding: Instill static-dynamic clues into structured bitstream for AI tasks

Abstract

Traditional media coding schemes typically encode image or video into a semantic-unknown binary stream, which fails to directly support downstream intelligent tasks at the bitstream level. Semantically Structured Image Coding (SSIC) (Sun et al., 2020) makes the first attempt to enable partial-decoding image intelligent task analysis via a Semantically Structured Bitstream (SSB). However, the SSIC considers image coding and its generated SSB only contains the static object information. In this paper, we propose an advanced Semantically Structured Video Coding (SSVC). Video signals contain more rich dynamic motion information and redundancy. Thus, we present a reformulation of semantically structured bitstream (SSB) in SSVC which contains both static object characteristics and dynamic motion clues. Specifically, we introduce optical flow to encode continuous motion information and reduce cross-frame redundancy via a predictive coding architecture, then the optical flow and residual information are reorganized into SSB, which enables the proposed SSVC could better adaptively support video-based downstream intelligent applications. Extensive experiments on various vision tasks demonstrate that the proposed SSVC framework could directly support multiple intelligent tasks just depending on a partially decoded bitstream, saving bitrate consumption for intelligent analytics.