Frame rate up-conversion assisted with camera auto exposure information

Abstract

Frame rate up conversion (FRC) is the process of converting between different frame rates for targeted display formats. Besides scanning format applications for large displays, FRC can be used to increase the frame rate of video at the receiver end for video telephony, video streaming or playback applications for mobile platforms where bandwidth savings are crucial. Many algorithms have been proposed for decoder/receiver side FRC. However, most of them are from video encoding/decoding point of view. We systematically studied the strategies of utilizing the camera 3A (auto exposure, auto white balance and auto focus) information to assist FRC process, while in this paper we focus on the technique using camera exposure information to assist the decoder FRC. In the proposed strategy the exposure information as well as other camera 3A related information is packetized as the meta data which is attached to the corresponding frame and transmitted together with the main video bit stream to the decoder side for FRC assistance. The meta data contains information such as zooming, auto focus, AE (auto exposure), AWB (auto white balance) statistics, scene change detection, global motion detected from motion sensors. The proposed meta data consists of camera specific information which is different than just sending motion vectors or mode information to aid FRC process. Compared to traditional FRC approaches used in mobile platforms, the proposed approach is a low-complexity, low-power solution which is crucial in resource constrained environments such as mobile platforms.