Frame Rate Control Research Articles

λ-Domain Rate Control via Wavelet-Based Residual Neural Network for VVC HDR Intra Coding.

High dynamic range (HDR) video offers a more realistic visual experience than standard dynamic range (SDR) video, while introducing new challenges to both compression and transmission. Rate control is an effective technology to overcome these challenges, and ensure optimal HDR video delivery. However, the rate control algorithm in the latest video coding standard, versatile video coding (VVC), is tailored to SDR videos, and does not produce well coding results when encoding HDR videos. To address this problem, a data-driven λ-domain rate control algorithm is proposed for VVC HDR intra frames in this paper. First, the coding characteristics of HDR intra coding are analyzed, and a piecewise R-λ model is proposed to accurately determine the correlation between the rate (R) and the Lagrange parameter λ for HDR intra frames. Then, to optimize bit allocation at the coding tree unit (CTU)-level, a wavelet-based residual neural network (WRNN) is developed to accurately predict the parameters of the piecewise R-λ model for each CTU. Third, a large-scale HDR dataset is established for training WRNN, which facilitates the applications of deep learning in HDR intra coding. Extensive experimental results show that our proposed HDR intra frame rate control algorithm achieves superior coding results than the state-of-the-art algorithms. The source code of this work will be released at https://github.com/TJU-Videocoding/WRNN.git.

Convolutional neural network-based optimal R-λ intra rate control in Versatile Video Coding

Since intra frame rate control has no prior information to rely on, it is difficult to obtain suitable initial rate control parameters for achieving optimal intra coding. The optimal bit allocation and λ decision are proposed based on a convolutional neural network (CNN) in R-λ intra frame rate control. First, according to the rate distortion (R-D) and rate lambda (R-λ) relationships of the coding tree unit (CTU), a CNN with four outputs is designed to predict the key parameters of R-D and R-λ curves. Second, with the R-D optimization of a frame, the coding optimization equation of the frame λ and the target bit can be built to derive the optimal CTU bit allocation. Last, a suitable CTU λ can be obtained based on the allocated bit and predicted R-λ curve for a CTU to minimize the total distortion of the current frame. The experimental results show that the proposed algorithm can improve the intra frame rate control performance by 0.31 dB with a good rate control accuracy of 4.76%.

Consistent Quality Oriented Rate Control in HEVC Via Balancing Intra and Inter Frame Coding

Consistent quality oriented rate control in video coding has attracted much more attention. However, the existing efforts only focus on decreasing variations between every two adjacent frames, but neglect coding trade-off problem between intra and inter frames. In this paper, we deal with it from a new perspective, where intra frame quantization parameter (IQP) and rate control are optimized for balanced coding. First, due to the importance of intra frames, a new framework is proposed for consistent quality oriented IQP prediction, and then we remove unqualified IQP candidates using the proposed penalty term. Second, we extensively evaluate possible features, and select target bits per pixel for all remaining frames, average and standard variance of frame QPs, where equivalent acquisition methods for QP features are given. Third, predicted IQPs are clipped effectively according to bandwidth and previous information for better bit rate accuracy. Compared with High Efficiency Video Coding (HEVC) reference baseline, experiments demonstrate that our method reduces quality fluctuation greatly by 37.2% on frame-level standard variance of peak-signal-noise-ratio (PSNR) and 45.1% on that of structural similarity (SSIM). Moreover, it also can have satisfactory results on Rate-Distortion (R-D) performance, bit accuracy and buffer control.

Quality driven modulation rate optimization for energy efficient wireless video relays

Modern mobile computing devices such as smart phones have become the major user-centric video data sources in future 5G wireless networks. However, battery energy efficiency becomes the critical bottleneck for such uplink multimedia contributors. In this paper we propose a new approach to reduce energy consumption while improving Quality of Experience (QoE), by innovatively investigating video encoding frame prioritization, mode, dependency, and rate control overheads, based on communication model of a transmitter device to an access-point via a relay device. In the proposed approach, we investigate various frame level encoding dependencies and different communication data rates to study their impacts on energy efficiency and QoE. We also incorporate some simple game theory concepts to determine the best compromise of energy-quality tradeoff between the transmitter and the relay. Simulation results demonstrate that cross layer exploration of encoding dependency and game theoretical rate control have significant energy saving and quality gain potentials.

Adaptive Gradient Information and BFGS Based Inter Frame Rate Control for High Efficiency Video Coding

In order to meet the emerging demands of high-fidelity video services, a new video coding standard — High Efficiency Video Coding (HEVC) is developed to improve the compression performance of high definition (HD) videos and save half of the bitrate for the same perceptual video quality compared with H.264/Advanced Video Coding (AVC). Rate control still plays a significant role in HD video data transmission via the communication channel. However, R-lambda model based HEVC rate control algorithm does not take the relationship between the encoding complexity and Human Visual System (HVS) into account, what’s more, the convergence speed of Least Mean Square (LMS) algorithm is slow. In this paper, an adaptive gradient information and Broyden Fletcher Goldfarb Shanno (BFGS) based R-lambda model (GBRL) is proposed for the inter frame rate control, where the gradient based on Sobel operator can effectively measure the frame-content complexity and BFGS algorithm converges speedily than LMS algorithm. Experimental results show that the proposed GBRL method can achieve bitrate error reduction and peak signal to noise ratio (PSNR) improvement especially for the sequences with large motion, compared to the state-of-the-art rate control methods. In addition, if the optimal initial quantization parameter (QP) prediction model based on linear regression can be incorporated into the proposed GBRL method, the performance of rate control can be further improved.

Smartphone Image Receiver Architecture for Optical Camera Communication

As an extension of visible light communications (VLC), optical camera communications (OCC) are poised to play an important role in the success of VLC technology. Comparing with VLC systems based on photodiodes, OCC systems have a number of advantages in terms of hardware, communications channels, and business trends. However, these systems also have several disadvantages, such as limitations on the data rate, the need for synchronization between the transmitter and receiver, and inter-channel interference. These issues are a result of limitations in the camera sampling rates, frame rate variations, and motion stabilization. To address these limitations, we propose a new image sensor architecture for smartphone-based OCC that incorporates three primary functions: motion stabilization, frame rate control, and auto-exposure control. First, the motion stabilization function combines image sensor data and gyroscope motion information in order to select accurate pixel data. Second, the frame rate control function increases the frame rate using an over-scan scheme and contributes to the data throughput and motion compensation in the time domain. Finally, we control both the auto-exposure and focus functions in order to avoid blurring effects and variations in the frame rate. Experiments show that the proposed receiver architecture with adaptive frame rates and motion stabilization holds great promise for OCC.

A region-based intra-frame rate control scheme by jointing inter-frame dependency and inter-frame correlation

During I frame switching, the subjective quality between I frame and P frames usually have obvious fluctuation due to different coding methods. The periodic temporal visual fluctuation will cause video flicker. According to extensive experiments, we observe that I frame flicker possess a strong regional characteristics and different region have different degree of flicker. Based on this observation, a region-based I frame rate control scheme is proposed to suppress I frame flicker according to the different characteristics of the moving and non-moving regions. Firstly, by jointly considering the inter-frame dependency between I frame and subsequent un-encoded P frames and the inter-frame correlation between I frame and previous encoded P frame, an optimization model is proposed to achieve the optimal QPs for different regions. Secondly, a region-based inter-frame dependency model is proposed to separately describe the inter-frame dependency of different regions, which can accurately describe their description of the inter-frame dependency. The experimental results demonstrate that the proposed scheme can efficiently suppress I frame flicker and maintain the smoothness of subjective quality. Moreover, the proposed scheme can achieve a PSNR gain by 0.26 dB on average when compared with the rate control scheme adopted by the HEVC reference software HM15.0.

A Novel Perceptual Intra Frame Rate Control for High Efficiency Video Coding

A Novel Perceptual Intra Frame Rate Control for High Efficiency Video Coding

An Efficient Frame-Content Based Intra Frame Rate Control for High Efficiency Video Coding

Rate control plays an important role in the rapid development of high-fidelity video services. As the High Efficiency Video Coding (HEVC) standard has been finalized, many rate control algorithms are being developed to promote its commercial use. The HEVC encoder adopts a new R-lambda based rate control model to reduce the bit estimation error. However, the R-lambda model fails to consider the frame-content complexity that ultimately degrades the performance of the bit rate control. In this letter, a gradient based R-lambda (GRL) model is proposed for the intra frame rate control, where the gradient can effectively measure the frame-content complexity and enhance the performance of the traditional R-lambda method. In addition, a new coding tree unit (CTU) level bit allocation method is developed. The simulation results show that the proposed GRL method can reduce the bit estimation error and improve the video quality in HEVC all intra frame coding.

Editorial Issue 26.1

Reference EPFL-ARTICLE-207397doi:10.1002/cav.1634View record in Web of Science Record created on 2015-04-13, modified on 2016-08-09

Streaming 3D deforming surfaces with dynamic resolution control

AbstractReal‐time streaming of shape deformations in a shared distributed virtual environment is a challenging task due to the difficulty of transmitting large amounts of 3D animation data to multiple receiving parties at a high frame rate. In this paper, we present a framework for streaming 3D shape deformations, which allows shapes with multi‐resolutions to share the same deformations simultaneously in real time. The geometry and motion of deforming mesh or point‐sampled surfaces are compactly encoded, transmitted, and reconstructed using the spectra of the manifold harmonics. A receiver‐based multi‐resolution surface reconstruction approach is introduced, which allows deforming shapes to switch smoothly between continuous multi‐resolutions. On the basis of this dynamic reconstruction scheme, a frame rate control algorithm is further proposed to achieve rendering at interactive rates. We also demonstrate an efficient interpolation‐based strategy to reduce computing of deformation. The experiments conducted on both mesh and point‐sampled surfaces show that our approach achieves efficient performance even if deformations of complex 3D surfaces are streamed. Copyright © 2013 John Wiley & Sons, Ltd.

Ensuring QoS with Adaptive Frame Rate and Feedback Control Mechanism in Video Streaming

Video over best-effort packet networks is cumbered by a number of factors including unknown and timevarying bandwidth, delay and losses, as well as many additional issues such as how to fairly share the network resources amongst many flows and how to efficiently perform one-to-many communication for popular content. This research investigates video streaming formats, encoding and compression techniques towards the development and simulation of a rate adaptation model to reduce packet loss. The thrust of this research aimed at enriching and enhancing the quality of video streaming over the wireless network. We developed both mathematical models which were thereafter simulated to depict the need for advancing the existing solution for packet scheduling towards recovery from packet loss and error handling in video streaming.Keywords: Quantity of Service (QOS), Adaptive Frame Rate, Video Streaming

Digital Driving Method for Low Frame Frequency and 256 Gray Scales in Liquid Crystal on Silicon Panels

A liquid crystal on silicon (LCoS) panel for projector applications is developed using a new digital driving method to increase the gray scale of the projected image without increasing the driving frequency. The LCoS panel, which has a wide video graphics array (WVGA) resolution, is fabricated using 0.18 μm CMOS process technology with an area of 9600 μm × 5800 μm. The projection system is successfully demonstrated using the fabricated LCoS panel and the proposed digital driving method. The measured results of the projection system show that the proposed driving method achieves an 8-bit gray scale at a driving frequency of 75.6 MHz, which is less than 13.7% of the frequency in previously reported methods such as frame rate control and pulse width modulation.

Joint optimization of continuity and quality for streaming video

This paper aims to reduce the amount of prebuffering required to ensure a maximum video continuity in streaming. Current approaches do this by slowing the playout frame rate of the decoder, this is known as adaptive media playout (AMP). However, doing this introduces playout distortion to the viewers as the video is played slower than its natural playout rate. We approach this by proposing a frame rate control scheme that jointly adjusts the encoder frame generation rate of the encoder and the playout frame rate of the decoder. In addition to using AMP to improve video continuity, we also allow the encoder to increase the encoder frame generation rate. This means the encoder will be sending more frames to the decoder to quickly increase the number of frames available at the playback buffer, thus lowering the chance of buffer underflow which causes discontinuity in video playback. At the same time, the increase in the number of frames at the playback buffer may mean that the decoder does not need to use AMP to delay the playback, thus lowering the playback distortion. However, the increase in encoder frame generation rate comes at a price because frame quality will need to decrease in order to meet the constraint on available network bandwidth. This implies that the scheme needs to find the optimal trade-off between frame quality, playout distortion and video continuity. To do that, we characterize the frame rate control problem using Lyapunov optimization. We then systematically derive the optimization policies. We also show that these policies can be decoupled into separate encoder and decoder optimization policies, thus allowing for a distributed implementation. Simulation results show significant reductions in the prebuffering requirements over a scheme that perform no frame rate control and lower playout distortions compared to the AMP schemes, while exhibiting a modest drop in frame quality.

Two Parallel Strategies for Real-time Spatial Video Denoising for Multi-core Processors

Video denoising is usually a time consuming process especially for large video files. With the advancement of the processor technology, it is possible to perform video denoising in real-time on multi-core processors. In this paper, we study parallel techniques for denoising real-time video on multi-core processor which work on both shared memory model and distributed memory model. We investigate two approaches: a block approach, which assigns a group of threads to each block of video frames; and a distributor approach, which uses one thread to distribute the frame data to each thread. Our experiments focus on the image denoising technique based on the total variation but the approach can be integrated with other image denoising algorithm like discrete wavelet transform (DWT) or diffusion technique. We found that by using the distributor strategy, we can achieve speedup which is 1.27 times faster than the block strategy and the video frame rate can be increased by 7.43%. Moreover, we also apply the prefetching technique which further enhance frame rate by 22.02% and frame rate control to stabilize frame rate and retain the original video length during denoising and playing in real-time. Our method also has good denoised quality which is better than previous work in [1] in average case. General Terms Algorithms, Framework, Image Processing, Video Processing, High Performance Computing, Parallel Computing Keywords video denoising, parallel computing, OpenMP, ROF model, total variation.

Joint Spatial-Temporal Quality Improvement Scheme for H.264 Low Bit Rate Video Coding via Adaptive Frameskip

Conventional rate control (RC) schemes for H.264 video coding usually regulate output bit rate to match channel bandwidth by adjusting quantization parameter (QP) at fixed full frame rate, and the passive frame skipping to avoid buffer overflow usually occurs when scene changes or high motions exist in video sequences especially at low bit rate, which degrades spatial-temporal quality and causes jerky effect. In this paper, an active content adaptive frame skipping scheme is proposed instead of passive methods, which skips subjectively trivial frames by structural similarity (SSIM) measurement between the original frame and the interpolated frame via motion vector (MV) copy scheme. The saved bits from skipped frames are allocated to coded key ones to enhance their spatial quality, and the skipped frames are well recovered based on MV copy scheme from adjacent key ones at the decoder side to maintain constant frame rate. Experimental results show that the proposed active SSIM-based frameskip scheme acquires better and more consistent spatial-temporal quality both in objective (PSNR) and subjective (SSIM) sense with low complexity compared to classic fixed frame rate control method JVT-G012 and prior objective metric based frameskip method.

A Testbed and Methodology for Comparing Live Video Frame Rate Control Methods

Most existing methods for video frame rate control were developed for prerecorded video. With the proliferation of mobile audiovisual communication devices, it is becoming important to develop frame rate control methods for live (i.e., not prerecorded) video as well. In this letter, we present a testbed and the corresponding methodology suitable for comparing such methods. We illustrate the methodology and the use of the testbed through comparison of an existing state-of-the-art frame rate control method against a simple new control method developed specifically for live video.

A reduction method of dynamic false contour utilizing the frame‐rate‐control method with suppression of the side effect on PDPs

Abstract— The plasma‐display panel (PDP) is a type of flat‐panel display that can display a high‐quality image. However, when moving images are displayed, annoying disturbances such as false contour noise occurs. This noise is called dynamic false contour (DFC). To achieve a higher‐quality image, DFC has to be reduced. Therefore, a new method to reduce DFC is proposed. To find a way to reduce DFC, a new evaluation value for it has been defined: the evaluation value of dynamic false contour (EVDFC). This value is defined on the basis of a person's subjective evaluation. By applying this value, the cause that generates DFC can be identified. On the basis of these studies, a new method for reducing DFC by applying frame‐rate control (FRC) with suppression of the side effect is proposed. This improved method can be used to provide high‐quality images.

Design of Real-Time Self-Frame-Rate-Control Foreground Detection for Multiple Camera Surveillance System

Emerging video surveillance technologies are based on foreground detection to achieve event detection automatically. Integration foreground detection with a modern multi-camera surveillance system can significantly increase the surveillance efficiency. The foreground detection often leads to high computational load and increases the cost of surveillance system when a mass deployment of end cameras is needed. This paper proposes a DSP-based foreground detection algorithm. Our algorithm incorporates a temporal data correlation predictor (TDCP) which can exhibit the correlation of data and reduce computation based on this correlation. With the DSP-oriented foreground detection, an adaptive frame rate control is developed as a low cost solution for multi-camera surveillance system. The adaptive frame rate control automatically detects the computational load of foreground detection on multiple video sources and adaptively tunes the TDCP to meet the real-time specification. Therefore, no additional hardware cost is required when the number of deployed cameras is increased. Our method has been validated on a demonstration platform. Performance can achieve real-time CIF frame processing for a 16-camera surveillance system by single-DSP chip. Quantitative evaluation demonstrates that our solution provides satisfied detection rate, while significantly reducing the hardware cost.

A robust FRC pattern design for visual artifacts and its hardware design in flat panel displays

Frame rate control (FRC) has been applied in monitors and TVs in order to reduce the costs for flat panel display manufacture. As FRC can represent the same gray scale level of an image with a smaller number of bits than that of the input image, it is a widely used technology. However, FRC causes visual artifacts by using the repeatedly predesigned FRC block pattern unit size in display devices. So, this paper analyzes the cause of these visual artifacts. In order to improve these visual artifacts, this paper proposes a pattern arrangement of the FRC unit blocks using, for the first time, a frame rolling method. The use of this pattern arrangement technique in FRC technology eliminates video noise. As it also enables the representation of various gray scales, the representation of gray scale for the image becomes softer and more delicate in flat panel displays.