Video Quality Enhancement Research Articles

Intelligent Visual Tracking in Unstabilized Videos

Visual tracking requirement is increasing day by day due to the availability of high-performance digital cameras at low prices. Visual tracking becomes a complex problem when cameras suffer with unwanted and unintentional motion, resulting in motion-blurred unstabilized video. The problem in hand becomes more challenging when the target of interest is to be detected automatically in this unstabilized video. This paper presents a comprehensive single intelligent solution for these problems. The proposed algorithm auto-detects the camera motion, filters out the unintentional motion while stabilizing the video keeping intentional motion only using speeded-up robust features (SURF) technique. Motion smear due to unstabilization is also removed, providing sharp stabilized video output with video quality enhancement of up to 20dB. Gabor filter is used innovatively for auto-detection of target of interest in each stabilized frame. Then the target is tracked using SURF method.

An Intelligent Fuzzy Logic-Based Content and Channel Aware Downlink Scheduler for Scalable Video over OFDMA Wireless Systems

The recent advancements of wireless technology and applications make downlink scheduling and resource allocations an important research topic. In this paper, we consider the problem of downlink scheduling for multi-user scalable video streaming over orthogonal frequency division multiple access (OFDMA) channels. The video streams are precoded using a scalable video coding (SVC) scheme. We propose a fuzzy logic-based scheduling algorithm, which prioritises the transmission to different users by considering video content, and channel conditions. Furthermore, a novel analytical model and a new performance metric have been developed for the performance analysis of the proposed scheduling algorithm. The obtained results show that the proposed algorithm outperforms the content-blind/channel aware scheduling algorithms with a gain of as much as 19% in terms of the number of supported users. The proposed algorithm allows for a fairer allocation of resources among users across the entire sector coverage, allowing for the enhancement of video quality at edges of the cell while minimising the degradation of users closer to the base station.

A quality enhancement framework with noise distribution characteristics for high efficiency video coding

Video coding effectively reduces the amount of video data while unavoidably producing compression noise. Compression noise can cause significant artifacts in compressed video, such as blocking, ringing, and blurring, which seriously affects the visual quality of videos and the value of videos for content analysis. In compressed video quality enhancement, few methods based on deep learning fully consider the relationship between video content and compression noise or the possibility of uniting the encoder or the decoder to enhance the quality of compressed video. In an approach different from existing methods, we propose a video quality enhancement framework based on the distribution characteristics of compression noise. The proposed framework consists of two parts: at the encoder, we propose a convolutional neural network (CNN)-based in-loop filtering network combined with noise distribution (IFN-ND) characteristics for the I frame instead of high efficiency video coding (HEVC) standard in-loop filters; at the decoder, we propose a CNN-based quality enhancement network combined with the noise distribution characteristics (PQEN-ND) for the P frames. The noise characteristics are extracted from the code stream to further improve the performance of the proposed networks. The experiments show that the proposed method can significantly improve the quality of HEVC compressed video, achieving an average 12.84% reduction in the BD rate and up to a 1.0476 dB increase in the peak signal-to-noise ratio (PSNR).

Biprediction-Based Video Quality Enhancement via Learning.

Convolutional neural networks (CNNs)-based video quality enhancement generally employs optical flow for pixelwise motion estimation and compensation, followed by utilizing motion-compensated frames and jointly exploring the spatiotemporal correlation across frames to facilitate the enhancement. This method, called the optical-flow-based method (OPT), usually achieves high accuracy at the expense of high computational complexity. In this article, we develop a new framework, referred to as biprediction-based multiframe video enhancement (PMVE), to achieve a one-pass enhancement procedure. PMVE designs two networks, that is, the prediction network (Pred-net) and the frame-fusion network (FF-net), to implement the two steps of synthesization and fusion, respectively. Specifically, the Pred-net leverages frame pairs to synthesize the so-called virtual frames (VFs) for those low-quality frames (LFs) through biprediction. Afterward, the slowly fused FF-net takes the VFs as the input to extract the correlation across the VFs and the related LFs, to obtain an enhanced version of those LFs. Such a framework allows PMVE to leverage the cross-correlation between successive frames for enhancement, hence capable of achieving high accuracy performance. Meanwhile, PMVE effectively avoids the explicit operations of motion estimation and compensation, hence greatly reducing the complexity compared to OPT. The experimental results demonstrate that the peak signal-to-noise ratio (PSNR) performance of PMVE is fully on par with that of OPT while its computational complexity is only 1% of OPT. Compared with other state-of-the-art methods in the literature, PMVE is also confirmed to achieve superior performance in both objective quality and visual quality at a reasonable complexity level. For instance, PMVE can surpass its best counterpart method by up to 0.42 dB in PSNR.

Quality enhancement of VVC intra-frame coding for multimedia services over the Internet

In this article, versatile video coding, the next-generation video coding standard, is combined with a deep convolutional neural network to achieve state-of-the-art image compression efficiency. The proposed hierarchical grouped residual dense network exhaustively exploits hierarchical features in each architectural level to maximize the image quality enhancement capability. The basic building block employed for hierarchical grouped residual dense network is residual dense block which exploits hierarchical features from internal convolutional layers. Residual dense blocks are then combined into a grouped residual dense block exploiting hierarchical features from residual dense blocks. Finally, grouped residual dense blocks are connected to comprise a hierarchical grouped residual dense block so that hierarchical features from grouped residual dense blocks can also be exploited for quality enhancement of versatile video coding intra-coded images. Various non-architectural and architectural aspects affecting the training efficiency and performance of hierarchical grouped residual dense network are explored. The proposed hierarchical grouped residual dense network respectively obtained 10.72% and 14.3% of Bjøntegaard-delta-rate gains against versatile video coding in the experiments conducted on two public image datasets with different characteristics to verify the image compression efficiency.

Analysis of audio and video quality of voice over LTE (VoLTE) call

LTE is a broadband wireless communication standard, including a voice service that is upgraded to 5 G, with the core network technology. In recent years Voice over Internet Protocol (VoIP) became more accessible to customers because of the increasing competence of Internet services and the development of voice call applications. According to its increased capacity, data rates, and lower latency, LTE meets the demand for high data rates for many mobile users. Voice over LTE (VoLTE) is the Cellular network system for transmitting voice over TCP. To network operators or system manufacturers, the quality of service (QoS) assessment of VoLTE coverage is essential. This article presents the evaluation analysis of video quality QoS Key Performance Induction (KPI) and the POLQA Mean Audio Performance Rating in the VoLTE application. This measurement can be done for various video quality analysis display formats and packeting mode. Audio quality analysis in the VoLTE call is done by measuring the statistical and distribution information for Uplink and Downlink Mean opinion score (MOS). MOS for video quality assessment (Average) is conducted. Graphical analyses of the video rating distributions verify the calculated QoS KPIs for video calls. The calculation analysis finishes with the verification of the option in VoLTE Call for video codec screen formats and packeting mode. Audio performance can be defined as the audio quality of the test device by the visual display of Downlink and Uplink. Such knowledge will help software manufacturers on the IMS stack to enhance video quality and identify audio output systems on MOS basis. In this, research discuss VoLTE and Video over LTE (ViLTE) performance issues. In the ITRI VoLTE test environment, we analyse VoLTE voice quality in measurement experiments. We will also pose ViLTE Call with a check in a FET industrial mobile network in multimode or multi-band interworking. Our study provides information on the quality of the VoLTE and ViLTE services for operators.

Spatio-Temporal Deformable Convolution for Compressed Video Quality Enhancement

Recent years have witnessed remarkable success of deep learning methods in quality enhancement for compressed video. To better explore temporal information, existing methods usually estimate optical flow for temporal motion compensation. However, since compressed video could be seriously distorted by various compression artifacts, the estimated optical flow tends to be inaccurate and unreliable, thereby resulting in ineffective quality enhancement. In addition, optical flow estimation for consecutive frames is generally conducted in a pairwise manner, which is computational expensive and inefficient. In this paper, we propose a fast yet effective method for compressed video quality enhancement by incorporating a novel Spatio-Temporal Deformable Fusion (STDF) scheme to aggregate temporal information. Specifically, the proposed STDF takes a target frame along with its neighboring reference frames as input to jointly predict an offset field to deform the spatio-temporal sampling positions of convolution. As a result, complementary information from both target and reference frames can be fused within a single Spatio-Temporal Deformable Convolution (STDC) operation. Extensive experiments show that our method achieves the state-of-the-art performance of compressed video quality enhancement in terms of both accuracy and efficiency.

Hardware-Assisted, Low-Cost Video Transcoding Solution in Wireless Networks

Wireless video streaming has become an extremely popular application in recent years. Internet video streaming to mobile devices, however, faces several challenges, e.g., unstable wireless connections, long latency, high jitter and etc. Bitrate adaptive streaming and video transcoding solutions are widely used to address the above-mentioned issues, however, there are still several shortcomings of these approaches. Such challenges hinder providing satisfactory quality of video streaming service to the mobile users. We propose a hardware-assisted, real-time video transcoding solution implemented on a commercial off-the-shelf device, Raspberry Pi. We employ the software and hardware coupled architecture in order to improve the performance/quality of video streaming and enhance the user satisfactions in wireless network. Our video transcoding solution can be applied to both the downlink and uplink streaming: for downlink stream, it can provide agile bitrate adaptation to sudden network dynamics and enhance video quality by running our transcoding solution at the wireless edge. It can be used to uplink stream for broadcasting live streams in real-time. We present the design and implementation of our video transcoding system in both cases with practical scenarios. The evaluation results reveal that our transcoding solution enhances the performance of video streaming compared with other adaptive bitrate streamings and it provides higher video quality without causing rebuffering or video stall. We bridge the gap between the wireless channel capacity and the video quality while providing a better streaming experience to end user.

Optimal weighed Holoentropy for video compression: an advanced HEVC

ABSTRACTIn recent times, the applications of multimedia are rising in a greedy mode and hence the amount of video transactions is also increasing exponentially. This has shouted great demands on effective models on video encoding and also for reducing the transmission channel congestion. This research work introduces a managing technique termed weighted encoding for High-Efficiency Video Coding (HEVC). HEVC, also termed as MPEG-H Part 2 and H.265 is a video compression standard that is widely utilized AVC (H.264 or MPEG-4 Part 10). When compared to AVC, HEVC grants double the ratio of data compression at a similar level of quality of the video or considerably enhanced video quality at a similar bit rate. This work intends to optimize the weight that adopted in HEVC for encoding. For this, this paper proposes a new Iterative based propagation update in the water wave Optimization Algorithm (IPU-WWO), which is the improved form of Water wave Optimization (WWO). The performance of proposed IPU-WWO is compared over other conventional methods like Artificial Bee Colony (ABC), Firefly (FF), Particle Swarm Optimization (PSO) and Genetic Algorithm (GA) with respect to Peak Signal to Noise Ratio (PSNR). By doing the encoding process, it minimizes the video size with perceptually better quality video or PSNR.

Multiresolution Wavelet Transform Based Anisotropic Diffusion for Removing Speckle Noise in a Real-Time Vision-Based Database

In this research article, a novel algorithm is introduced to identify the noisy pixels in video frames and correct them to enhance video quality. The technique consists of three stages: fragmentation of the video sequences to respective 2D frames, noisy pixel identification in the 2D frames, and denoising the pixels to obtain original pixels. Due to the complexity in the background and the change in appearance of the body in motion, noise variation occurs. Various researchers discuss that in order to denoise the video sequences, spatio-temporal filtering is required which identifies noise and preserves the edges. In the first stage, the video sequences are analyzed for the removal of redundant frames. This is done by using the video fragmentation process in the MATLAB toolbox. In the next stage, color smoothing is applied to the target frames for processing the flat regions and identifying all the noisy pixels. In the final stage, an improvised multiresolution wavelet transform based anisotropic diffusion filtering is applied which enhances the denoising process in horizontal, vertical, and diagonal sub bands of the video frame signal. The proposed technique can remove the speckle noise and estimate the motion by preserving the minute details of the processed video frames.

A deep learning approach for quality enhancement of surveillance video

The growing number of surveillance cameras imposes great demand on high efficiency video coding. Although modern video coding standards have significantly improved the video coding efficiency, they are designed for general video rather than surveillance video. The special characteristics of surveillance video leave a large space for further performance improvement. In this paper, we leverage a deep learning approach to enhance the quality of compressed surveillance video. More specifically, we formulate the problem of frame enhancement as a regression problem and design a Residual Squeeze-and-Excitation Network (RSE-Net), to address it. RSE-Net extensively exploits the non-linear mapping between the reconstructed frame and the ground truth, with only a small number of parameters. Moreover, By improving You Only Look Once (YOLO) network, we successfully detect the grouped vehicles within a frame. A novel model training scheme is then developed through learning from the grouped vehicles. With the proposed scheme, we train a global model for both foreground and background of surveillance video. Experimental results show that our method achieves average 0.40 dB, 0.22 dB and 0.24 dB PSNR gains over H.265/HEVC anchor in AI, LDP and RA configurations, and produces visually pleasing results when applied to compressed surveillance video.

Object-Based Approach for Adaptive Source Coding of Surveillance Video

Intelligent analysis of surveillance videos over networks requires high recognition accuracy by analyzing good-quality videos that however introduce significant bandwidth requirement. Degraded video quality because of high object dynamics under wireless video transmission induces more critical issues to the success of smart video surveillance. In this paper, an object-based source coding method is proposed to preserve constant quality of video streaming over wireless networks. The inverse relationship between video quality and object dynamics (i.e., decreasing video quality due to the occurrence of large and fast-moving objects) is characterized statistically as a linear model. A regression algorithm that uses robust M-estimator statistics is proposed to construct the linear model with respect to different bitrates. The linear model is applied to predict the bitrate increment required to enhance video quality. A simulated wireless environment is set up to verify the proposed method under different wireless situations. Experiments with real surveillance videos of a variety of object dynamics are conducted to evaluate the performance of the method. Experimental results demonstrate significant improvement of streaming videos relative to both visual and quantitative aspects.

Efficient Computation Offloading in Mobile Cloud Computing for Video Streaming Over 5G

In this paper, we investigate video quality enhancement using computation offloading to the mobile cloud computing (MCC) environment. Our objective is to reduce the computational complexity required to covert a low-resolution video to high-resolution video while minimizing computation at the mobile client and additional communication costs. To do so, we propose an energy-efficient computation offloading framework for video streaming services in a MCC over the fifth generation (5G) cellular networks. In the proposed framework, the mobile client offloads the computational burden for the video enhancement to the cloud, which renders the side information needed to enhance video without requiring much computation by the client. The cloud detects edges from the upsampled ultra-high-resolution video (UHD) and then compresses and transmits them as side information with the original low-resolution video (e.g., full HD). Finally, the mobile client decodes the received content and integrates the SI and original content, which produces a high-quality video. In our extensive simulation experiments, we observed that the amount of computation needed to construct a UHD video in the client is 50%-60% lower than that required to decode UHD video compressed by legacy video encoding algorithms. Moreover, the bandwidth required to transmit a full HD video and its side information is around 70% lower than that required for a normal UHD video. The subjective quality of the enhanced UHD is similar to that of the original UHD video even though the client pays lower communication costs with reduced computing power.

A Convex Optimization Framework for Video Quality and Resolution Enhancement From Multiple Descriptions.

Transmission and compression technologies advancement over the past decade led to a shift of multimedia content towards cloud systems. Multiple copies of the same video are available through numerous distribution systems. Different compression levels, algorithms and resolutions are used to match the requirements of particular applications. As 4k display technologies are rapidly adopted, resolution enhancement algorithms are of vital importance. Current solutions do not take into account the particularities of different video encoders, while video reconstruction methods from compressed sources do not provide resolution enhancement. In this paper, we propose a multi source compressed video enhancement framework, where each description can have a different compression level and resolution. Using a variational formulation based on a modern proximal dual splitting algorithm, we efficiently combine multiple descriptions of the same video. Two applications are proposed: combining two compressed low resolution (LR) descriptions of a video sequence into a high resolution (HR) description and enhancing a compressed HR video using a LR compressed description. Tests are performed over multiple video sequences encoded with high efficiency video coding, at different compression levels and resolutions obtained through multiple down-sampling methods.

Expanding-Window BATS Code for Scalable Video Multicasting Over Erasure Networks

In this paper we consider scalable video multicasting over erasure networks with heterogeneous video quality requirements. With random linear network coding (RLNC) applied at the intermediate nodes the information received by the destinations is determined by the associated channel rank distributions based on which we obtain the optimal achievable code rate at the source node. We show that although a concatenation of priority encoded transmission (PET) with RLNC achieves the optimal code rate it incurs prohibitive high coding complexity. On the other hand batched sparse (BATS) code has been recently proposed for unicast networks which has low coding complexity with near-optimal overhead. However the existing BATS code design cannot be applied for multicast networks with heterogeneous channel rank distributions at different destinations. To this end we propose a novel expanding window BATS (EW-BATS) code where the input symbols are grouped into overlapped windows according to their importance levels. The more important symbols are encoded with lower rate and hence they can be decoded by more destinations while the less important symbols are encoded with higher rate and are only decoded by the destinations with high throughput for video quality enhancement. Based on asymptotical performance analysis we formulate the linear optimization problems to jointly optimize the degree distributions for each window and the window selection probabilities. Simulation results show that the proposed EW-BATS code satisfies the decoding requirements with much lower transmission overhead compared with separate BATS code where the degree distributions are separately optimized for each destination.

Video surveillance systems-current status and future trends

Within this survey an attempt is made to document the present status of video surveillance systems. The main components of a surveillance system are presented and studied thoroughly. Algorithms for image enhancement, object detection, object tracking, object recognition and item re-identification are presented. The most common modalities utilized by surveillance systems are discussed, putting emphasis on video, in terms of available resolutions and new imaging approaches, like High Dynamic Range video. The most important features and analytics are presented, along with the most common approaches for image / video quality enhancement. Distributed computational infrastructures are discussed (Cloud, Fog and Edge Computing), describing the advantages and disadvantages of each approach. The most important deep learning algorithms are presented, along with the smart analytics that they utilize. Augmented reality and the role it can play to a surveillance system is reported, just before discussing the challenges and the future trends of surveillance.

Multimedia Relay Resource Allocation for Energy Efficient Wireless Networks: High-Layer Content Prioritization With Low-Layer Diversity Cooperation

With the popularity of versatile mobile multimedia applications and the pressing need to reduce energy consumption in future wireless networks, new challenges have been posed to maintain adequate coverage, quality of service, and reliability for big-scale multimedia traffic. Specifically, in this paper we introduce a cooperative multimedia relay framework and investigate premium-regular-diversity-based multimedia resource allocation treatment approaches. We also comparatively study five generic wireless multimedia relay scenarios including direct transmission, single relay, multiple pure relays, relay cooperation with space-time coding, and relay coordination with distributed beamforming. The key contribution of the new relay resource allocation framework is that packet importance priority diversity at the application layer is jointly considered with spatial diversity and relay coordination protocols at lower layers. Premium packets and regular packets of multimedia streams are adaptively allocated to relay nodes, and transmission resources are optimally allocated with regards to energy budget. Extensive simulation results demonstrate that the proposed resource allocation paradigm using multirelay coordination has significant video quality enhancement and energy saving potentials to support future big-data wireless multimedia traffic.

Energy-efficient joint video encoding and transmission framework for WVSN

In this paper, we propose an energy-efficient joint video encoding and transmission framework for network lifetime extension, under an end-to-end video quality constraint in the Wireless Video Sensor Networks (WVSN). This framework integrates an energy-efficient and adaptive intra-only video encoding scheme based on the H.264/AVC standard, that outputs two service differentiated macroblocks categories, namely the Region Of Interest and the Background. Empirical models describing the physical behavior of the measured energies and distortions, during the video encoding and transmission phases, are derived. These models enable the video source node to dynamically adapt its video encoder’s configuration in order to meet the desired quality, while extending the network lifetime. An energy-efficient and reliable multipath multi-priority routing protocol is proposed to route the encoded streams to the sink, while considering the remaining energy, the congestion level as well as the packet loss rates of the intermediate nodes. Moreover, this protocol interacts with the application layer in order to bypass congestion situations and continuously feed it with current statistics. Through extensive numerical simulations, we demonstrate that the proposed framework does not only extend the video sensor lifetime by 54%, but it also performs significant end-to-end video quality enhancement of 35% in terms of Mean Squared Error (MSE) measurement.

Quality enhancement of video on demand implementation in peer-to-peer networks by optimum chunk length in the BitTorrent

BitTorrent in efficiency content distribution is a major concern efficiency among the researchers of this field with respect to streaming video on demand (VoD) production. BitTorrent is not appropriate for real-time applications; therefore, in order to apply it in VoD it should go through the necessary changes. Most of the available studies have greatly focused on changes in methods regarding chunk and peer selections method regarding BitTorrent, which proposed methods have improved the quality of VoD to a certain degree, while, the effect of chunk size on quality of video has been of less concern among them. Noting the fact that the buffer is used on VoD, the specified time for filling the buffer would allow for appropriate management of the chunk’s length. The Bit error rate and the time overhead of the operating algorithm parameters, somehow effect the chunk size. Because of bit error rate, the probability of correctly received chunks with great length is much less, that is, offering shorter pieces, while these pieces would lead to formation of more pieces in a buffer. The results indicate that a specific amount of time is required for obtaining the buffer’s content, and it must be dividable into more chunks. Running algorithms for each chunk generate a greater overhead which would result in of the QoS reduction. This overhead would make the bigger pieces perform better. As for the opposite impacts of these two parameters on the chunk size, in this article the optimal length of the chunks is found by considering both the effective characteristic. This optimal length is an established balance between the correctly received chunks’ rate and the greater rate of the buffer context obtained in a specified time.

Error resilient video transmission in ad hoc networks using layered and multiple description coding

The mesh structure of ad hoc networks, provides the possibility of establishing two disjoint paths from a sender to a receiver. Transmission of video over such networks due to their unpredictability and difficulty in securing reliable channels is challenging. Layered Coding (LC) and Multiple Description Coding (MDC) are two different techniques which can benefit from path diversity for robust video communication and also to adapt with preferences of users\network. This paper presents an approach to provide error resilient video transmission over a variety of network conditions and applications needs using combined LC and MDC schemes. In the proposed method two descriptions of each layer are generated in the FMO format of the H.264/AVC standard. Unlike the conventional approaches, in our work macroblocks of each layer are divided into two paths. Hence, in the bursty error conditions the error will be smoothly spread in all layers. For better protection and more network compatibility, the base layer is data partitioned and its important part (DP_A) is repeated in both paths. Simulation results show transmission of duplicated DP_A and the non-corresponding descriptions of two layers together on disjoint paths, can improve the error concealment of the decoder and consequently enhance video quality by up to 2 dB.