Video Enhancement Research Articles

Paper] A Cache Decision Policy for QoE Enhancement of Video and Audio Transmission over ICN/CCN

Paper] A Cache Decision Policy for QoE Enhancement of Video and Audio Transmission over ICN/CCN

Multi-Frame Quality Recovery Model for Compressed Video Enhancement

Multi-Frame Quality Recovery Model for Compressed Video Enhancement

A real-time interactive restoration system for intraoral digital videos using segment anything model.

Poor conditions in the intraoral environment often lead to low-quality photos and videos, hindering further clinical diagnosis. To restore these digital records, this study proposes a real-time interactive restoration system using segment anything model. Intraoral digital videos, obtained from the vident-lab dataset through an intraoral camera, serve as the input for interactive restoration system. The initial phase employs an interactive segmentation module leveraging segment anything model. Subsequently, a real-time intraframe restoration module and a video enhancement module were designed. A series of ablation studies were systematically conducted to illustrate the superior design of interactive restoration system. Our quantitative evaluation criteria contain restoration quality, segmentation accuracy, and processing speed. Furthermore, the clinical applicability of the processed videos was evaluated by experts. Extensive experiments demonstrated its performance on segmentation with a mean intersection-over-union of 0.977. On video restoration, it leads to reliable performances with peak signal-to-noise ratio of 37.09 and structural similarity index measure of 0.961, respectively. More visualization results are shown on the https://yogurtsam.github.io/iveproject page. Interactive restoration system demonstrates its potential to serve patients and dentists with reliable and controllable intraoral video restoration.

DeepStream: Video Streaming Enhancements using Compressed Deep Neural Networks

InIn HTTP Adaptive Streaming (HAS), each video is divided into smaller segments, and each segment is encoded at multiple pre-defined bitrates to construct a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">bitrate ladder</i> . To optimize bitrate ladders, per-title encoding approaches encode each segment at various bitrates and resolutions to determine the convex hull. From the convex hull, an optimized bitrate ladder is constructed, resulting in an increased Quality of Experience (QoE) for end-users. With the ever-increasing efficiency of deep learning-based video enhancement approaches, they are more and more employed at the client-side to increase the QoE, specifically when GPU capabilities are available. Therefore, scalable approaches are needed to support end-user devices with both CPU and GPU capabilities (denoted as CPU-only and GPU-available end-users, respectively) as a new dimension of a bitrate ladder. To address this need, we propose <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DeepStream</i> , a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">scalable content-aware</i> per-title encoding approach to support both CPU-only and GPU-available end-users. ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i</i> ) To support <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">backward compatibility</i> , <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DeepStream</i> constructs a bitrate ladder based on any existing per-title encoding approach. Therefore, the video content will be provided for legacy end-user devices with CPU-only capabilities as a base layer (BL). ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ii</i> ) For high-end end-user devices with GPU capabilities, an enhancement layer (EL) is added on top of the base layer comprising lightweight video super-resolution deep neural networks (DNNs) for each bitrate-resolution pair of the bitrate ladder. A content-aware video super-resolution approach leads to higher video quality, however, at the cost of bitrate overhead. To reduce the bitrate overhead for streaming content-aware video super-resolution DNNs, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DeepCABAC</i> , context-adaptive binary arithmetic coding for DNN compression, is used. Furthermore, the similarity among ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i</i> ) segments within a scene and ( <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ii</i> ) frames within a segment are used to reduce the training costs of DNNs. Experimental results show bitrate savings of 34% and 36% to maintain the same PSNR and VMAF, respectively, for GPU-available end-users, while the CPU-only users get the desired video content as usual.

ПОКРАЩЕННЯ ВІДЕОПОСЛІДОВНОСТІ В СИСТЕМАХ ВІДЕОАНАЛІТИКИ

A video enhancement method based on applying gamma correction is proposed. Its realization implies processing all video frames using a single algorithm. The proposed algorithm, due to the preservation of inter-frame coherence throughout the entire video, significantly reduces the time spent searching for the optimal value of the gamma parameter, which ensures the highest quality of the frame reaching the maximum possible level in the automatic mode in the sense of both visual observation and the determination of key points and the selection of contours of objects in images. The method is characterized by high adaptability to sudden changes in the lighting of the scene, preservation of interframe coherence, and the absence of any side negative artifacts in the enhanced video. A toolkit for automatically determining the optimal value of the gamma parameter for video frames is developed. It significantly increases the efficiency of video analytics systems, image and video segmentation and processing processes due to reducing the negative impact of the scene lighting mode on image quality. Keywords: gamma-correction, video analytics system, video sequence enhancement, histogram, cumulative histogram, video processing, interframe coherence.

Adaptive Locally-Aligned Transformer for low-light video enhancement

Low-light enhancement is a crucial task that aims to enhance the under-exposed input in computer vision. While state-of-the-art static single-image enhancement methods have made remarkable progress, yet, few attempts are explored the spatial-temporal sequence problem in low-light video enhancement. In this paper, we propose a simple yet highly effective method, termed as Adaptive Locally-Aligned Transformer (ALAT) for low-light video enhancement based on visual transformers. ALAT consists of three parts: feature encoder, locally-aligned transformer block (LATB) and pyramid feature decoder. Specifically, the transformer block enables the network to model the long-range spatial and appearance dependencies in videos due to its self-attention parallel computing mechanism. However, different from some previous approaches directly using the vanilla transformer, we consider that locality is significant in low-level vision tasks since the misaligned contextual local features (i.e., edges, shapes) may affect the prediction quality. Therefore, the proposed LATB is designed to align the video pixel with its most relevant ones adaptively in the local region to preserve the regional content information. Furthermore, we publish a new real-world low-light video dataset, named ExpressWay, to fill the gaps in the lack of dynamic low-light video scenarios, which contains high-quality videos with moving objects in both dark- and bright-light conditions. We conduct experiments on five benchmarks under three comprehensive settings including synthesized, static and our proposed dynamic low-light video datasets. Extensive experimental results show that our ALAT can outperform the previous state-of-the-arts by a large margin of 0.20∼1.10dB. Our method can be also extended to other video enhancement applications. The project is available at https://github.com/y1wencao/LLVE-ALAT.

Learning Degradation-Robust Spatiotemporal Frequency-Transformer for Video Super-Resolution.

Video Super-Resolution (VSR) aims to restore high-resolution (HR) videos from low-resolution (LR) videos. Existing VSR techniques usually recover HR frames by extracting pertinent textures from nearby frames with known degradation processes. Despite significant progress, grand challenges remain to effectively extract and transmit high-quality textures from high-degraded low-quality sequences, such as blur, additive noises, and compression artifacts. This work proposes a novel degradation-robust Frequency-Transformer (FTVSR++) for handling low-quality videos that carry out self-attention in a combined space-time-frequency domain. First, video frames are split into patches and each patch is transformed into spectral maps in which each channel represents a frequency band. It permits a fine-grained self-attention on each frequency band so that real visual texture can be distinguished from artifacts. Second, a novel dual frequency attention (DFA) mechanism is proposed to capture the global and local frequency relations, which can handle different complicated degradation processes in real-world scenarios. Third, we explore different self-attention schemes for video processing in the frequency domain and discover that a "divided attention" which conducts joint space-frequency attention before applying temporal-frequency attention, leads to the best video enhancement quality. Extensive experiments on three widely-used VSR datasets show that FTVSR++ outperforms state-of-the-art methods on different low-quality videos with clear visual margins. Code and pre-trained models are available at https://github.com/researchmm/FTVSR.

Blind quality-based pairwise ranking of contrast changed color images using deep networks

Next-generation multimedia networks are expected to provide systems and applications with top Quality of Experience (QoE) to users. To this end, robust quality evaluation metrics are critical. Unfortunately, most current research focuses only on modeling and evaluating mainly distortions across the pipeline of multimedia networks. While distortions are important, it is also as important to consider the effects of enhancement and other manipulations of multimedia content, especially images and videos. In contrast to most existing works dedicated to evaluating image/video quality in its traditional context, very few research efforts have been devoted to Image Quality Enhancement Assessment (IQEA) and more specifically, Contrast Enhancement Evaluation (CEE). Our contribution fills this gap by proposing a pairwise ranking scheme for estimating and evaluating the perceptual quality of image contrast change (contrast enhancement and/or contrast-distorted images) process. We propose a novel Deep Learning-based Blind Quality pairwise Ranking scheme for Contrast-Changed (Deep-BQRCC) images. This method provides an automatic pairwise ranking of a set of contrast-changed images. The proposed framework is based on using a pair of Convolutional Neural Networks (CNN) together with a saliency-based attention model and a color-difference visual map. Extensive experiments were conducted to validate the effectiveness of the proposed workflow through an ablation analysis. Different combinations of CNN models and pooling strategies were analyzed. The proposed Deep-BQRCC approach was evaluated over three dedicated publicly available datasets. The experimental results showed an increase in performance within a range of 3–10% compared to state-of-the-art IQEA measures.

Wavelet energy-based adaptive retinex algorithm for low light mobile video enhancement

ABSTRACT Our paper presents an adaptive multiscale retinex algorithm and a new wavelet energy metric to improve low-light video captured on mobile devices. Initially, we extract RGB frames from the video and convert them to hue-saturation-value (HSV) format, preserving the hue channel to prevent common RGB colour shifting issues. Saturation channel enhancement is achieved through histogram equalization (HE), extending the dynamic range. The adaptive retinex algorithm enhances the value channel, quantified by our new wavelet energy metric. Combining the modified value and saturation channels improves the contrast of the reconstructed image. As a final step, we transform the HSV video back to RGB and restore naturalness using a modified colour restoration technique. The proposed approach has been tested on over 300 images and videos. It is evident from the experimental results presented that the proposed method lowers noise and halo artifacts more effectively than existing methods.

Endoscopic image classification algorithm based on Poolformer.

Image desmoking is a significant aspect of endoscopic image processing, effectively mitigating visual field obstructions without the need for additional surgical interventions. However, current smoke removal techniques tend to apply comprehensive video enhancement to all frames, encompassing both smoke-free and smoke-affected images, which not only escalates computational costs but also introduces potential noise during the enhancement of smoke-free images. In response to this challenge, this paper introduces an approach for classifying images that contain surgical smoke within endoscopic scenes. This classification method provides crucial target frame information for enhancing surgical smoke removal, improving the scientific robustness, and enhancing the real-time processing capabilities of image-based smoke removal method. The proposed endoscopic smoke image classification algorithm based on the improved Poolformer model, augments the model's capacity for endoscopic image feature extraction. This enhancement is achieved by transforming the Token Mixer within the encoder into a multi-branch structure akin to ConvNeXt, a pure convolutional neural network. Moreover, the conversion to a single-path topology during the prediction phase elevates processing speed. Experiments use the endoscopic dataset sourced from the Hamlyn Centre Laparoscopic/Endoscopic Video Dataset, augmented by Blender software rendering. The dataset comprises 3,800 training images and 1,200 test images, distributed in a 4:1 ratio of smoke-free to smoke-containing images. The outcomes affirm the superior performance of this paper's approach across multiple parameters. Comparative assessments against existing models, such as mobilenet_v3, efficientnet_b7, and ViT-B/16, substantiate that the proposed method excels in accuracy, sensitivity, and inference speed. Notably, when contrasted with the Poolformer_s12 network, the proposed method achieves a 2.3% enhancement in accuracy, an 8.2% boost in sensitivity, while incurring a mere 6.4 frames per second reduction in processing speed, maintaining 87 frames per second. The results authenticate the improved performance of the refined Poolformer model in endoscopic smoke image classification tasks. This advancement presents a lightweight yet effective solution for the automatic detection of smoke-containing images in endoscopy. This approach strikes a balance between the accuracy and real-time processing requirements of endoscopic image analysis, offering valuable insights for targeted desmoking process.

Spatio-Temporal Coherence of mmWave/THz Channel Characteristics and Their Forecasting Using Video Frame Prediction Techniques

Channel state information in millimeter wave (mmWave) and terahertz (THz) communications systems is vital for various tasks ranging from planning the optimal locations of BSs to efficient beam tracking mechanisms to handover design. Due to the use of large-scale phased antenna arrays and high sensitivity to environmental geometry and materials, precise propagation models for these bands are obtained via ray-tracing modeling. However, the propagation conditions in mmWave/THz systems may theoretically change at very small distances, that is, 1 mm–1 μm, which requires extreme computational effort for modeling. In this paper, we first will assess the effective correlation distances in mmWave/THz systems for different outdoor scenarios, user mobility patterns, and line-of-sight (LoS) and non-LoS (nLoS) conditions. As the metrics of interest, we utilize the angle of arrival/departure (AoA/AoD) and path loss of the first few strongest rays. Then, to reduce the computational efforts required for the ray-tracing procedure, we propose a methodology for the extrapolation and interpolation of these metrics based on the convolutional long short-term memory (ConvLSTM) model. The proposed methodology is based on a special representation of the channel state information in a form suitable for state-of-the-art video enhancement machine learning (ML) techniques, which allows for the use of their powerful prediction capabilities. To assess the prediction performance of the ConvLSTM model, we utilize precision and recall as the main metrics of interest. Our numerical results demonstrate that the channel state correlation in AoA/AoD parameters is preserved up until approximately 0.3–0.6 m, which is 300–600 times larger than the wavelength at 300 GHz. The use of a ConvLSTM model allows us to accurately predict AoA and AoD angles up to the 0.6 m distance with AoA being characterized by a higher mean squared error (MSE). Our results can be utilized to speed up ray-tracing simulations by selecting the grid step size, resulting in the desired trade-off between modeling accuracy and computational time. Additionally, it can also be utilized to improve beam tracking in mmWave/THz systems via a selection of the time step between beam realignment procedures.

Adaptive rule-based colour component weight assignment strategy for underwater video enhancement

ABSTRACT Images and videos collected in an underwater environment often have low contrast, blur, and colour cast due to two significant sources of distortion; light scattering and absorption. In an underwater image/video, suspended particles attenuate red and blue components more than green channels. This article presents two adaptive weight allocation strategies based on rule assignment for red, green, and blue channels. Firstly, an improved balanced contrast enhancement technique (IBCET) is proposed using an adaptive contrast enhancement scheme based on colour component weight assignment. Secondly, a modified fuzzy contrast enhancement technique which obtains the intensification factor based on the weight of each component is developed. Finally, principal component analysis fusion is employed to improve the overall colour and contrast of the output video frames. Qualitative and quantitative evaluations on the standard underwater video dataset are demonstrated to validate the improved performance of the proposed algorithm.

ALIVE: Adaptive-Chromaticity for Interactive Low-light Image and Video Enhancement

Ray tracing remains of interest to Computer Graphics community with its elegant framing of how light interacts with objects, being able to easily support multiple light sources, and simple framework of merging synthetic and real cameras. Recent trends to provide implementations at the chip-level means raytracing’s constant quest of realism would propel its usage in real-time applications. AR/VR, Animations, 3DGames Industry, 3D-large scale simulations, and future social computing platforms are just a few examples of possible major impact. Raytracing is also appealing to HCI community because raytracing extends well along the 3D-space and time, seamlessly blending both synthetic and real cameras at multiple scales to support storytelling. This presentation will include a few milestones from my work such as the Slicing Extent technique and Directed Safe Zones. Our recent applications of applying machine learning techniques creating novel synthetic views, which could also provide a future doorway to handle dynamic scenes with more compute power as needed, will also be presented. It is once again renaissance for ray tracing which for last 50+ years has remained the most elegant technique for modeling light phenomena in virtual worlds at whatever scale compute power could support.

Application of video image processing in sports action recognition based on particle swarm optimization algorithm

The existing sports training methods are mainly aimed at the sports field environment. The traditional sports training is only based on the coaches' visual inspection and combined with their own experience to put forward suggestions, which is relatively inefficient, thus limiting the rise of athletes' sports training level to a certain extent. Based on this background, combining traditional physical education teaching methods with video image processing technology, especially using particle swarm optimization algorithm, can promote the application of human motion recognition technology in physical training. This paper mainly investigates the optimization process of particle swarm optimization algorithm and discusses the development of particle swarm optimization algorithm; Secondly, through video decoding, image noise removal, video enhancement and other methods, complete video image processing and establish the structure of the manikin to achieve the collection of key points of the target, and then collect relevant data with experimental methods The results show that the motion recognition system proposed in this paper can effectively detect the changes of athletes' sampling point path, and can be compared with standard movements, which has a good auxiliary role. With the application of video image processing technology in sports training becoming more and more common, athletes can analyze their training videos in a more intuitive way and find out shortcomings, so as to improve the training effect. This paper studies particle swarm optimization algorithm and applies it to the field of video image processing, which promotes the development of sports action recognition technology based on video processing.

Low-Light Video Enhancement with Synthetic Event Guidance

Low-light video enhancement (LLVE) is an important yet challenging task with many applications such as photographing and autonomous driving. Unlike single image low-light enhancement, most LLVE methods utilize temporal information from adjacent frames to restore the color and remove the noise of the target frame. However, these algorithms, based on the framework of multi-frame alignment and enhancement, may produce multi-frame fusion artifacts when encountering extreme low light or fast motion. In this paper, inspired by the low latency and high dynamic range of events, we use synthetic events from multiple frames to guide the enhancement and restoration of low-light videos. Our method contains three stages: 1) event synthesis and enhancement, 2) event and image fusion, and 3) low-light enhancement. In this framework, we design two novel modules (event-image fusion transform and event-guided dual branch) for the second and third stages, respectively. Extensive experiments show that our method outperforms existing low-light video or single image enhancement approaches on both synthetic and real LLVE datasets. Our code will be available at https://gitee.com/mindspore/models/tree/master/research/cv/LLVE-SEG.

Spatio-temporal propagation and reconstruction for low-light video enhancement

Different from the single low-light image enhancement, the key for video enhancement is to improve temporal stability in spatial and temporal varying manner. However, most methods fail to effectively align images or features of adjacent frames to fuse spatio-temporal information and achieve limited enhancement performance, especially in dynamic videos with scene changes and moving targets. In this paper, we propose a spatio-temporal propagation and reconstruction (STPR) network to enable video representation learning for low-light video enhancement. Specifically, we develop a recursively supervised-based pyramid residual dense structure for extracting expressive context features from video sequences, the unique structural design enlarges the receptive field in multi-scale space with limited training parameters. Moreover, to strengthen the interaction of across-frame features, a feature propagation subnet is present to achieve precise alignment and effective fusion of spatio-temporal features. After obtaining the aggregated features of adjacent frames, the spatio-temporal feature reconstruction subnet exploits temporal dependence to improve the quality of target frames and enforce temporal consistency. Experimental results on several low-light video datasets demonstrate that our network outperforms state-of-the-art methods both in quantitative and visual comparisons.

Non-contact human respiratory rate measurement under dark environments by low-light video enhancement

A non-contact human respiratory rate measurement algorithm under dark environments is proposed in this study. The algorithm takes the low-light video as input, which includes four steps: time-domain preprocessing, space-domain preprocessing, image enhancement, and respiratory rate measurement. Firstly, the time-domain preprocessing and space-domain preprocessing run in turn to filter out the image noise and environment interference. Then, a Retinex-based image enhancement algorithm works for image enhancement. The human respiratory rate is measured at last. Experiments on a self-made dataset show that an average increase of 2.3 % in accuracy can be obtained by the proposed method. Specifically, high-precision measurement results can be achieved when the light intensity is not less than 200 lux and the detection distance is not far from 1.5 m.

A Survey on Compression Domain Image and Video Data Processing and Analysis Techniques

A tremendous amount of image and video data are being generated and shared in our daily lives. Image and video data are typically stored and transmitted in compressed form in order to reduce storage space and transmission time. The processing and analysis of compressed image and video data can greatly reduce input data size and eliminate the need for decompression and recompression, thereby achieving significant savings in memory and computation time. There exists a body of research on compression domain data processing and analysis. This survey focuses on the work related to image and video data. The papers cited are categorized based on their target applications, including image and video resizing and retrieval, information hiding and watermark embedding, image and video enhancement and segmentation, object and motion detection, as well as pattern classification, among several other applications. Key methods used for these applications are explained and discussed. Comparisons are drawn among similar approaches. We then point out possible directions of further research.

OVQE: Omniscient Network for Compressed Video Quality Enhancement

How to use information from temporal, spatial, and frequency domain dimensions is crucial for the quality enhancement of compressed video. The state-of-the-art methods generally design powerful networks to fuse the spatiotemporal information of the videos. But the spatiotemporal information of the entire video is not fully utilized and effectively fused, resulting in the learned context information that is not closely related to the target frame. In addition, various compressed videos have varying degrees of frequency domain information loss. The previous methods ignored the non-uniform distortion of compressed video in different frequency domains and did not design unique algorithms for different frequency domains, so the real texture details of the video could not be restored. In this paper, we propose an omniscient network, which learns video spatiotemporal and omni-frequency information more effectively. The omniscient network consists of two novel components: a Spatio-Temporal Feature Fusion (STFF) module and an Omni-Frequency Adaptive Enhancement (OFAE) block. The former aims to capture spatiotemporal information in adjacent frames, while the latter aims to adaptively recover different frequency domains of compressed video. The information is designed to be bidirectionally propagated in a grid manner such that the omni-enhanced results can be applied. Extensive experiments show that our method outperforms the state-of-the-art method in terms of objective metrics, subjective visual effects, and model complexity.

Real-time Image Enhancement with Attention Aggregation

Image enhancement has stimulated significant research works over the past years for its great application potential in video conferencing scenarios. Nevertheless, most existing image enhancement approaches are still struggling to find a good tradeoff that reduces the computational cost as much as possible while maintaining plausible result quality. Recently, curve-based mapping methods are proposed and have shown great potential for real-time and high-quality image enhancement of arbitrary resolutions. In this article, we take advantage of the curve-based mapping representation and focus on further improving the enhancement quality and robustness, while minimizing additional computational costs. Specifically, we (1) carefully re-formulate the curve function to improve learning stability, and (2) aggregate different semantic attention into the curve regression process, which can overcome the major problems of curve-based methods that generate moderate results with low contrast. The semantic attention is jointly learned with the supervision from class activation mapping of pre-trained feature extractors, thus reducing the manual annotation cost of semantic labels. Experiments have shown that our proposed method significantly improves curve-based methods both qualitatively and quantitatively, achieving visually plausible results compared with other deep neural network-based enhancement methods, and maintains a very low computational cost, i.e., taking 18.7 ms for a 360p image on a single P40 GPU. Extensive experiments demonstrate that our method is also capable of video enhancement tasks.