Video Enhancement Algorithm Research Articles

Deep Spatio-Temporal Network for Low-SNR Cryo-EM Movie Frame Enhancement.

Cryo-EM in single particle analysis is known to have low SNR and requires to utilize several frames of the same particle sample to restore one high-quality image for visualizing that particle. However, the low SNR of cryo-EM movie and motion caused by beam striking make the task very challenging. Video enhancement algorithms in computer vision shed new light on tackling such tasks by utilizing deep neural networks. However, they are designed for natural images with high SNR. Meanwhile, the lack of ground truth in cryo-EM movie seems to be one major limiting factor of the progress. Hence, we present a synthetic cryo-EM movie generation pipeline, which can produce realistic diverse cryo-EM movie datasets with low-SNR movie frames and multiple ground truth values. Then we propose a deep spatio-temporal network (DST-Net) for cryo-EM movie frame enhancement trained on our synthetic data. Spatial and temporal features are first extracted from each frame. Spatio-temporal fusion and high-resolution re-constructor are designed to obtain the enhanced output. For evaluation, we train our model on seven synthetic cryo-EM movie datasets and infer on real cryo-EM data. The experimental results show that DST-Net can achieve better enhancement performance both quantitatively and qualitatively compared with others.

Quality enhancement of compressed screen content video by cross-frame information fusion

In recent years, with the rise of various online learning platforms and game live broadcast industries, screen content video, a special type of video, is gradually emerging, and its traffic on the Internet is also increasing. Therefore, how to effectively enhance the quality of the screen content video has become an urgent problem to be solved. There exist a few successful compressed video enhancement algorithms. However, since there are a large number of areas with similar colors in the compressed screen content video, the traditional algorithms based on optical flow and deformable convolution cannot align the screen content video frames well. Specifically, for screen content videos containing animations and games, we propose a screen content video quality enhancement network based on the cross-fusion of multi-frame information. It includes a feature extraction module, a feature fusion module, an edge detail recovery module, and a reconstruction module. Our main contribution is the alignment-free quality enhancement framework based on cross-frame information fusion instead of traditional alignment based approaches. Through our experiments, the best results have been achieved on 13 screen content videos containing animations and games compressed by the SCC branch of HEVC/H.265.

Detail Preserving Low Illumination Image and Video Enhancement Algorithm Based on Dark Channel Prior.

In low illumination situations, insufficient light in the monitoring device results in poor visibility of effective information, which cannot meet practical applications. To overcome the above problems, a detail preserving low illumination video image enhancement algorithm based on dark channel prior is proposed in this paper. First, a dark channel refinement method is proposed, which is defined by imposing a structure prior to the initial dark channel to improve the image brightness. Second, an anisotropic guided filter (AnisGF) is used to refine the transmission, which preserves the edges of the image. Finally, a detail enhancement algorithm is proposed to avoid the problem of insufficient detail in the initial enhancement image. To avoid video flicker, the next video frames are enhanced based on the brightness of the first enhanced frame. Qualitative and quantitative analysis shows that the proposed algorithm is superior to the contrast algorithm, in which the proposed algorithm ranks first in average gradient, edge intensity, contrast, and patch-based contrast quality index. It can be effectively applied to the enhancement of surveillance video images and for wider computer vision applications.

Spatio-temporal deep learning model for distortion classification in laparoscopic video

Background: Laparoscopy is a surgery performed in the abdomen without making large incisions in the skin and with the aid of a video camera, resulting in laparoscopic videos. The laparoscopic video is prone to various distortions such as noise, smoke, uneven illumination, defocus blur, and motion blur. One of the main components in the feedback loop of video enhancement systems is distortion identification, which automatically classifies the distortions affecting the videos and selects the video enhancement algorithm accordingly. This paper aims to address the laparoscopic video distortion identification problem by developing fast and accurate multi-label distortion classification using a deep learning model. Current deep learning solutions based on convolutional neural networks (CNNs) can address laparoscopic video distortion classification, but they learn only spatial information. Methods: In this paper, utilization of both spatial and temporal features in a CNN-long short-term memory (CNN-LSTM) model is proposed as a novel solution to enhance the classification. First, pre-trained ResNet50 CNN was used to extract spatial features from each video frame by transferring representation from large-scale natural images to laparoscopic images. Next, LSTM was utilized to consider the temporal relation between the features extracted from the laparoscopic video frames to produce multi-label categories. A novel laparoscopic video dataset proposed in the ICIP2020 challenge was used for training and evaluation of the proposed method. Results: The experiments conducted show that the proposed CNN-LSTM outperforms the existing solutions in terms of accuracy (85%), and F1-score (94.2%). Additionally, the proposed distortion identification model is able to run in real-time with low inference time (0.15 sec). Conclusions: The proposed CNN-LSTM model is a feasible solution to be utilized in laparoscopic videos for distortion identification.

A Low-Cost and High-Throughput FPGA Implementation of the Retinex Algorithm for Real-Time Video Enhancement

For video applications in a special environment such as medical imaging, space exploration, and underwater exploration, the video captured by an image sensor is often deteriorated because of low lighting conditions. Therefore, it is necessary to enhance the part of the image that is too dark to distinguish details while maintaining the remaining part with the same brightness. The retinex algorithm is widely used to restore naturalness of a video, especially exhibiting outstanding performance in the enhancement of a dark area. However, it demands large computational complexity because of its intricate structure, such as the Gaussian filter and exponentiation operations, and consequently, it is difficult to process in real time. This article presents a low-cost and high-throughput design of the retinex video enhancement algorithm. The hardware (HW) design is implemented using a field-programmable gate array (FPGA), and it supports a throughput of 60 frames/s for a $1920 \times 1080$ image with negligible latency. The proposed FPGA design minimizes HW resources while maintaining the quality and the performance by using a small line buffer instead of a frame buffer, by applying the concept of approximate computing for the complex Gaussian filter, and by designing a new and nontrivial exponentiation operation. The proposed design makes it possible to significantly reduce HW resources (up to 79.22% of total resources) compared to existing systems and is compatible with commercialized devices through the standard HDMI/DVI video ports.

Multi-scale Guided Image and Video Fusion: A Fast and Efficient Approach

In this paper, we propose a general purpose, simple and fast fusion algorithm based on guided image filter. The proposed method can well combine useful source image information into the fused image supported by multi-scale image decomposition, structure transferring property, visual saliency detection, and weight map construction. Multi-scale image decomposition is appropriate to represent and manipulate image features at various scales. Structure transferring property enabled by our algorithm can induce structures of one source image into the other. A new visual saliency detection based on guided image filter introduced in this paper is able to extract significant regions from visually different images of the same scene. The choice of weight maps helped to integrate the complementary information pixel by pixel at each scale. Experimental outcomes of the proposed method are compared and analyzed with traditional and recent guided image filter-based fusion algorithms in terms of visual quality, fusion metrics and run time. In addition, to enhance fusion results further we made an effort to find a suitable image and video enhancement algorithm. The fusion performance analysis clearly indicates that the proposed method is very promising along with less run time.

Video Enhancement with Task-Oriented Flow

Many video enhancement algorithms rely on optical flow to register frames in a video sequence. Precise flow estimation is however intractable; and optical flow itself is often a sub-optimal representation for particular video processing tasks. In this paper, we propose task-oriented flow (TOFlow), a motion representation learned in a self-supervised, task-specific manner. We design a neural network with a trainable motion estimation component and a video processing component, and train them jointly to learn the task-oriented flow. For evaluation, we build Vimeo-90K, a large-scale, high-quality video dataset for low-level video processing. TOFlow outperforms traditional optical flow on standard benchmarks as well as our Vimeo-90K dataset in three video processing tasks: frame interpolation, video denoising/deblocking, and video super-resolution.

Video-Based Evidence Analysis and Extraction in Digital Forensic Investigation

As a result of the popularity of smart mobile devices and the low cost of surveillance systems, visual data are increasingly being used in digital forensic investigation. Digital videos have been widely used as key evidence sources in evidence identification, analysis, presentation, and report. The main goal of this paper is to develop advanced forensic video analysis techniques to assist the forensic investigation. We first propose a forensic video analysis framework that employs an efficient video/image enhancing algorithm for the low quality of footage analysis. An adaptive video enhancement algorithm based on contrast limited adaptive histogram equalization (CLAHE) is introduced to improve the closed-circuit television (CCTV) footage quality for the use of digital forensic investigation. To assist the video-based forensic analysis, a deep-learning-based object detection and tracking algorithm are proposed that can detect and identify potential suspects and tools from footages.

Video Enhancement for Medical and Surveillance Applications

Live video laparoscopic surgery and visual surveillance has been an important research in the past few years. Laparoscopic surgery is a modern video tracking surgical technique in which operations are performed distantly from their location through small incisions (usually 0.5–1.5 cm) in a different place in the body. The visual surveillance system starts with motion detection or tracking. This motion object detection method, attempts to locate connected regions that define, relate the moving objects within the scene; like frame-to-frame difference, background subtraction and motion analysis, the object is detected and tracked from an enhanced video. For these we have defined two different approaches. First one is the Laparoscope Surgery video and Surveillance video Enhancement (LSSVE) algorithm and another algorithm used for only surveillance application background subtraction algorithm. The main goal of the video enhancement algorithm is many objects or activities of interest occur in a blood vessel and dark environment which cannot be seen easily without enhancement. For this development, we follow three different methods, In the second algorithm of background subtraction using dynamic threshold and a mixture of Gaussian three different methods are used effectively for object detection and compared their basis of performance on the accessible tacking the accurate location and detection. In laparoscope surgery, a camera and light provide feedback to the surgeon, who sees the enlarged and video enhanced surgical elements on a TV monitor. The video surveillance system does frame differences, after the object foreground detection, the parameters like speed, velocity and motion are determined. Keywords: Background modelling, background subtraction, live video laparoscopic surgery, LSSVE algorithm, object tracking, visual surveillance.

A New Efficient Illumination Compensation Algorithm for Enhancing Nighttime Videos

Since many activities of interest often occur in the dark environment, nighttime video enhancement is important for video surveillance. In this paper, we propose a novel and effective nighttime video enhancement algorithm for video surveillance applications by using illumination compensation, which fuses video frames from high quality daytime background and low quality nighttime video. For further improving the perceptual quality of the moving objects, an algorithm based on object region ratio average is also proposed. Experimental results show that the proposed algorithm can obviously improve the visual quality from the conventional methods.

국방용 감시카메라를 위한 적응적 영상화질 개선 알고리즘

국경 GOP(General Out Post)나 해안선 지역에 설치되어 운용되는 국방용 감시카메라는 급변하는 기상 및 조명환경에 의해 영상의 화질이 왜곡되고 열화되는 현상이 빈번하게 발생한다. 본 논문에서는 이러한 기상 및 조명환경 변화에 적응적인 화질개선 알고리즘을 제안한다. 기존 화질개선 알고리즘은 급변하는 환경조건에서 영상에 따라 극심한 성능편차를 보이는 문제점이 있는데, 이를 해결하기 위해 레티넥스(Retinex) 모델을 근간으로 환경변화에 적응적인 보정곡선을 이용하여 안개 및 저조도 환경에서 영상의 가시성을 향상시켜 높은 대비와 색상을 자연스럽게 재현하였고 실시간 환경변화에 적응토록 하였다. 또한, HSV 색모델을 가중 혼합하여 색 항상성 (Color Constancy)을 유지시켰으며, 개선과정 중 잡음(Noise)을 제거하여 보다 선명한 영상을 출력토록 하였다. 제안 알고리즘은 실험을 통해 기존 알고리즘 대비 주관적 평가인 MOS 1단계 향상효과 및 객관적 평가인 PSNR 15% 성능향상의 우수성을 입증하였다. 향후 국방감시 카메라 및 시스템에 적용되어 GOP나 해안선 지역의 열악한 기상조건으로부터 열화된 영상을 개선하여 적 침투 및 경계감시 식별에 도움을 주어 시스템의 신뢰성 향상에 기여할 것으로 기대한다. Surveillance cameras in national border and coastline area often occur the video distortion because of rapidly changing weather and light environments. It is positively necessary to enhance the distorted video quality for keeping surveillance. In this paper, we propose an adaptive video enhancement algorithm in the various environment changes. To solve an unstable performance problem of the existing method, the proposed method is based on Retinex algorithm and uses enhanced curves which is adapted in foggy and low-light conditions. In addition, we mixture the weighted HSV color model to keep color constancy and reduce noise to obtain clear images. As a results, the proposed algorithm improves the performance of well-balanced contrast enhancement and effective color restoration without any quality loss compared with the existing algorithm. We expect that this method will be used in surveillance camera systems and offer help of national defence with reliability.

A flexible FPGA implementation for illuminance–reflectance video enhancement

Digital cameras, new generation phones, commercial TV sets and, in general, all modern devices for image acquisition and visualization can benefit from algorithms for image enhancement suitable to work in real time and preferably with limited power consumption. Among the various methods described in the scientific literature, Retinex-based approaches are able to provide very good performances, but unfortunately they typically require a high computational effort. In this article, we propose a flexible and effective architecture for the real-time enhancement of video frames, suitable to be implemented in a single FPGA device. The video enhancement algorithm is based on a modified version of the Retinex approach. This method, developed to control the dynamic range of poorly illuminated images while preserving the visual details, has been improved by the adoption of a new model to perform illuminance estimation. The video enhancement parameters are controlled in real time through an embedded microprocessor which makes the system able to modify its behavior according to the characteristics of the input images, and using information about the surrounding light conditions.

Repairing imperfect video enhancement algorithms using classification-based trained filters

There are numerous video processing algorithms and modules available. When the algorithms are not optimally tuned, undesired results may happen in the processed video signals, e.g. blurring, overshoots/downshoots, loss of details and aliasing. When the video processing modules are fixed, e.g. when the modules are implemented in hardware/chips, it is highly desirable to repair those unpleasant effects caused by certain imperfect algorithms. In this paper, we propose a solution based on classification and least squares trained filters to repair/patch low-quality video processing modules at the back end of a video chain. Extensive experiments show that the repairing method can significantly improve the video quality without modifying the original processing modules.

Image-based fusion for video enhancement of night-time surveillance

In this paper, a novel image-based fusion video enhancement algorithm is proposed for night-time video surveillance applications by a combination of illumination fusion and based on moving objects fusion. The proposed algorithm fuses video frames from high quality day-time and night-time background with low quality night-time videos. For improving the perception quality of the moving objects, based on moving objects of region fusion method is proposed. Experimental results show the effectiveness of the proposed algorithm.

Depth Video Enhancement for Haptic Interaction Using a Smooth Surface Reconstruction

We present a depth video enhancement algorithm in order to provide high quality haptic interaction. As the telecommunication technology emerges rapidly, the depth image-based haptic interaction is becoming viable for broadcasting applications. Since a real depth map usually contains discrete and rugged noise, its haptic interaction produces the distorted force feedback. To resolve these problems, we propose a two-step refinement and adaptive sampling algorithm. In the first step, noise is removed by the median-filtering technique in 2D image space. Since not all pixels can be used to reconstruct the 3D mesh due to limited system resources, the filtered map is adaptively sampled based on the depth variation. Sampled 2D pixels, called feature points, are triangulated and projected onto 3D space. In the second refinement step, we apply the Gaussian smoothing technique to the reconstructed 3D surface. Finally, 3D surfaces are rendered to compute a smooth depth map from Z-buffer.

Image feature and noise detection based on statistical hypothesis tests and their applications in noise reduction

In many video processing applications in the field of consumer electronics such as digital TV, it is well understood that the presence of a noise limits the performance of video enhancement functions due to the time-varying characteristics of the noise. The basic difficulty is that the noise and the signal are difficult to be distinguished. This paper proposes image feature and noise detection algorithms, which effectively distinguish the noise from the image feature or vice versa. Specifically, the proposed algorithms provide a way of measuring the degree of noise with respect to the degree of image feature. The fundamental idea behind the proposed algorithms is to derive a statistical measure to estimate the fact that a noise has a random characteristic whereas an image feature has a spatial correlation among the associated neighbor samples. With the proposed algorithms, many video enhancement algorithms such as noise reduction or sharpness enhancement can be adaptively performed although a time varying noise is presented.

A new enhancement method for digital video applications

A new approach for digital video enhancement is proposed in order to achieve optimal picture quality for MPEG encoded video. It takes account of the special characteristics of the encoded video by defining a usefulness metric for enhancement (UME). This metric is calculated based on coding information and local spatial features to quantify how much a pixel can be enhanced without boosting coding artifacts. Motion estimation and scene change detection are integrated in the proposed algorithm to improve the temporal consistency. The idea proposed here can be applied to improve the performance of video enhancement algorithms for digital compressed sources. To demonstrate the potential of the new approach, the UME is applied to a spatial-domain sharpness enhancement algorithm. Experimental results show significant improvement with respect to the algorithm without the UME for various video sequences containing coding artifacts. This new approach can be incorporated into future digital video products (e.g. DVD recorders, set-top boxes, digital TVs, etc) in order to achieve superior picture quality for encoded video.