Up-sampling Method Research Articles

A GPU-based statistical image up-sampling method by using edge templates

Image up-sampling is very important in different fields, so an appropriate high-quality fast and efficient image up-sampling method is needed. Many interpolation-based up-sampling methods have been proposed by many researchers, but the quality of the resulting images is not satisfactory. The details of these images often cannot be accepted when we use them in many fields. On one hand, some of these methods are very fast, but produce images that are lacking many details and information of the original image; the others can produce high quality images, but the methods are very slow. In this paper, we propose a fast statistical image up-sampling method, and we use GPU to accelerate our up-sampling algorithm. We can obtain high quality images based on reducing the input resolution-grids dependency artefacts. And we can rebuild low resolution images' sharp edges fast and get high-quality up-sampled images in real time. We have applied this method in the multi-resolution texture generation of large scale terrain rendering. Experiments prove that our method can achieve ideal effects in real time.

Iterative projection reconstruction for fast and efficient image upsampling

With the development of ultra-high-resolution display devices, the visual perception of fine texture details is becoming increasingly important. Traditional image upsampling methods suffer from either loss of high-frequency texture details or very high time cost. In this paper, we propose an iterative projection reconstruction (IPR) method for fast and efficient image upsampling. The proposed method refines high-frequency texture details with an iterative projection process, and utilizes the pre-computed projection matrix to accelerate the example-based image reconstruction. As a result, the proposed method can reproduce fine texture details with low time cost. Experimental results demonstrate that the proposed method outperforms some state-of-the-art methods.

Depth Map Down-Sampling and Coding Based on Synthesized View Distortion

In this paper, we propose a depth map down-sampling and coding scheme that minimizes the view synthesis distortion. Moreover, a solution for the optimal depth map down-sampling problem that minimizes the depth-caused distortion in the virtual view by exploiting the depth map and the associated texture information along with the up-sampling method to be used in the decoder side is derived. Furthermore, to enhance compression performance, the synthesized view distortion, which is evaluated by emulating the interpolation and the virtual view synthesis process, is used in the optimization objective function for coding mode selection in the video encoder. Experimental results show that both the proposed depth map down-sampling and encoding methods lead to good performance, and the average bit rate reduction is 2.62 $\%$ compared with 3D-AVC.

Fast robust detection of edges in noisy depth images

Depth edges play an important role in depth image upsampling. Many recent upsampling methods rely on the prior images of depth edges to preserve sharp depth edges in restored depth images. However, recent depth edge detection methods are not robust against the noise in depth images. Some methods are also too time-consuming. We propose a method to efficiently detect edges in depth images. The proposed method is very simple but very robust against the noise in depth images. It is also fast and has near O(1) implementation. We apply the proposed method to the existing edge guided depth image upsampling. Experimental results on both simulated and real data show the effectiveness of the proposed method.

Unified Saliency Detection Model Using Color and Texture Features.

Saliency detection attracted attention of many researchers and had become a very active area of research. Recently, many saliency detection models have been proposed and achieved excellent performance in various fields. However, most of these models only consider low-level features. This paper proposes a novel saliency detection model using both color and texture features and incorporating higher-level priors. The SLIC superpixel algorithm is applied to form an over-segmentation of the image. Color saliency map and texture saliency map are calculated based on the region contrast method and adaptive weight. Higher-level priors including location prior and color prior are incorporated into the model to achieve a better performance and full resolution saliency map is obtained by using the up-sampling method. Experimental results on three datasets demonstrate that the proposed saliency detection model outperforms the state-of-the-art models.

A comparative performance analysis of DCT-based and Zernike moments-based image up-sampling techniques

The DCT-based image up-sampling methods are very effective and simple for image super-resolution in the DCT domain. We compare the performance of three major DCT-based image interpolation methods and propose Zernike moments (ZMs)-based up-sampling method. The ZMs-based up-sampling has many useful characteristics which have been described in this paper. Two of the main characteristics are as follows. First, it represents a set of large number of orthogonal basis functions which represent fine details in the image, thus preserving important image structures. Second, the ZMs coefficients represent the edge details in any orientation whereas the DCT encodes the edge details along horizontal and vertical directions. Detailed experimental results demonstrate the superior performance of the ZMs-based up-sampling approach as compared to the DCT-based approaches.

Overview of efficient high-quality state-of-the-art depth enhancement methods by thorough design space exploration

High-quality 3D content generation requires high-quality depth maps. In practice, depth maps generated by stereo-matching, depth sensing cameras, or decoders, have low resolution and suffer from unreliable estimates and noise. Therefore, depth enhancement is necessary. Depth enhancement comprises two stages: depth upsampling and temporal post-processing. In this paper, we extend our previous work on depth upsampling in two ways. First we propose PWAS-MCM, a new depth upsampling method, and we show that it achieves on average the highest depth accuracy compared to other efficient state-of-the-art depth upsampling methods. Then, we benchmark all relevant state-of-the-art filter-based temporal post-processing methods on depth accuracy by conducting a parameter space search to find the optimum set of parameters for various upscale factors and noise levels. Then we analyze the temporal post-processing methods qualitatively. Finally, we analyze the computational complexity of each depth upsampling and temporal post-processing method by measuring the throughput and hardware utilization of the GPU implementation that we built for each method.

Depth map upsampling with image decomposition

Two novel approaches for depth map upsampling are presented. A depth upsampling method using image decomposition is first proposed. Most of previous depth upsampling techniques have suggested the use of colour image as a guide. Unlike the conventional algorithms, the method decomposes the colour image into its structure and texture layers, and then makes use of the structure component instead of the colour image in the reconstruction of depth values. Furthermore, the structure information-based method is extended to a hybrid depth upsampling approach, which takes the advantages of both structure and colour maps. Experimental results demonstrate that the proposed depth map upsampling methods perform better than the previous algorithms in terms of the bad pixel rate.

Single Anisotropic 3-D MR Image Upsampling via Overcomplete Dictionary Trained From In-Plane High Resolution Slices.

In magnetic resonance (MR), hardware limitation, scanning time, and patient comfort often result in the acquisition of anisotropic 3-D MR images. Enhancing image resolution is desired but has been very challenging in medical image processing. Super resolution reconstruction based on sparse representation and overcomplete dictionary has been lately employed to address this problem; however, these methods require extra training sets, which may not be always available. This paper proposes a novel single anisotropic 3-D MR image upsampling method via sparse representation and overcomplete dictionary that is trained from in-plane high resolution slices to upsample in the out-of-plane dimensions. The proposed method, therefore, does not require extra training sets. Abundant experiments, conducted on simulated and clinical brain MR images, show that the proposed method is more accurate than classical interpolation. When compared to a recent upsampling method based on the nonlocal means approach, the proposed method did not show improved results at low upsampling factors with simulated images, but generated comparable results with much better computational efficiency in clinical cases. Therefore, the proposed approach can be efficiently implemented and routinely used to upsample MR images in the out-of-planes views for radiologic assessment and postacquisition processing.

가중치 결합 양방향 필터를 이용한 깊이 지도의 업샘플링 방법

A depth map is an image which contains 3D distance information. Generally, it is difficult to acquire a high resolution (HD), noise-removed, good quality depth map directly from the camera. Therefore, many researches have been focused on acquisition of the high resolution and the good quality depth map by up-sampling and pre/post image processing of the low resolution depth map. However, many researches are lack of effective up-sampling for the edge region which has huge impact on image perceptual-quality. In this paper, we propose an up-sampling method, based on joint bilateral filter, which improves up-sampling of the edge region and visual quality of synthetic images by adopting different weights for the edge parts that is sensitive to human perception characteristics. The proposed method has gains in terms of PSNR and subjective video quality compared to previous researches.

Fast Minimax Path-Based Joint Depth Interpolation

We propose a fast minimax path-based depth interpolation method. The algorithm computes for each target pixel varying contributions from reliable depth seeds, and weighted averaging is used to interpolate missing depths. Compared with state-of-the-art joint geodesic upsampling method which selects the $K$ nearest seeds to interpolate missing depths with $O(Kn)$ complexity, our method does not need to limit the number of seeds to $K$ and reduces the computational complexity to $O(n)$ . In addition, the minimax path chooses a path with the smallest maximum immediate pairwise pixel difference on it, so it tends to preserve sharp depth discontinuities better. In contrast to the results of previous depth upsampling algorithms, our approach can provide accurate depths with fewer artifacts.

WITHDRAWN: Joint depth map interpolation using minimax paths

We propose a minimax path-based depth interpolation method. The algorithm computes for each target pixel varying contributions from reliable depth seeds, and weighted averaging is used to interpolate missing depths. Compared with state-of-the-art joint geodesic upsampling method which selects the K nearest seeds to interpolate missing depths with O(Kn) complexity, our method does not need to limit the number of seeds to K and reduces the computational complexity to O(n). In addition, the minimax path chooses a path with the smallest maximum immediate pairwise pixel difference on it, so it tends to preserve sharp depth discontinuities better. In contrast to the results of previous depth upsampling algorithms, our approach provides accurate depths with fewer artifacts.

Notice of Violation of IEEE Publication Principles: Image Guided Depth Map Upsampling using Anisotropic TV- ${L^2}$

Notice of Violation of IEEE Publication Principles “Image Guided Depth Map Upsampling using Anisotropic TV-L2” by Xiaozhou Zhu, Xin Song, and Xiaoqian Chen in the IEEE Signal Processing Letters, Vol 22, Issue 3, March 2015, pp. 318-321 After careful and considered review of the content and authorship of this paper by a duly constituted expert committee, this paper has been found to be in violation of IEEE's Publication Principles. This paper contains significant portions of text from the paper cited below that were reused without attribution. “Image Guided Depth Upsampling using Anisotropic Total Generalized Variation” by David Ferstl, Christian Reinbacher, Rene Ranftl, Matthias Ruether, and Horst Bischof in the Proceedings of the IEEE International Conference on Computer Vision, December 2013, pp. 993-1000 In this letter, we present a novel upsampling method to enhance the spatial resolution of depth maps. Given a low resolution depth map from a depth sensor and a registered high resolution intensity image from a camera, we formulate it as a convex optimization problem and solve it using the first-order primal-dual algorithm. The formulation combines an ${L^2}$ data term and an anisotropic total variation (TV) regularization term, thus it is more robust to noise and better preserves the fine details. Experimental results on Middlebury and real world datasets demonstrate that our method achieves favorable performance compared with state-of-the-art upsampling methods.

Multi-level Modified Finite Radon Transform Network for Image Upsampling

A local line-like feature is the most important discriminate information in the image upsampling scenario. In recent example-based upsampling methods, grayscale and gradient features are often adopted to describe the local patches, but these simple features cannot accurately characterize complex patches. In this paper, we present a feature representation of local edges by means of a multilevel filtering network, namely, multilevel modified finite Radon transform network (MMFRTN). In the proposed MMFRTN, the MFRT is utilized in the filtering layer to extract the local line-like feature; the nonlinear layer is set to be a simple local binary process; for the feature-pooling layer, we concatenate the mapped patches as the feature of local patch. Then, we propose a new example-based upsampling method by means of the MMFRTN feature. Experimental results demonstrate the effectiveness of the proposed method over some state-of-the-art methods.

Reliability-Based Multiview Depth Enhancement Considering Interview Coherence

Color-plus-depth video format has been increasingly popular in 3-D video applications, such as auto-stereoscopic 3-D TV and freeview TV. The performance of these applications is heavily dependent on the quality of depth maps since intermediate views are synthesized using the corresponding depth maps. This paper presents a novel framework for obtaining high-quality multiview color-plus-depth video using a hybrid sensor, which consists of multiple color cameras and depth sensors. Given multiple high-resolution color images and low quality depth maps obtained from the color cameras and depth sensors, we improve the quality of the depth map corresponding to each color view by increasing its spatial resolution and enforcing interview coherence. Specifically, a new up-sampling method considering the interview coherence is proposed to enhance multiview depth maps. This approach can improve the performance of the existing up-sampling algorithms, such as joint bilateral up-sampling and weighted mode filtering, which have been developed to enhance a single-view depth map only. In addition, an adaptive approach of fusing multiple input low-resolution depth maps is proposed based on the reliability that considers camera geometry and depth validity. The proposed framework can be extended into the temporal domain for temporally consistent depth maps. Experimental results demonstrate that the proposed method provides better multiview depth quality than the conventional single-view-based methods. We also show that it provides comparable results, yet much more efficiently, to other fusion approaches that employ both depth sensors and stereo matching algorithm together. Moreover, it is shown that the proposed method significantly reduces bit rates required to compress the multiview color-plus-depth video.

Adaptive Support of Spatial–Temporal Neighbors for Depth Map Sequence Up-sampling

Depth map up-sampling methods have achieved remarkable improvement by exploiting sensor fusion techniques where they assume that the depth map discontinuities and image edges coincide, and the depth values of the temporal neighbors are stable during time variation. However, inherent noise of depth data acquired by active range sensors often violates these assumptions, and results in undesirable error propagation. To alleviate the error propagation, this letter presents a new adaptive supporting method of spatially-temporally neighboring samples. On the basis of a spatial-temporal Markov random field model, the weight coefficients of the smoothness terms are adaptively computed according to the reliability of neighboring samples. The experiments show that the proposed method outperforms the previous works in terms of quantitative and qualitative criteria.

깊이맵 업샘플링 방법의 객관적 성능 측정과 3D 시각적 피로도의 관계 분석

깊이맵은 3D 입체영상의 생성을 위해 중요한 요소이다. 하지만 깊이 카메라를 이용하여 획득한 깊이맵들은 낮은 해상도를 갖는 단점이 있기 때문에 이를 고해상도로 변환하는 연구들이 활발하게 진행되고 있다. 이러한 연구들은 일반적으로 PSNR, Sharpness Degree, Blur Metric 등과 같은 객관적인 평가방법으로 성능을 검증해왔다. 이러한 평가방법 이외에 DIBR로 가상시점(virtual view)을 생성하여 주관적으로 평가하는 연구도 있으나, 입체영상을 생성하여 깊이맵 업샘플링의 성능을 분석하는 것은 많지 않다. 본 논문에서는 다양한 깊이맵 업샘플링 방법들을 이용하여 생성된 입체영상의 주관적 평가와 업샘플링 방법의 객관적 평가 결과의 상관관계 및 선형회귀법을 이용하여 관련성을 분석한다. 실험결과에서는 에지 PSNR이 시각적 피로도와의 상관관계가 가장 높고, Blur Metric은 가장 낮다는 것을 보여준다. 또한 선형회귀에서는 최적의 입체영상을 얻을 수 있는 객관적 평가의 가중치를 구하고, 기존 또는 새로운 업샘플링 알고리즘의 3D성능을 예측할 수 있는 공식을 보여준다. A depth map is an important component for stereoscopic image generation. Since the depth map acquired from a depth camera has a low resolution, upsamling a low-resolution depth map to a high-resolution one has been studied past decades. Upsampling methods are evaluated by objective evaluation tools such as PSNR, Sharpness Degree, Blur Metric. As well, the subjective quality is compared using virtual views generated by DIBR (depth image based rendering). However, works on the analysis of the relation between depth map upsampling and stereoscopic images are relatively few. In this paper, we investigate the relationship between subjective evaluation of stereoscopic images and objective performance of upsampling methods using cross correlation and linear regression. Experimental results demonstrate that the correlation of edge PSNR and visual fatigue is the highest and the blur metric has lowest correlation. Further, from the linear regression, we found relative weights of objective measurements. Further we introduce a formulae that can estimate 3D performance of conventional or new upsampling methods.

A New Transcoding Scheme for Scalable Video Coding to H.264/AVC

Requests from various video terminals push video servers to equip with scalability for video contents distribution in different ways. Scalable Video Coding (SVC) as the extension of H.264/AVC standard can provide the scalability for video servers by encoding videos into one base layer and several enhancement layers. To enable mobile devices without scalability receive videos at their best extent, converting bit-streams from SVC into H.264/AVC becomes the key technique. Bit-stream rewriting is the simplest way without quality loss. However, rewriting is not a real transcoding scheme , since it needs to modify SVC encoder s . This paper proposes a novel transcoding approach to support spatial scalability by minimizing the distortions generated from re-encoding process . The proposed scheme keeps the input bit-streams’ information at maximum and adopts the hybrid upsampling method to do residue scaling, which can reduce the transcoding distortion into minimization. Experimental results demonstrate that the loss of the rate-distortion (RD) performance of the proposed transcoding scheme is better than Full Decoding Re-encoding (FDR) which can get the highest video quality in general sense, by achieving up to 0.9 dB Y-PSNR gain while saving 95%~97% processing time .

Efficient and Stable Sparse-to-Dense Conversion for Automatic 2-D to 3-D Conversion

Various important 3-D depth cues, such as focus, motion, occlusion, and disparity, can only be estimated reliably at distinct sparse image locations, such as edges and corners. Hence, for 2-D to 3-D video conversion, a stable and smooth sparse-to-dense conversion is required to propagate these sparse estimates to the complete video. To this end, optimization, segmentation, and triangulation-based approaches have been proposed recently. While optimization-based approaches produce accurate dense maps, the resulting energy functions are very hard to minimize within the stringent requirements of real-time video processing. In addition, segmentation and triangulation-based approaches can cause incorrect delineation of object boundaries. Dense maps that are independently estimated from video images suffer from temporal instabilities. To deal with the real-time issue, we propose an innovative low latency, line scanning based sparse-to-dense conversion algorithm with a low computational complexity. To mitigate the stability and smoothness issues, we additionally propose a recursive spatiotemporal postprocessing and an efficient joint bilateral up-sampling method. We illustrate the performance of the resulting sparse-to-dense converter on dense defocus maps. We also show a subjective assessment of 2-D to 3-D conversion results using a paired comparison on a variety of challenging low-depth-of-field test sequences. The results demonstrate that the proposed approach achieves equal 3-D depth and video quality as state-of-the-art sparse-to-dense converters with a significantly reduced computational complexity and memory usage.

Up-sampling oriented frame rate reduction

Reducing the frame rate of a sequence results in the information loss of interpolated frames that cannot be recovered by temporal up-sampling. The most prevalent frame rate reduction method is the direct down-sampling, which maintains the original frames of the input sequence with a constant interval. It achieves wide applications due to simplicity. However, it neglects the influences on interpolated frames brought by temporal up-sampling. To maintain more information during down-sampling, which will be used for interpolation, this paper proposes an up-sampling oriented frame rate reduction by hinging the temporal up-sampling and down-sampling. It is designed with the goal of improving the quality of up-sampled frames while keeping the down-sampled sequence faithful to the input one. For a particular temporal up-sampling method, e.g., motion compensation frame interpolation, the optimal down-sampled frame is the one that not only minimizes the difference between the original and the down-sampled frames but also minimizes the difference between the original and the corresponding up-sampled frames. Users can make a tradeoff between the two difference terms by selecting a proper lagrange factor depending on the applications. Furthermore, to decrease the space and computational complexity, a block-wise implementation is also devised in this paper. Experimental results conducted on various sequences demonstrate that the proposed frame rate reduction is able to significantly improve the quality of up-sampled frames compared with other down-sampling methods.