Joint Upsampling Research Articles

Local look-up table upsampling for accelerating image processing

The resolution of cameras is increasing, and speedup of various image processing is required to accompany this increase. A simple way of acceleration is processing the image at low resolution and then upsampling the result. Moreover, when we can use an additional high-resolution image as guidance formation for upsampling, we can upsample the image processing results more accurately. We propose an approach to accelerate various image processing by downsampling and joint upsampling. This paper utilizes per-pixel look-up tables (LUTs), named local LUT, which are given a low-resolution input image and output pair. Subsequently, we upsample the local LUT. We can then generate a high-resolution image only by referring to its local LUT. In our experimental results, we evaluated the proposed method on several image processing filters and applications: iterative bilateral filtering, ℓ0\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\ell _0$$\\end{document} smoothing, local Laplacian filtering, inpainting, and haze removing. The proposed method accelerates image processing with sufficient approximation accuracy, and the proposed outperforms the conventional approaches in the trade-off between accuracy and efficiency. Our code is available at https://fukushimalab.github.io/LLF/.

Task-Specific Scene Structure Representations

Understanding the informative structures of scenes is essential for low-level vision tasks. Unfortunately, it is difficult to obtain a concrete visual definition of the informative structures because influences of visual features are task-specific. In this paper, we propose a single general neural network architecture for extracting task-specific structure guidance for scenes. To do this, we first analyze traditional spectral clustering methods, which computes a set of eigenvectors to model a segmented graph forming small compact structures on image domains. We then unfold the traditional graph-partitioning problem into a learnable network, named Scene Structure Guidance Network (SSGNet), to represent the task-specific informative structures. The SSGNet yields a set of coefficients of eigenvectors that produces explicit feature representations of image structures. In addition, our SSGNet is light-weight (56K parameters), and can be used as a plug-and-play module for off-the-shelf architectures. We optimize the SSGNet without any supervision by proposing two novel training losses that enforce task-specific scene structure generation during training. Our main contribution is to show that such a simple network can achieve state-of-the-art results for several low-level vision applications including joint upsampling and image denoising. We also demonstrate that our SSGNet generalizes well on unseen datasets, compared to existing methods which use structural embedding frameworks. Our source codes are available at https://github.com/jsshin98/SSGNet.

Joint Upsampling for Refocusing Light Fields Derived With Hybrid Lenses

As a 3D world sensing instrument, the light field camera provides an effective measure to perceive the depth information, which enables the fascinating function to refocus after exposure. However, existing light field cameras suffer from limited spatial resolution. Recently, hybrid imaging systems have been widely explored to overcome this limitation, where only the central sub-view (sub-aperture image) has a high resolution. In this paper, we propose a novel refocus scheme for the hybrid imaging system, which derives the central-view refocused image at low resolution, then jointly upsamples it under the guidance of the high-resolution central sub-view. The state-of-the-art joint upsampling methods are based on the local-linear-relation (LLR) or the consistent-structure (CS) assumptions, which are barely valid for our application scenario. We propose a linear fusion block and stack the proposed blocks into a lightweight fully convolutional network named linear fusion network (LFN) for refocused image upsampling. We have conducted extensive experiments to evaluate the proposed method. Both qualitative and quantitative measures are explored. Experimental results indicate that LFN outperforms existing methods on the task of refocused image upsampling and shows that the proposed refocus scheme is practical.

Ultrasound Image Segmentation of Thyroid Nodules Based On Joint Up-sampling

In view of the problem of inaccurate segmentation of thyroid nodule ultrasound images, a method of thyroid nodule ultrasound image segmentation based on joint up-sampling is proposed. In order to solve the problem of serious speckle noise in the original ultrasound data, preprocessing is performed to enhance the contrast between the nodule area and the background area. In order to achieve accurate positioning of the nodule target area, a joint up-sampling module is designed to fuse the context information of ordinary standard convolution and expansion convolution with different expansion coefficients. Experimental results show that the proposed method achieves 93.19% accuracy and dice similarity coefficient is 0.8558, which is better than other existing thyroid nodule segmentation network models.

Multitask bilateral learning for real‐time image enhancement

AbstractNowadays, deep neural networks (DNNs) for image processing are becoming more complex; thus, reducing computational cost is increasingly important. This study highlights the construction of a DNN for real‐time image processing, training various image processing operators efficiently through multitask learning. For real‐time image processing, the proposed algorithm takes a joint upsampling approach through bilateral guided upsampling. For multitask learning, the overall network is based on an encoder‐decoder architecture, which consists of encoding, processing, and decoding components, in which the encoding and decoding components are shared by all the image processing operators. In the processing component, a semantic guidance map, which contains processing information for each image processing operator, is estimated using simple linear shifts of the shared deep features. Through these components, the proposed algorithm requires an increase of only 5% in the number of parameters to add another image processing operator and achieves faster and higher performance than that of deep‐learning‐based joint upsampling methods in local image processing as well as global image processing.

Joint Up-sampling High Dynamic Range Images with a Guidance Image

Joint Up-sampling High Dynamic Range Images with a Guidance Image

Joint upsampling of random color distance maps for fast salient region detection

The human visual system is capable of rapid response, even in the presence of massive quantities of visual information. This is possible because it restricts the operation of further processing stages to a small, potentially important, subset of the incoming information. This mechanism is called visual attention and is drawn by distinctive, visually salient, regions of the scene. Detection of visually salient regions is widely employed in vision-based applications, since a reduction in visual search space can lead to significant improvement in computational performance. Despite recent advances in salient region detection, most efforts have focused on improving accuracy, at the expense of increased execution time, significantly hindering their applicability. To address this, a fast and accurate salient region detection method is presented in this work, based on an efficient saliency estimate called random color distance map. This estimate is joint upsampled into an accurate saliency map, which is assessed and compared to saliency maps obtained by other four state-of-the-art methods on the MSRA1K, MSRA10K and SED2 datasets, showing that it is highly competitive in both accuracy and execution time.

Robust Guided Image Filtering Using Nonconvex Potentials.

Filtering images using a guidance signal, a process called guided or joint image filtering, has been used in various tasks in computer vision and computational photography, particularly for noise reduction and joint upsampling. This uses an additional guidance signal as a structure prior, and transfers the structure of the guidance signal to an input image, restoring noisy or altered image structure. The main drawbacks of such a data-dependent framework are that it does not consider structural differences between guidance and input images, and that it is not robust to outliers. We propose a novel SD (for static/dynamic) filter to address these problems in a unified framework, and jointly leverage structural information from guidance and input images. Guided image filtering is formulated as a nonconvex optimization problem, which is solved by the majorize-minimization algorithm. The proposed algorithm converges quickly while guaranteeing a local minimum. The SD filter effectively controls the underlying image structure at different scales, and can handle a variety of types of data from different sensors. It is robust to outliers and other artifacts such as gradient reversal and global intensity shift, and has good edge-preserving smoothing properties. We demonstrate the flexibility and effectiveness of the proposed SD filter in a variety of applications, including depth upsampling, scale-space filtering, texture removal, flash/non-flash denoising, and RGB/NIR denoising.

Guided Image Filtering

In this paper, we propose a novel explicit image filter called guided filter. Derived from a local linear model, the guided filter computes the filtering output by considering the content of a guidance image, which can be the input image itself or another different image. The guided filter can be used as an edge-preserving smoothing operator like the popular bilateral filter [1], but it has better behaviors near edges. The guided filter is also a more generic concept beyond smoothing: It can transfer the structures of the guidance image to the filtering output, enabling new filtering applications like dehazing and guided feathering. Moreover, the guided filter naturally has a fast and nonapproximate linear time algorithm, regardless of the kernel size and the intensity range. Currently, it is one of the fastest edge-preserving filters. Experiments show that the guided filter is both effective and efficient in a great variety of computer vision and computer graphics applications, including edge-aware smoothing, detail enhancement, HDR compression, image matting/feathering, dehazing, joint upsampling, etc.

Model-based adaptive resolution upconversion of degraded images

Resolution upconversion of a degraded image is an ill-posed inverse problem that is even harder than video superresolution due to lack of redundant observations from reference frames. To overcome this difficulty an adaptive 2D piecewise autoregressive (PAR) model is used to strengthen the constraints on the solution of the inverse problem. The PAR model can be fit to local image waveforms by adjusting its parameters. But estimating the model parameters needs the knowledge of the very original high-resolution pixels to be estimated by the model. We resolve this chicken-and-egg dilemma by adaptive nonlinear least-squares joint estimation of both model parameters and original pixels. This non-linear estimation problem is solved by the method of structured total least-squares, constrained by the degradation function (e.g., the point spread function of a camera plus noises) that forms the observed low-resolution image. As such, this work offers a unified general framework for joint upsampling, deconvolution, and denoising. Moreover, the upsampling can be carried out at an arbitrary scale rather than power of two. Experiments show that the proposed NEARU technique outperforms current methods in both PSNR and subjective visual quality, and its advantage becomes greater for larger scaling factors.