Image Upsampling Research Articles

Detail-preserving Joint Image Upsampling

Image operators can be instrumental to computational imaging and photography. However, many of them are computationally intensive. In this article, we propose an effective yet efficient joint upsampling method to accelerate various image operators. We show that edge-preserving filtering can be facilitated with a downsampling-and-upsampling process. Moreover, when the extent of smoothing is mild, the process is detail preserving, i.e., the fine details lost in the low-resolution (LR) images can be accurately restored in the high-resolution (HR) images. Given an HR input and an LR output of an operator, we downsample the HR input and calculate its affinities to the HR input. By applying the affinities to the LR output, we promote its resolution. Due to the strong detail-preserving property, the HR output derived in the previous step may exhibit aliasing artifacts around the salient edges. We further refine it based on the linear relations in a small neighborhood to rid the artifacts. Experiments on various image operators show that our method achieves superior quality over the state-of-the-art joint upsampling methods. Furthermore, the running time of our method is linear to the number of pixels. Our naive implementation derives 1080P images in real time (24 fps) on an NVIDIA GTX 3070 GPU.

Research on Image Up-scaling and Super-resolution Based on Convolutional Neural Network

Image upsampling and super-resolution techniques are of great significance in many fields, which can improve image quality and enhance information extraction capabilities. Image upsampling and super-resolution techniques have wide applications in the real world, such as image quality enhancement, satellite and remote sensing images, and security and surveillance. This paper describes and analyzes a machine learning approach to up-scaling a low resolution images using Convolutional Neural Networks. We build on previous works on single image super-resolution. We train 3 structurally similar but different models and obtain an improvement of up to 3% compared to the more common image upscaling method, Bicubic Interpolation.

ESPC-BCS-Net: A network-based CS method for underwater image compression and reconstruction

The Internet of Underwater Things (IoUT) is a typical energy-limited and bandwidth-limited system where the technical bottleneck is the asymmetry between the massive demand for information access and the limited communication bandwidth. Therefore, storing and transmitting high-quality underwater images is a challenging task. The data measured by cameras need to be effectively compressed before transmission to reduce storage and reconstruc-ted with minor errors, which is the best solution. Compressed sensing (CS) theory breaks through the Nyquist sampling theorem and has been widely used to reconstruct sparse signals accurately. For adaptive sampling underwater images and improving the reconstruction performance, we propose the ESPC-BCS-Net by combining the advantages of CS and Deep Learning. The ESPC-BCS-Net consists of three parts: Sampling-Net, ESPC-Net, and BCS-Net. The parameters (e.g. sampling matrix, sparse transforms, shrinkage thresholds, etc.) in ESPC-BCS-Net are learned end-to-end rather than hand-crafted. The Sampling-Net achieves adaptive sampling by replacing the sampling matrix with a convolutional layer. The ESPC-Net implements image upsampling, while the BCS-Net is used to image reconstruction. The efficient sub-pixel layer of ESPC-Net effectively avoids blocking artifacts. The visual and quantitative evaluation of the experimental results shows that the underwater image reconstruction still performs well when the CS ratio is 0.1 and the PSNR of the reconstructed underwater images is above 29.

Joint Upsampling for Refocusing Light Fields Derived With Hybrid Lenses

As a 3D world sensing instrument, the light field camera provides an effective measure to perceive the depth information, which enables the fascinating function to refocus after exposure. However, existing light field cameras suffer from limited spatial resolution. Recently, hybrid imaging systems have been widely explored to overcome this limitation, where only the central sub-view (sub-aperture image) has a high resolution. In this paper, we propose a novel refocus scheme for the hybrid imaging system, which derives the central-view refocused image at low resolution, then jointly upsamples it under the guidance of the high-resolution central sub-view. The state-of-the-art joint upsampling methods are based on the local-linear-relation (LLR) or the consistent-structure (CS) assumptions, which are barely valid for our application scenario. We propose a linear fusion block and stack the proposed blocks into a lightweight fully convolutional network named linear fusion network (LFN) for refocused image upsampling. We have conducted extensive experiments to evaluate the proposed method. Both qualitative and quantitative measures are explored. Experimental results indicate that LFN outperforms existing methods on the task of refocused image upsampling and shows that the proposed refocus scheme is practical.

Real-time whole-brain imaging of hemodynamics and oxygenation at micro-vessel resolution with ultrafast wide-field photoacoustic microscopy

High-speed high-resolution imaging of the whole-brain hemodynamics is critically important to facilitating neurovascular research. High imaging speed and image quality are crucial to visualizing real-time hemodynamics in complex brain vascular networks, and tracking fast pathophysiological activities at the microvessel level, which will enable advances in current queries in neurovascular and brain metabolism research, including stroke, dementia, and acute brain injury. Further, real-time imaging of oxygen saturation of hemoglobin (sO2) can capture fast-paced oxygen delivery dynamics, which is needed to solve pertinent questions in these fields and beyond. Here, we present a novel ultrafast functional photoacoustic microscopy (UFF-PAM) to image the whole-brain hemodynamics and oxygenation. UFF-PAM takes advantage of several key engineering innovations, including stimulated Raman scattering (SRS) based dual-wavelength laser excitation, water-immersible 12-facet-polygon scanner, high-sensitivity ultrasound transducer, and deep-learning-based image upsampling. A volumetric imaging rate of 2 Hz has been achieved over a field of view (FOV) of 11 × 7.5 × 1.5 mm3 with a high spatial resolution of ~10 μm. Using the UFF-PAM system, we have demonstrated proof-of-concept studies on the mouse brains in response to systemic hypoxia, sodium nitroprusside, and stroke. We observed the mouse brain’s fast morphological and functional changes over the entire cortex, including vasoconstriction, vasodilation, and deoxygenation. More interestingly, for the first time, with the whole-brain FOV and micro-vessel resolution, we captured the vasoconstriction and hypoxia simultaneously in the spreading depolarization (SD) wave. We expect the new imaging technology will provide a great potential for fundamental brain research under various pathological and physiological conditions.

Downsampled depth encoding for enhanced 3D range geometry compression.

The use of three-dimensional (3D) range geometry is expanding across a variety of disciplines ranging from medicine to the visual arts. A large amount of information is available in 3D range geometry, causing some applications to be limited in their ability to effectively store or transmit captured data. To help alleviate this constraint, a variety of 3D range data compression techniques have been proposed. One method, multiwavelength depth (MWD) encoding, smoothly encodes 3D range geometry into the three color channels of a 2D RGB image. To the best of our knowledge, we present a novel compression enhancement to further reduce file sizes that employs image downsampling, MWD encoding, and lossless (e.g., PNG) or lossy (e.g., JPEG) compression. Image upsampling is used to return downsampled encodings to their original resolution from which the 3D information is then decoded. The proposed method is robust to various scales of downsampling and levels of lossy compression. For example, when this method was applied with 50% downsampling and JPEG 85 to an encoding of a 3D face scan, a compression ratio of 68.85:1 versus the raw data was achieved with a global RMS reconstruction accuracy of 98.77%. Experimental results demonstrate that the proposed method can provide substantial file size savings at minimal reduction in overall reconstruction accuracy.

Fast image scanning microscopy with efficient image reconstruction

Image Scanning Microscopy is an advanced linear super-resolution fluorescence microscopy technique. Unlike traditional methods that introduce complex optical system designs, ISM is capble of taking the advantage of computational method to achieve super-resolution. As ISM involves large-scale localizations to all scanned-positions in most experiments, robust and accurate algorithms are required to avoid artifacts. Meanwhile, the existing ISM image reconstruction method is mainly based on post-processing technique due to high computing complexity. Therefore, large-scale images processing in ISM image reconstruction becomes too computationally intensive, which impedes the development of fast ISM. In this article, a robust ISM scanning-position localization technique and an efficient accurate image reconstruction technique for large scale ISM reconstruction are proposed. In particular, the localization of the ISM scanning position is realized by efficient image correlation and accurate robust Gaussian fitting. For simplicity and accuracy, the image reconstruction step introduces an efficient sub-pixel reassignment technique, by which image up-sampling is completely avoided. The simulation results show high performance in the proposed localization technique. The experimental result shows that the proposed image reconstruction technique can reach 20 times faster than existing method, which makes it very suitable for large-scale image processing in ISM image reconstruction, especially in a real-time ISM system.

UPDResNN: A Deep Light-Weight Image Upsampling and Deblurring Residual Neural Network

The physical process used in CCD cameras for image formation makes it imperative to simultaneously upsample and deblur the captured images. In this paper, we provide an efficient scheme to solve this problem through a nonlinear end-to-end mapping carried out by a novel deep light-weight residual neural network. The proposed network is designed based on two main modules, namely, image upsampling and image deblurring, aimed for carrying out simultaneously the two tasks involved with the problem. The proposed network employs a residual block with a capacity of generating features in multiple receptive fields and enhancing the network's representational capability. The proposed network is extensively experimented using benchmark datasets for image restoration.

An Energy-Efficient Edge Computing Paradigm for Convolution-Based Image Upsampling

State-of-the-art deep learning solutions for image upsampling are currently trained using either resize or sub-pixel convolution to learn kernels that generate high fidelity images with minimal artifacts. However, performing inference with these learned convolution kernels requires memory-intensive feature map transformations that dominate time and energy costs in real-time applications. To alleviate this pressure on memory bandwidth, we propose a novel energy-efficient edge computing paradigm that confines the use of resize or sub-pixel convolution to training in the cloud by transforming learned convolution kernels to deconvolution kernels before deploying them for inference as a functionally equivalent deconvolution. These kernel transformations, intended as a one-time cost when shifting from training to inference, enable a systems designer to use each algorithm in their optimal context by preserving the image fidelity learned when training in the cloud while minimizing data transfer penalties during inference at the edge. We compare the inference properties of these convolution-based image upsampling algorithms and introduce a novel deconvolution inference algorithm, which we refer to as REVD2. To demonstrate the benefits of our approach, we upsample images selected from the BSD300 dataset using a pre-trained single-image super resolution network provided by the PyTorch model zoo. Using quantitative models of incurred time and energy costs to analyze this deep neural network, we estimate that using REVD2 for inference at the edge improves system latency by $2.1\times $ or $2.8\times $ and energy efficiency by $2.1\times $ or $2.7\times $ when respectively compared to sub-pixel or resize convolution counterparts.

Learning Spatially Variant Linear Representation Models for Joint Filtering.

Joint filtering mainly uses an additional guidance image as a prior and transfers its structures to the target image in the filtering process. Different from existing approaches that rely on local linear models or hand-designed objective functions to extract the structural information from the guidance image, we propose a new joint filtering method based on a spatially variant linear representation model (SVLRM), where the target image is linearly represented by the guidance image. However, learning SVLRMs for vision tasks is a highly ill-posed problem. To estimate the spatially variant linear representation coefficients, we develop an effective approach based on a deep convolutional neural network (CNN). As such, the proposed deep CNN (constrained by the SVLRM) is able to model the structural information of both the guidance and input images. We show that the proposed approach can be effectively applied to a variety of applications, including depth/RGB image upsampling and restoration, flash deblurring, natural image denoising, and scale-aware filtering. In addition, we show that the linear representation model can be extended to high-order representation models (e.g., quadratic and cubic polynomial representations). Extensive experimental results demonstrate that the proposed method performs favorably against the state-of-the-art methods that have been specifically designed for each task.

Image super-resolution model using an improved deep learning-based facial expression analysis

Image upsampling and noise removal are important tasks in digital image processing. Single-image upsampling and denoising influence the quality of the resulting images. Image upsampling is known as super-resolution (SR) and referred to as the restoration of a higher-resolution image from a given low-resolution image. In facial expression analysis, the resolution of the original image directly affects the reliability and validity of the emotional analysis. Hence, optimization of the resolution of the extracted original image during emotion analysis is important. In this study, a model is proposed, which applies an image super-resolution method to an algorithm that classifies emotions from facial expressions.

Build receptive pyramid for efficient color image compression artifact reduction

A traditional image pyramid is an effective way to extract multiscale features. However, for image restoration, downsampling and upsampling lose details in the original resolution and result in an over smooth result. To overcome this defect, we propose a receptive pyramid (RP) that replaces downsampling by dilated convolution to extract multiscale features on the original resolution and make a full resolution correction. We present an RP-based convolution neural network named receptive pyramid convolutional network (RPCN) for efficient color image compression artifact reduction (CAR). Specifically, we propose residual connected convolution blocks as the baseline of RPCN. RPs work in convolution blocks to extract the hierarchical multiscale features for local feature correction. Moreover, the global feature fusion and vector correction are also introduced to further exploit the hierarchical features from the baseline. Benefiting from the RP, our RPCN achieves state-of-the-art performance with a much smaller model and much faster running speed for the CAR task.

Noise-Robust MRI Upsampling Using Adaptive Local Steering Kernel

Upsampling and denoising of magnetic resonance images are conventionally performed separately, which would introduce unwanted artifacts such as blurring. To address this problem, we propose an innovative adaptive interpolation framework to achieve simultaneous image upsampling, denoising, and detail enhancement. First, local steering kernel (LSK) function is leveraged to adapt the interpolation weights according to geometric structures in the magnetic resonance (MR) images. An adaptive sharpening of the LSK weight matrix and a Rician bias correction are then adopted to remove Rician noise and enhance fine details. In this regard, the adaptive LSK extends the zero-order point estimation framework to higher orders of regression, and therefore facilitates edge preservation and detail reconstruction. The post-processing Rician correction avoids the bias caused by the asymmetry of Rician noise distributions. Experimental results using both real and synthetic clinical MR cranial images (with and without noise) demonstrated that our algorithm produced better reconstruction results than several traditional interpolation-based upsampling methods, including nearest neighbor (NN), non-local means (NLM), self-learning super resolution (SLSR), Gaussian process regression (GPR), and even comparable to four deep-learning-based methods but with less data requirements and computational complexity. The proposed technique resulted in PSNR and SSIM values were ~3%-16% higher than any of the other traditional algorithms tested, and our method recovered more clear textures from noisy images compared with deep-learning-based methods. As such, the presented technique is a viable new approach to MR upsampling, particularly for noisy images.

A Study of Image Upsampling and Downsampling Filters

In this paper, a set of techniques used for downsampling and upsampling of 2D images is analyzed on various image datasets. The comparison takes into account a significant number of interpolation kernels, their parameters, and their algebraical form, focusing mostly on linear interpolation methods with symmetric kernels. The most suitable metrics for measuring the performance of upsampling and downsampling filters’ combinations are presented, discussing their strengths and weaknesses. A test benchmark is proposed, and the obtained results are analyzed with respect to the presented metrics, offering explanations about specific filter behaviors in general, or just in certain circumstances. In the end, a set of filters and parameters recommendations is offered based on extensive testing on carefully selected image datasets. The entire research is based on the study of a large set of research papers and on a solid discussion of the underlying signal processing theory.

FFT multichannel interpolation and application to image super-resolution

This paper presents an innovative set of tools to support a methodology for the multichannel interpolation (MCI) of a discrete signal. It is shown that a bandlimited signal f can be exactly reconstructed from finite samples of gk (1 ≤ k ≤ M) which are the responses of M linear systems with input f. The proposed interpolation can also be applied to approximate non-bandlimited signals. Quantitative error is analyzed to ensure its effectiveness in approximating non-bandlimited signals and its Hilbert transform. Based on the FFT technique, a fast algorithm which brings high computational efficiency and reliability for MCI is presented. The standout performance of MCI is illustrated by several simulations. Additionally, the proposed interpolation is applied to the single image super-resolution (SISR). Its superior performance in accuracy and speed of SISR is demonstrated by the experimental studies. Our results are compared qualitatively and quantitatively with the state-of-the-art methods in image upsampling and reconstruction by using the standard measurement criteria.

Deep Color Guided Coarse-to-Fine Convolutional Network Cascade for Depth Image Super-Resolution.

Depth image super-resolution is a significant yet challenging task. In this paper, we introduce a novel deep color guided coarse-to-fine convolutional neural network (CNN) framework to address this problem. First, we present a datadriven filter method to approximate the ideal filter for depth image super-resolution instead of hand-designed filters. Based on large data samples, the filter learned is more accurate and stable for upsampling depth image. Second, we introduce a coarse-to-fine CNN to learn different sizes of filter kernels. In coarse stage, larger filter kernels are learned by CNN to achieve crude high-resolution depth image. As to fine stage, the crude high-resolution depth image is used as the input so that smaller filter kernels are learned to gain more accurate results. Benefit from this network, we can progressively recover the high frequency details. Third, we construct a color guidance strategy that fuses color difference and spatial distance for depth image upsampling. We revise the interpolated high-resolution depth image according to the corresponding pixels in highresolution color maps. Guided by color information, the depth of high-resolution image obtained can alleviate texture copying artifacts and preserve edge details effectively. Quantitative and qualitative experimental results demonstrate our state-of-the-art performance for depth map super-resolution.

Magnetic Resonance Image Upsampling via Shape-Adaptive Data Fitting

Magnetic Resonance Image Upsampling via Shape-Adaptive Data Fitting

Image up-sampling using deep cascaded neural networks in dual domains for images down-sampled in DCT domain

Recent researches show that the high-frequency discrete cosine transform (DCT) coefficients can be estimated from low-frequency DCT coefficients by exploiting the spatial correlations. Hence, images coded by DCT such as JPEG/MJPEG/H.264, etc., can be down-sampled in DCT domain, where the high-frequency information can be accurately restored through image up-sampling. In this letter, we propose a novel deep neural network using the cascaded fully connected layers and convolution layers in dual domains (DCT and spatial domains), in order to restore high-frequency DCT coefficients from observed low-frequency DCT coefficients by exploiting the DCT inter-block and spatial correlations. In the proposed network, many recent techniques are adopted, including residual network in dual domains, batch normalization, denseNet, etc. Experimental results show that the proposed cascaded networks in dual domains significantly outperforms the state-of-the-art DCT up-sampling methods in terms of PSNR (0.63–2.57 dB gain), SSIM values, and subjective evaluations on standard image datasets Set5 and Set14.

Displacement field determination using an iterative optical flow strategy

This study presents a new strategy of retrieving displacement fields from two different images or two video frames obtained at different times using an iterative optical flow method. First, illumination modeling and image preprocessing are required before the displacement calculations can be performed. Due to the displacement continuity assumption, the displacement differences between the neighboring points are small. Then, the initial displacements of the unprocessed points are determined by the displacements of the processed neighboring points. Next, an iterative optical flow method is performed with a small window size to obtain the displacement offsets between the matching point pairs. The calculation is performed at the subpixel level in each iteration, and thus image interpolation is necessary. In this paper, image upsampling of the deformed image is applied using bi-cubic spline interpolation to achieve the subpixel intensity before all the calculations. This improvement avoids the requirement for image interpolation to be in all iterations. Several numerical tests and real image datasets were used to validate the performance of the proposed method. The results were compared with those obtained by a traditional digital image correlation method using the traditional forward additive Newton–Raphson algorithm and the iterative least square algorithm. Both tests showed that the proposed algorithm exhibits excellent performance regarding computational accuracy and speed.

Technique for two-dimensional displacement field determination using a reliability-guided spatial-gradient-based digital image correlation algorithm.

This paper proposed a novel in-plane displacement field measurement algorithm using an optical flow strategy. We built a linear illumination model between images before and after deformation to guarantee intensity invariability. We used image upsampling and a reliability-guided strategy to find the matching points accurate to 0.5pixels in the reference and deformed images. The criterion to determine the reliability is zero-mean normalized cross-correlation coefficient. Afterward, we used the brightness constancy constraint combined with Lucas-Kanade optical flow constraint in a specific image region to obtain an overdetermined linear equation. We applied the noniterative least-squares algorithm to solve the equations and to achieve the displacement offsets. This research utilized multithread calculation to handle the complete cracking applications. We estimated the computing efficiency and calculation precision of the proposed method through a series of experimental speckle patterns. All results demonstrated the correctness, effectiveness, and robustness of the proposed method.