Up-sampling Factor Research Articles

Large Factor Image Super-Resolution With Cascaded Convolutional Neural Networks

Recently, convolutional neural networks (CNNs) have attracted considerable attention in single image super-resolution (SISR) and have enabled great performance improvements. However, most of the existing methods super-resolve input images to the desired size with an interpolation operation during the beginning stage, which brings about heavy aliasing artifacts and high computational costs. Especially for large upsampling factors (e.g., 8×), it remains a challenge to restore high-quality results for deeply degraded images. To tackle this problem, we propose a cascaded super-resolution convolutional neural network (CSRCNN), which takes a single low-resolution (LR) image as an input and reconstructs high-resolution (HR) images in a progressive way. At each cascaded level, to help converge and improve the accuracy, a novel U-net based block with backprojection is first introduced, which exploits the mutual relation between HR and LR feature spaces. A refined block following the U-net block is also used to reconstruct the realistic texture details. In addition, we naturally utilize the strategy of curriculum learning, organizing the learning process from easy (small factors) to hard (large factors). Comprehensive experiments on benchmark datasets demonstrate that the proposed network achieves superior results compared with those of other state-of-the-art methods, particularly with the 8× upsampling factor.

Frequency-Dependent Depth Map Enhancement via Iterative Depth-Guided Affine Transformation and Intensity-Guided Refinement

Recently, deep convolutional neural network sho-ws significant improvement for intensity-guided depth map enhancement. The most networks focus on either increasing depth or easing features propagation via residual learning and dense connection. However, it has not been explicitly considered yet to mitigate the artifacts caused by the differences of the distributions between the depth map and the corresponding color image, e.g., edge misalignment. In this paper, a novel depth-guided affine transformation is used to filter out the unrelated intensity features, which is further used to refine the depth features. Since the quality of initial depth features is low, the depth-guided intensity features filtering and the intensity-guided depth features refinement are iteratively performed, which progressively promotes effects of such tasks. To make full use of the iterations, all the refined depth features are dense connected followed by a 1 × 1 convolution layer. In addition, to improve the performance in the case of large upsampling factors (e.g., 16×), the depth features are enhanced from coarse to fine. In each frequency-dependent refinement of the depth features, the above iterative subnetwork as well as the residual learning are introduced. The proposed method is tested for the noise-free and noisy cases which compares against 16 state-of-the-art methods. Our experimental results show the improved performances based on the qualitative and quantitative evaluations.

Physical layer security performance analysis of the time reversal transmission system

Thanks to its characteristics of temporal compression and spatial focusing, the time reversal (TR) transmission system has the intrinsic capability of anti-eavesdropping. This study analyses the physical layer security performance of the TR transmission system. The probability density functions and cumulative distribution functions of the signal-to-noise ratio of the legitimate user and eavesdropper are given. On this basis, the theoretical expressions of the ergodic capacities of the legitimate channel and the wiretap channel, and the achievable ergodic secrecy rate are derived. The bit error rates of the legitimate user and eavesdropper for binary phase-shift keying modulation are given too. The correctness of the theoretical derivation is verified by simulation. For the legitimate user, the signal power increases as the number of paths increases, which indicates a higher diversity gain can be obtained, and the inter-symbol interference power decreases as the up-sampling factor increases. So, the signal-to-interference-plus-noise ratio (SINR) of the received signal can be significantly improved by appropriately reducing the spectral efficiency. However, the received signal's SINR of the eavesdropper cannot be promoted obviously. That is to say, compared with the eavesdropper, the legitimate user can achieve a higher rate and a lower BER, so secure transmission can be realised.

Pyramid-Structured Depth MAP Super-Resolution Based on Deep Dense-Residual Network

Although deep convolutional neural networks (DCNN) show significant improvement for single depth map (SD) super-resolution (SR) over the traditional counterparts, most SDSR DCNNs do not reuse the hierarchical features for depth map SR resulting in blurred high-resolution (HR) depth maps. They always stack convolutional layers to make network deeper and wider. In addition, most SDSR networks generate HR depth maps at a single level, which is not suitable for large up-sampling factors. To solve these problems, we present pyramid-structured depth map super-resolution based on deep dense-residual network. Specially, our networks are made up of dense residual blocks that use densely connected layers and residual learning to model the mapping between high-frequency residuals and low-resolution (LR) depth map. Furthermore, based on the pyramid structure, our network can progressively generate depth maps of various levels by taking advantages of features from different levels. The proposed network adopts a deep supervision scheme to reduce the difficulty of model training and further improve the performance. The proposed method is evaluated on Middlebury datasets which shows improved performance compared with 6 state-of-the-art methods.

Multiple Cycle-in-Cycle Generative Adversarial Networks for Unsupervised Image Super-Resolution.

With the help of convolutional neural networks (CNN), the single image super-resolution problem has been widely studied. Most of these CNN based methods focus on learning a model to map a low-resolution (LR) image to a highresolution (HR) image, where the LR image is downsampled from the HR image with a known model. However, in a more general case when the process of the down-sampling is unknown and the LR input is degraded by noises and blurring, it is difficult to acquire the LR and HR image pairs for traditional supervised learning. Inspired by the recent unsupervised imagestyle translation applications using unpaired data, we propose a multiple Cycle-in-Cycle network structure to deal with the more general case using multiple generative adversarial networks (GAN) as the basis components. The first network cycle aims at mapping the noisy and blurry LR input to a noise-free LR space, then a new cycle with a well-trained ×2 network model is orderly introduced to super-resolve the intermediate output of the former cycle. The number of total cycles depends on the different up-sampling factors (×2, ×4, ×8). Finally, all modules are trained in an end-to-end manner to get the desired HR output. Quantitative indexes and qualitative results show that our proposed method achieves comparable performance with the state-of-the-art supervised models.

G-GANISR: Gradual generative adversarial network for image super resolution

Adversarial methods have demonsterated to be signifiant at generating realistic images. However, these approaches have a challenging training process which partially attributed to the performance of discriminator. In this paper, we proposed an efficient super-resolution model based on generative adversarial network (GAN), to effectively generate reprehensive information and improve the test quality of the real-world images. To overcome the current issues, we designed the discriminator of our model based on the Least Square Loss function. The proposed network is organized by a gradual learning process from simple to advanced, which means from the small upsampling factors to the large upsampling factor that helps to improve the overall stability of the training. In particular, to control the model parameters and mitigate the training difficulties, dense residual learning strategy is adopted. Indeed, the key idea of proposed methodology is (i) fully exploit all the image details without losing information by gradually increases the task of discriminator, where the output of each layer is gradually improved in the next layer. In this way the model efficiently generates a super-resolution image even up to high scaling factors (e.g. × 8). (ii) The model is stable during the learning process, as we use least square loss instead of cross-entropy. In addition, the effects of different objective function on training stability are compared. To evaluate the model we conducted two sets of experiments, by using the proposed gradual GAN and the regular GAN to demonstrate the efficiency and stability of the proposed model for both quantitative and qualitative benchmarks.

Performance Analysis of Time Reversal Communication Systems

The time reversal (TR) transmission technique can harvest the signal energy from multiple paths and focus the signal power on an intended space–time point. Different from the works which analyze the average signal-to-noise-ratio (SNR), this letter derives the probability density function (PDF) and cumulative distribution function (CDF) of the SNR at the receiver in a TR communication system. Furthermore, based on the derived pdf, the ergodic capacity, outage probability, and bit error rate (BER) when binary phase-shift keying is used are also obtained. The simulation results show that the theoretical pdf and cdf of the SNR are correct, and the ergodic capacity, in theory, is consistent with the simulation result, so are the outage probability and the BER. In the TR system, the inter-symbol interference decreases as the up-sampling factor increases, and the diversity gain increases as the number of paths increases, that is, the received signal quality can be improved by reducing the spectral efficiency.

A multi-image super-resolution algorithm applied to thermal imagery

Although thermal imaging is a widely used technique in many applications and is under continuous development, one of its limits is the relatively coarse spatial resolution. Nevertheless, in the last years, a number of super-resolution algorithms have been developed which allow to enhance the resolution of the images. They can be divided in two main different categories: single-image or multiple image-based algorithms. In this work, a multiple image-based algorithm for the super-resolution was implemented, tested and applied to terrestrial thermal imaging with the aim to overcome the limitation of the low resolution. In particular, the method relies on the use of many images acquired from slightly different positions to obtain, thanks to the redundancy of observations, a super-resolution frame having an upsampling factor of four. Several tests were performed on synthetic datasets, and the accuracy of the obtained super-resolution images was investigated. Moreover, an original algorithm capable to identify gross errors during the image registration phase, which is one of the crucial phases, is presented and its reliability assessed. Results showed the effectiveness of the proposed method on both common visible images and thermal infrared ones, since discrepancies between reconstructed and reference values are reduced by 18 and 25% respectively, when compared with a conventional bicubic algorithm. Finally, the proposed method was tested on a case study concerning the thermal survey of the façade of a historical building in Bologna (Palazzo D’Accursio). A dataset of real thermal frames was acquired and a super-resolution image of the subject was generated through the developed algorithm. Strengths and weaknesses of the method were analysed and discussed in the paper.

Realistic single-image super-resolution using autoencoding adversarial networks

The accuracy and efficiency of single-image super-resolution (SR) using techniques based on convolutional neural networks have recently shown much improvement. However, most of the existing algorithms aim at improving the peak signal-to-noise ratio by minimizing the mean squared error between the ground-truth images and the generated SR images. This leads to a lack of high-frequency information and nonconformance with the perception of human eyes. To reconstruct realistic natural images in SR with large up-sampling factors, we combine the benefits of some recent approaches and propose a method based on autoencoding adversarial networks. The proposed architecture includes a generator, which is a symmetric encode–decode network used to extract feature maps and recover high-resolution images, and a conditional discriminator, which is used to determine whether the generated image is from the real distribution or not. In addition, we extract high-level features from a pretrained network to optimize the perceptual loss and make the output more precise. Compared with several state-of-the-art methods, our proposed method shows outstanding performance in recovering fine texture details. The mean opinion score shows that our method yields results that are more satisfactory to human perception than the other methods under comparison.

Guided depth image reconstruction from very sparse measurements

Depth images captured from modern depth cameras generally suffer from low spatial resolution, noise, and missing regions. These kinds of images cannot be used directly in applications related to depth images, e.g., robot navigation, 3DTV, and augmented reality, which basically need high-resolution input images with no noise o missing regions to function properly. To address the problem of low spatial resolution, noise degradation, and missing regions in depth images, we propose methods based on a guidance color image for depth reconstruction (DR) from sparse depth inputs and depth image super-resolution (SR). We also suggest a scenario wherein these problems can be integrated and addressed simultaneously. Further, we also demonstrate applications of the proposed approach for depth image denoising and depth image inpainting. In our approach, the guidance color image is used for obtaining the segment cues by applying mean-shift (MS) or simple linear iterative clustering (SLIC) segmentation on it. These strong segment cues help in aiding the DR and SR problems by considering the corresponding segments in the input depth image, and estimate the unknown pixels by either plane fitting or median filling approaches. Furthermore, we explore both direct and pyramidal (hierarchical) approaches for SR and DR-SR for higher upsampling factor. As such, our approaches are relatively simpler than some of the contemporary methods, yet the experimental results of the proposed methods show superior performance as compared with some other state-of-the-art DR and SR methods.

Deeply Supervised Depth Map Super-Resolution as Novel View Synthesis

Deep convolutional neural network (DCNN) has been successfully applied to depth map super-resolution and outperforms existing methods by a wide margin. However, there still exist two major issues with these DCNN based depth map super-resolution methods that hinder the performance: i) The low-resolution depth maps either need to be up-sampled before feeding into the network or substantial deconvolution has to be used; and ii) The supervision (high-resolution depth maps) is only applied at the end of the network, thus it is difficult to handle large up-sampling factors, such as $\times 8, \times 16$. In this paper, we propose a new framework to tackle the above problems. First, we propose to represent the task of depth map super-resolution as a series of novel view synthesis sub-tasks. The novel view synthesis sub-task aims at generating (synthesizing) a depth map from different camera pose, which could be learned in parallel. Second, to handle large up-sampling factors, we present a deeply supervised network structure to enforce strong supervision in each stage of the network. Third, a multi-scale fusion strategy is proposed to effectively exploit the feature maps at different scales and handle the blocking effect. In this way, our proposed framework could deal with challenging depth map super-resolution efficiently under large up-sampling factors (e.g. $\times 8, \times 16$). Our method only uses the low-resolution depth map as input, and the support of color image is not needed, which greatly reduces the restriction of our method. Extensive experiments on various benchmarking datasets demonstrate the superiority of our method over current state-of-the-art depth map super-resolution methods.

Generalized Rational Sampling Rate Conversion Polyphase FIR Filter

In this letter, we derived a mathematical expression that can help in developing a generalized rational sampling rate conversion polyphase finite impulse response filter, for all relatively prime values of the upsampling ( $L$ ) and downsampling ( $M$ ) factors. In contrast to the existing approaches, the proposed structure efficiently exploits the involved noncausality to eliminate input delay requirements. In addition, the minimization of output delay requirement is presented. A numerical example is also studied to validate the proposed structure. We further evaluate the performance of the proposed structure in terms of total delay requirements ( $\mathcal {D}$ ), multiplication complexity ( $\mathcal {M}$ ), and addition complexity ( $\mathcal {A}$ ). Compared to a similar recent approach, the proposed structure is found to be more efficient in terms of $\mathcal {D}$ for $L and $L>M$ . However, it has reduced computational complexity in terms of $\mathcal {A}$ than the existing approach for $L and same for $L>M$ , whereas both approaches have same $\mathcal {M}$ for $L and $L>M$ .

Non-convex joint bilateral guided depth upsampling

Blurring depth edges and texture copy artifacts are challenging issues for guided depth map upsampling. They are caused by the inconsistency between depth edges and corresponding color edges. In this paper, we extend the well-known Joint Bilateral Upsampling (JBU) (Kopf et al. 2007) with a novel non-convex optimization framework for guided depth map upsampling, which is denoted as Non-Convex JBU (NCJBU). We show that the proposed NCJBU can well handle the edge inconsistency by making use of the property of both the guidance color image and the depth map. Through comprehensive experiments, we show that our NCJBU can preserve sharp depth edges and properly suppress texture copy artifacts. In addition, we present a data driven scheme to properly determine the parameter in our model such that fine details and sharp depth edges are well preserved even for a large upsampling factor (e.g., 8 ×). Experimental results on both simulated and real data show the effectiveness of our method.

Accurate efficient carrier estimation for single-shot digital holographic imaging.

Off-axis digital holograms are commonly processed using the Fourier transform method. This method requires estimation of the carrier frequency peak in the two-dimensional Fourier domain. The accuracy of peak location information using the standard fast Fourier transform (FFT) routines is limited by the pixel size, thus giving rise to a residual carrier component in the reconstructed image. In this Letter, we use an efficient two-step process for cross-term peak location to sub-pixel accuracy where the standard FFT-based peak finding procedure is followed by a high-resolution discrete Fourier transform (DFT) calculation at the nominal peak location. It is shown that the residual tilt artifacts due to a small change in the magnitude of the carrier frequency or due to the rotation of the fringe pattern are uniformly removed with this method. For a digital hologram of size M×N pixels, the high-resolution DFT calculation with an up-sampling factor of α requires an additional computational effort of O(MNα) which is marginal when α≪M,N and, thus, provides a practical single-shot method for accurate demodulation of digital holograms.

A Vision-Based Sensor for Noncontact Structural Displacement Measurement.

Conventional displacement sensors have limitations in practical applications. This paper develops a vision sensor system for remote measurement of structural displacements. An advanced template matching algorithm, referred to as the upsampled cross correlation, is adopted and further developed into a software package for real-time displacement extraction from video images. By simply adjusting the upsampling factor, better subpixel resolution can be easily achieved to improve the measurement accuracy. The performance of the vision sensor is first evaluated through a laboratory shaking table test of a frame structure, in which the displacements at all the floors are measured by using one camera to track either high-contrast artificial targets or low-contrast natural targets on the structural surface such as bolts and nuts. Satisfactory agreements are observed between the displacements measured by the single camera and those measured by high-performance laser displacement sensors. Then field tests are carried out on a railway bridge and a pedestrian bridge, through which the accuracy of the vision sensor in both time and frequency domains is further confirmed in realistic field environments. Significant advantages of the noncontact vision sensor include its low cost, ease of operation, and flexibility to extract structural displacement at any point from a single measurement.

Novel Block-Formulation and Area-Delay-Efficient Reconfigurable Interpolation Filter Architecture for Multi-Standard SDR Applications

A poly-phase based interpolation filter computation involves an input-matrix and coefficient-matrix of size $(P\times M)$ each, where $P$ is the up-sampling factor and $M=N/P$ , $N$ is the filter length. The input-matrix and the coefficient-matrix resizes when $P$ changes. An analysis of interpolation filter computation for different up-sampling factors is made in this paper to identify redundant computations and removed those by reusing partial results. Reuse of partial results eliminates the necessity of matrix resizing in interpolation filter computation. A novel block-formulation is presented to share the partial results for parallel computation of filter outputs of different up-sampling factors. Using the proposed block formulation, a parallel multiplier-based reconfigurable architecture is derived for interpolation filter. The most remarkable aspect of the proposed architecture is that, it does not require reconfiguration to compute filter outputs of an interpolation filter for different up-sampling factor. The proposed structure has regular data-flow and it has no overhead complexity for its reconfigurable feature unlike the existing structures. Besides, the proposed structure has significantly less register complexity than the existing structure and its register complexity is independent of the block-size. Moreover, the proposed structure can support higher input-sampling frequency than the existing structure. ASIC synthesis result shows that the proposed structure for block-size 4, filter length 32, and up-sampling factor 8, involves 13.6 times more area and offers 245 times higher maximum input-sampling frequency compared with the existing multiplier-less structure. It involves 18.6 times less area-delay-product (ADP) and 9.5 times less energy per output (EPO) than the existing multiplier-less structure.

Depth map upsampling using compressive sensing based model

We propose a new method to enhance the lateral resolution of depth maps with registered high-resolution color images. Inspired by the theory of compressive sensing (CS), we formulate the upsampling task as a sparse signal recovery problem that solves an underdetermined system. With a reference color image, the low-resolution depth map is converted into suitable sampling data (measurements). The signal recovery problem, defined in a constrained optimization framework, can be efficiently solved by variable splitting and alternating minimization. Experimental results demonstrate the effectiveness of our CS-based method: it competes favorably with other state-of-the-art methods with large upsampling factors and noisy depth inputs.

Implementation of Linear-Phase FIR Filters for a Rational Sampling-Rate Conversion Utilizing the Coefficient Symmetry

This paper proposes an efficient structure for implementing a linear-phase finite-impulse-response (FIR) filter of an arbitrary order <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex Notation="TeX">$N$</tex></formula> for the sampling-rate conversion by a rational factor of <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex Notation="TeX">$L/M$</tex> </formula> , where <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex Notation="TeX">$L(M)$</tex> </formula> is the integer upsampling (downsampling) factor to be performed before (after) the actual filter. In this implementation, the coefficient symmetry of the linear-phase filter is exploited as much as possible and the number of delay elements is kept as low as possible while utilizing the following facts. When increasing (decreasing) the sampling rate by a factor of <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex Notation="TeX">$L(M)$</tex></formula> , only every <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex Notation="TeX">$L$</tex></formula> th input sample has a nonzero value (only every <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex Notation="TeX">$M$</tex> </formula> th output sample has to be evaluated). In this way, the number of required multiplications per output sample is reduced approximately by a factor of two compared with the conventional polyphase implementation. The proposed implementation is first illustrated using two examples. Based on these examples, guidelines are then given on how to efficiently realize an <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex Notation="TeX">$N$</tex></formula> th-order linear-phase FIR filter for a sampling-rate converter having an arbitrary rational conversion factor <formula formulatype="inline" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex Notation="TeX">$L/M$</tex></formula> . Finally, the implementation complexity of the proposed approach is evaluated and some examples are included, showing the efficiency of the proposed implementation compared with other existing ones.

Improved digital processing method used for image motion measurement based on hybrid opto-digital joint transform correlator

Hybrid opto-digital joint transform correlator (HODJTC) is effective for image motion measurement, but it is different from the traditional joint transform correlator because it only has one optical transform and the joint power spectrum is directly input into a digital processing unit to compute the image shift. The local cross-correlation image can be directly obtained by adopting a local Fourier transform operator. After the pixel-level location of cross-correlation peak is initially obtained, the up-sampling technique is introduced to relocate the peak in even higher accuracy. With signal-to-noise ratio ? 20 dB, up-sampling factor k ? 10 and the maximum image shift ? 60 pixels, the root-mean-square error of motion measurement accuracy can be controlled below 0.05 pixels.

Matrix Structures and Parallel Algorithms for Image Superresolution Reconstruction

Computational resolution enhancement (superresolution) is generally regarded as a memory-intensive process due to the large matrix-vector calculations involved. In this paper, a detailed study of the structure of the $n^2\times n^2$ superresolution matrix is used to decompose the matrix into nine matrices of size $l^2\times l^2$, where l is the upsampling factor. As a result, previously large matrix-vector products can be broken into many small, parallelizable products. An algorithm is presented that utilizes the structural results to perform superresolution on compact, highly parallel architectures such as field-programmable gate arrays.