Multiple Low-resolution Images Research Articles

Self-learning based joint multi image super-resolution and sub-pixel registration

Multi Image Super-resolution (MISR) refers to the task of enhancing the spatial resolution of a stack of low-resolution (LR) images representing the same scene. Although many deep learning-based single image super-resolution (SISR) technologies have recently been developed, deep learning has not been widely exploited for MISR, even though it can achieve higher reconstruction accuracy because more information can be extracted from the stack of LR images. One of the primary obstacles encountered by deep networks when addressing the MISR problem is the variability in the number of LR images that act as input to the network. This impedes the feasibility of adopting an end-to-end learning approach, because the varying number of input images makes it difficult to construct a training dataset for the network. Another challenge arises from the requirement to align the LR input images to generate high-resolution (HR) image of high quality, which requires complex and sophisticated methods.In this paper, we propose a self-learning based method that can simultaneously perform super-resolution and sub-pixel registration of multiple LR images. The proposed method trains a neural network with only the LR images as input and without any true target HR images; i.e., the proposed method requires no extra training dataset. Therefore, it is easy to use the proposed method to deal with different numbers of input images. To our knowledge this is the first time that a neural network is trained using only LR images to perform a joint MISR and sub-pixel registration. Experimental results confirmed that the HR images generated by the proposed method achieved better results in both quantitative and qualitative evaluations than those generated by other deep learning-based methods.

A new degradation model and an improved SRGAN for multi-image super-resolution reconstruction

ABSTRACT In order to solve the problems existing in multi-image super-resolution reconstruction methods, such as the difficulty of acquiring and processing multiple low-resolution images, the inability to make full use of the complementary information between different images, and the loss of details, this paper proposes a new degradation model and an improved SRGAN for multi-image super-resolution reconstruction. Firstly, a new degradation model considering both masked autoencoding and downsampling (DMMD) is designed, which can simulate the complex degradation environment in real life well and reduce the difficulty of obtaining and processing multiple low-resolution images. Then, we design a weight setting strategy for image fusion to make full use of the complementary information between different low-resolution images. To enhance the network's attention and propagation of high-frequency effective information in the feature map, the convolution block attention module (CBAM) is introduced into the generator network of SRGAN model, and a SRGAN combined with CBAM (SRGANCBAM) is designed. Finally, the fused low-resolution image is input into SRGANCBAM for reconstruction to obtain its corresponding high-resolution image. The experimental results show that our DMMD can solve the problem that it is difficult to acquire and process multiple low-resolution images. The complementary information between different images is fully utilized by the reasonable weight setting strategy. On four public datasets, the PSNR value of our SRGANCBAM can be 0.532, 0.207, 0.357 and 0.537 higher than the baseline model, respectively. Compared with the state-of-the-art methods, our SRGANCBAM achieves higher values of evaluation metrics and reconstructs clearer and more realistic images.

Multi-image super-resolution based low complexity deep network for image compressive sensing reconstruction

Deep learning (DL) has been widely utilized in image compressive sensing (CS) to enhance the quality and speed of reconstruction. The typical deep network for CS reconstruction comprises an initial reconstruction subnetwork, followed by a cascaded deep refinement reconstruction subnetworks. This paper introduces a new low-complexity image CS deep reconstruction framework, GSRCS, which leverages multi-image based deep super-resolution technology to better address the cost constraints of practical applications. The proposed initial reconstruction module generates multiple low-resolution images in parallel by grouping the input measurements, while a high-quality, high-resolution reconstructed image is produced through a multi-image deep super-resolution network. The theoretical derivation and experimental results demonstrate that this method significantly reduces system complexity in terms of parameters and floating-point arithmetic operations, while achieving competitive reconstruction performance compared to the state-of-the-arts. Specifically, the average number of parameters is reduced by over 63%, and the computational complexity is decreased by more than 88%.

Exponential Fusion of Interpolated Frames Network (EFIF-Net): Advancing Multi-Frame Image Super-Resolution with Convolutional Neural Networks.

Convolutional neural networks (CNNs) have become instrumental in advancing multi-frame image super-resolution (SR), a technique that merges multiple low-resolution images of the same scene into a high-resolution image. In this paper, a novel deep learning multi-frame SR algorithm is introduced. The proposed CNN model, named Exponential Fusion of Interpolated Frames Network (EFIF-Net), seamlessly integrates fusion and restoration within an end-to-end network. Key features of the new EFIF-Net include a custom exponentially weighted fusion (EWF) layer for image fusion and a modification of the Residual Channel Attention Network for restoration to deblur the fused image. Input frames are registered with subpixel accuracy using an affine motion model to capture the camera platform motion. The frames are externally upsampled using single-image interpolation. The interpolated frames are then fused with the custom EWF layer, employing subpixel registration information to give more weight to pixels with less interpolation error. Realistic image acquisition conditions are simulated to generate training and testing datasets with corresponding ground truths. The observation model captures optical degradation from diffraction and detector integration from the sensor. The experimental results demonstrate the efficacy of EFIF-Net using both simulated and real camera data. The real camera results use authentic, unaltered camera data without artificial downsampling or degradation.

Snapshot compressive structured illumination microscopy.

We propose a snapshot compressive structured illumination microscopy (SoSIM) system to increase the number of reconstructed resolution-enhanced (RE) images per second and reduce the data bandwidth by capturing compressed measurements. In this system, multiple low-resolution images are encoded by a high-speed digital micro-mirror device with random binary masks. These images are then captured by a low-speed camera as a snapshot compressed measurement. Following this, we adopt an efficient deep neural network to reconstruct nine images with different structured illumination patterns from a single measurement. The reconstructed images are then combined into a single-frame RE image using the method of spectral synthesis in the frequency domain. When the camera operates at 100 frames per second (fps), we can eventually recover dynamic RE videos at the same speed with 100 fps.

Long-range Fourier ptychographic imaging of the dynamic object with a single camera.

Fourier ptychographic imaging technology is a new imaging method proposed in recent years. This technology captures multiple low-resolution images, and synthesizes them into a high-resolution image in the Fourier domain by a phase retrieval algorithm, breaking through the diffraction limit of the lens. In the field of macroscopic Fourier ptychographic imaging, most of the existing research generally focus on high-resolution imaging of static objects, and applying Fourier ptychographic imaging technology to dynamic objects is a hot research area now. At present, most of the researches are to use camera arrays combined with multiplexed lighting, deep learning or other algorithms, but the implementation of these methods is complicated or costly. Based on the diffraction theory of Fourier optics, this paper proposes that by expanding and focusing the illumination area, we can apply Fourier ptychographic imaging technology with a single camera to moving objects within a certain range. Theoretical analysis and experiments prove the feasibility of the proposed method. We successfully achieve high-resolution imaging of the dynamic object, increasing the resolution by about 2.5 times. This paper also researches the impact of speckles in the illuminated area on imaging results and proposes a processing method to reduce the impact of speckles.

Investigation of Different Pre-processing Quality Enhancement Techniques on X-ray Images

To maximize the accuracy of classification for medical images especially in chest- X ray, we need to improve quality of CXR images or high resolute images will be needed. Pneumonia is a lung infection caused by organism like bacteria or virus. Mostly Chest X-Ray (CXR) is used to detect the infection, but due to limitation of existing equipment, bandwidth, storage space we obtain low quality images. Spatial resolution of medical images is reduced due to image acquisition time, low radiation dose. Quality in medical images plays a major role for clinical diagnosis of disease in deep learning. There is no doubt that noise, low resolution and annotations in chest images are major constraint to the performance of deep learning. Researchers used famous image enhancement algorithms: Histogram equalization (HE), Contrast-limited Adaptive Histogram Equalization (CLAHE), De-noising, Discrete Wavelet Transform (DWT), Gamma Correction (GC), but it is still a challenging task to improve features in images. Computer vision and Super resolution are growing fields of deep learning. Super resolution is also feasible for mono chromatic medical images, which improve the region of interest. Multiple low-resolution images mix with high resolution and then reconstruct a target input image to high quality image by using Super Convolution Neural Network (SRCNN). The objective evaluation based on pixel difference-based PSNR and Human visual system SSIM metric are used for quality measurement. In this study we achieve effective value of PSNR (40 to 43 dB) by considering 30 images of different category (normal, viral or bacterial pneumonia) and SSIM value varies from 97% to 98%. The experiment shows that image quality of CXR is increased by SRCNN, and then high qualitative images will be used for further classification, so that significant parameter of accuracy will be finding in diagnosis of disease in deep learning.

Sparsely sampled Fourier ptychography in diffuse reflection

Fourier ptychography microscopy combines the idea of aperture synthesis, which enables the synthesis of multiple low-resolution images in the frequency domain to achieve large field-of-view and high-resolution imaging results. The Fourier ptychography imaging technique applied to the field of far-field diffuse reflectance imaging needs to consider the impact of diffuse reflectance speckle formed by the object’s rough surface on the imaging results. Therefore, far-field diffuse reflectance Fourier ptychography often requires more than microscopic imaging of low-resolution images to achieve super-resolution. In response, we design a sparse sampling method that can reduce the number of required low-resolution images to one-fourth of the original sampling, and does not affect the imaging resolution. Experimental results show that our method can significantly improve imaging efficiency and achieve super-resolution far-field imaging with fast sampling.

Gradient Prior Dilated Convolution Network for Remote Sensing Image Super-Resolution

Super-resolution (SR) aims to recover highresolution (HR) image from a single or multiple low-resolution (LR) images, compensating for the limitations of satellite sensor imaging. Deep convolutional neural networks (CNN) have made great achievement in remote sensing image super resolution. In this letter, we propose a novel gradient prior dilated convolutional network (GPDCN) for remote sensing images SR, which obtainscontextual spatial connections and alleviates structural distortions. The GPDCN comprise a multi-scale feature extraction network and a feature reconstruction network. The former employs a double-path dilated residual block (DPDRB) with dilation convolution to increase a receptive field, a global selfattention module (GSA) to detect long-range reliance among image patches, and a gradient propagation network (GPN) to extract high-level gradient information. The latter uses the MHOA module to reconstruct the feature by collecting the highorder characteristics of multiple frequency bands. Experiments with the Massachusetts_Roads and 3K VEHICLE_SR datasets demonstrate that the GPDCN outperforms recent techniques concerning both quantitative and qualitative measures.

Fourier ptychographic microscopy with untrained deep neural network priors.

We propose a physics-assisted deep neural network scheme in Fourier ptychographic microscopy (FPM) using untrained deep neural network priors (FPMUP) to achieve a high-resolution image reconstruction from multiple low-resolution images. Unlike the traditional training type of deep neural network that requires a large labelled dataset, this proposed scheme does not require training and instead outputs the high-resolution image by optimizing the parameters of neural networks to fit the experimentally measured low-resolution images. Besides the amplitude and phase of the sample function, another two parallel neural networks that generate the general pupil function and illumination intensity factors are incorporated into the carefully designed neural networks, which effectively improves the image quality and robustness when both the aberration and illumination intensity fluctuation are present in FPM. Reconstructions using simulated and experimental datasets are demonstrated, showing that the FPMUP scheme has better image quality than the traditional iterative algorithms, especially for the phase recovery, but at the expense of increasing computational cost. Most importantly, it is found that the FPMUP scheme can predict the Fourier spectrum of the sample outside synthetic aperture of FPM and thus eliminate the ringing effect of the recovered images due to the spectral truncation. Inspired by deep image prior in the field of image processing, we may impute the expansion of Fourier spectrums to the deep prior rooted in the architecture of the careful designed four parallel deep neural networks. We envisage that the resolution of FPM will be further enhanced if the Fourier spectrum of the sample outside the synthetic aperture of FPM is accurately predicted.

Super-Resolution Electrochemical Impedance Imaging With a 512 × 256 CMOS Sensor Array

Super-resolution imaging is a family of techniques in which multiple lower-resolution images can be merged to produce a single image at higher resolution. While super-resolution is often applied to optical systems, it can also be used with other imaging modalities. Here we demonstrate a 512 × 256 CMOS sensor array for micro-scale super-resolution electrochemical impedance spectroscopy (SR-EIS) imaging. The system is implemented in standard 180 nm CMOS technology with a 10 μm × 10 μm pixel size. The sensor array is designed to measure the mutual capacitance between programmable sets of pixel pairs. Multiple spatially-resolved impedance images can then be computationally combined to generate a super-resolution impedance image. We use finite-element electrostatic simulations to support the proposed measurement approach and discuss straightforward algorithms for super-resolution image reconstruction. We present experimental measurements of sub-cellular permittivity distribution within single green algae cells, showing the sensor's capability to produce microscale impedance images with sub-pixel resolution.

Super-Resolution Images Methodology Applied to UAV Datasets to Road Pavement Monitoring

The increasingly widespread use of smartphones as real cameras on drones has allowed an ever-greater development of several algorithms to improve the image’s refinement. Although the latest generations of drone cameras let the user achieve high resolution images, the large number of pixels to be processed and the acquisitions from multiple lengths for stereo-view often fail to guarantee satisfactory results. In particular, high flight altitudes strongly impact the accuracy, and result in images which are undefined or blurry. This is not acceptable in the field of road pavement monitoring. In that case, the conventional algorithms used for the image resolution conversion, such as the bilinear interpolation algorithm, do not allow high frequency information to be retrieved from an undefined capture. This aspect is felt more strongly when using the recorded images to build a 3D scenario, since its geometric accuracy is greater when the resolution of the photos is higher. Super-Resolution algorithms (SRa) are utilized when registering multiple low-resolution images to interpolate sub-pixel information The aim of this work is to assess, at high flight altitudes, the geometric precision of a 3D model by using the the Morpho Super-Resolution™ algorithm for a road pavement distress monitoring case study.

Research on Low-Resolution Image Fusion Algorithm Based on Deep Learning

The emergence of portable devices provides great convenience for image acquisition, but the image resolution is too low, which affects the identification and use of the image. Multiframe super-resolution algorithm makes a great contribution to extracting image features, but there is a problem with too much computation. Based on this, this paper proposes a low-resolution image fusion algorithm based on deep learning. Based on the introduction of extracting image features from low-resolution images, this paper proposes to fuse multiple low-resolution images, reduce the time of calculation error per frame, reduce the amount of calculation, extract feature information, and improve the alexnet network model to realize image extraction. The convolutional self-coding fusion network is used to realize image fusion and complete the reconstruction and fusion of low-resolution images. In the simulation analysis of the fusion algorithm, objective indexes such as network structure and loss function are selected to evaluate the effectiveness of the algorithm. This paper reduces the computational complexity of image acquisition and ensures that the amount of calculation can be reduced in each iteration. Analyze the effectiveness of the fusion algorithm, comprehensively evaluate the objective indicators, and verify the advantages of the algorithm through indicators such as gradient change and information entropy.

Addressing phase-curvature in Fourier ptychography.

In Fourier ptychography, multiple low resolution images are captured and subsequently combined computationally into a high-resolution, large-field of view micrograph. A theoretical image-formation model based on the assumption of plane-wave illumination from various directions is commonly used, to stitch together the captured information into a high synthetic aperture. The underlying far-field (Fraunhofer) diffraction assumption connects the source, sample, and pupil planes by Fourier transforms. While computationally simple, this assumption neglects phase-curvature due to non-planar illumination from point sources as well as phase-curvature from finite-conjugate microscopes (e.g., using a single-lens for image-formation). We describe a simple, efficient, and accurate extension of Fourier ptychography by embedding the effect of phase-curvature into the underlying forward model. With the improved forward model proposed here, quantitative phase reconstruction is possible even for wide fields-of-views and without the need of image segmentation. Lastly, the proposed method is computationally efficient, requiring only two multiplications: prior and following the reconstruction.

Ultrasound Image Enhancement using Super Resolution

Within today's day and age, amongst women who fall under the bracket of being of reproductive age, when discussion about Gynecologic tumors comes forward, Uterine Fibroids are found to be abundant. Since early detection for tumors is statutory for ensuring recovery, ultrasound detection remains prominent for being one of the most convenient Image-capturing methodologies. Although given the convenience of the method, the quality of the images could be deemed sub-optimal, a need to enhance the image arises for successful detection by appropriate algorithms. This all facilitates the development of a technique with a harmonious combination of filters and Super-Resolution Convolutional neural network (SRCNN).Super Resolution technique in itself is altered to make it feasible for monochromatic medical images, which essentially highlights the contagious aspect, i.e., the tumor region of the image. Thus, firstly the captured noisy image is denoised by making use of Spatial filters, which ensures denoising while preserving the essential edges. Then, this output of the filtered image is treated as an input for the Super-resolution technique, as described above. Within the said technique, multiple low-resolution images are combined in Super-Resolution to regenerate a target input image with higher resolution and quality. Furthermore, Subjective, quantitative, and visual execution evaluations were conducted using Peak Signal-to-Noise-Ratio (PSNR) for quantitative and Universal Quality Index (UQI) and structural similarity index metric (SSIM). Upon completion, these results of comparing the quality of super-resolved images on various data sets are provided in numerical form.These comparisons state that the experimental results have shown a substantial improvement in the results by the fusion technique of hybrid filters with Super Resolution convolutional neural network, for instance, the PSNR value achieved is in the range of 40% to 50%. As a result, the proposed technique has demonstrated outstanding performance in all cases, as evidenced by the findings that the proposed algorithms are robust and ultrasound images taken under various lighting conditions and parameters work efficiently and without egregious errors.

SUPER-RESOLUTION IMAGES ON MOBILE SMARTPHONE AIMED AT 3D MODELING

Abstract. In these last years have been calculated several general algorithms to process the images acquired by mobile devices. Nevertheless, they don't always work at their full effectiveness due the numerous constraints and external issues. The latest generation smartphones, pushed by higher user expectations, have increasingly performing camera functions, in particular way, about the always increasing resolution, with a large number of pixels to process or two cameras for stereo view. Considering that pixels in an image sensor synthetizes the number of incoming photons, taking a photograph with a digital camera means to applying a low-pass filter to a scene where the tiny textures, such as characters that measure a few pixels, are observed blurry. This happens when using digital zoom or when the visibility is compromised or during the night artificial lights. For all these reasons the conventional image converter resolution algorithms (as bilinear interpolation algorithm) don’t work with the high-frequency information of a scene once lost. All these aspects are more relevant if we are taking photos to carry out a 3D scenario. Indeed, the 3D model will have an higher geometric accuracy if the image resolution will be higher. Super-Resolution algorithms (SRa) are classified into two categories: (1) approaches that reconstruct a high-resolution image from itself and (2) approaches that register multiple low-resolution images to interpolate sub-pixel information. In this paper we verify the geometric accuracy of a 3D model, when using the Morpho Super-Resolution™ algorithm, also in critical condition. This algorithm doesn’t require pixel shift, indeed, some cameras have a functionality named pixel shift, which captures multiple images while shifting the image sensor.

Double moiré localized plasmon structured illumination microscopy

Abstract Structured illumination microscopy (SIM) is a well-established fluorescence imaging technique, which can increase spatial resolution by up to a factor of two. This article reports on a new way to extend the capabilities of structured illumination microscopy, by combining ideas from the fields of illumination engineering and nanophotonics. In this technique, plasmonic arrays of hexagonal symmetry are illuminated by two obliquely incident beams originating from a single laser. The resulting interference between the light grating and plasmonic grating creates a wide range of spatial frequencies above the microscope passband, while still preserving the spatial frequencies of regular SIM. To systematically investigate this technique and to contrast it with regular SIM and localized plasmon SIM, we implement a rigorous simulation procedure, which simulates the near-field illumination of the plasmonic grating and uses it in the subsequent forward imaging model. The inverse problem, of obtaining a super-resolution (SR) image from multiple low-resolution images, is solved using a numerical reconstruction algorithm while the obtained resolution is quantitatively assessed. The results point at the possibility of resolution enhancements beyond regular SIM, which rapidly vanishes with the height above the grating. In an initial experimental realization, the existence of the expected spatial frequencies is shown and the performance of compatible reconstruction approaches is compared. Finally, we discuss the obstacles of experimental implementations that would need to be overcome for artifact-free SR imaging.

Multi-Image Super Resolution of Remotely Sensed Images Using Residual Attention Deep Neural Networks

Convolutional Neural Networks (CNNs) consistently proved state-of-the-art results in image Super-resolution (SR), representing an exceptional opportunity for the remote sensing field to extract further information and knowledge from captured data. However, most of the works published in the literature focused on the Single-image Super-resolution problem so far. At present, satellite-based remote sensing platforms offer huge data availability with high temporal resolution and low spatial resolution. In this context, the presented research proposes a novel residual attention model (RAMS) that efficiently tackles the Multi-image Super-resolution task, simultaneously exploiting spatial and temporal correlations to combine multiple images. We introduce the mechanism of visual feature attention with 3D convolutions in order to obtain an aware data fusion and information extraction of the multiple low-resolution images, transcending limitations of the local region of convolutional operations. Moreover, having multiple inputs with the same scene, our representation learning network makes extensive use of nestled residual connections to let flow redundant low-frequency signals and focus the computation on more important high-frequency components. Extensive experimentation and evaluations against other available solutions, either for Single or Multi-image Super-resolution, demonstrated that the proposed deep learning-based solution can be considered state-of-the-art for Multi-image Super-resolution for remote sensing applications.

Super Resolution Based an Adaptive Two Pass Normalized Convolution Using Multiple Low Resolution Images

Super resolution (SR) using signal processing is technique to identify the missing high frequency contents due to limitations of imaging hardware, accomplished by using spatio-temporal information available from LR images. The multi-image SR methods effort to reinstate High Resolution (HR) image commencing numerous wraparound artefacted and degraded Low Resolution (LR) images. In past three decades various researchers contributed in the field, but all are intuitive SR mechanisms. The proposed SR method is based on several LR images of the sight to be super resolved. Registration step of SR applies Keren’s algorithm which is iterative and more accurate. Registration algorithm findings include both translational and rotational motion parameters, where rotation angle is obtained by using an affine transformation produced using Gaussian pyramid. The reconstruction is accomplished by structure-adaptive Normalized Convolution with double pass (SANC-D), using first robust and then adaptive approach. Subjective and objective analysis of the results of reconstruction indicates that the algorithm gives better super resolution images than the others at the expense of execution time of algorithm and it is system dependent.

Super-resolution reconstruction of real infrared images acquired with unmanned aerial vehicle.

Super-resolution (SR) technology provides a far promising computational imaging approach in obtaining a high-resolution (HR) image (or image sequences) from observed multiple low-resolution (LR) images by incorporating complementary information. In this paper, a three-stage SR method is proposed to generate a HR image from infrared (IR) LR Images acquired with Unmanned Aerial Vehicle (UAV). The proposed method integrates a high-level image capturing process and a low-level SR process. In this integrated process, we incorporate UAV path optimization, sub-pixel image registration, and sparseness constraint into a computational imaging framework of a region of interest (ROI). To refine ROI complementary feathers, we design an optimal flight control scheme to acquire adequate image sequences from multi-angles. In particular, a phase correlation approach achieving reliable sub-pixel image feature matching is adapted, on the basis of which an effective sparseness regularization model is built to enhance the fine structures of the IR image. Unlike most traditional multiple-frame SR algorithms that mainly focus on signal processing and achieve good performances when using standard test datasets, the performed experiments with real-life IR sequences indicate the three-stage SR method can also deal with practical LR IR image sequences collected by UAVs. The experimental results demonstrate that the proposed method is capable of generating HR images with good performance in terms of edge preservation and detail enhancement.