High-resolution Image Patches Research Articles

Enhancing Representation Learning With Spatial Transformation and Early Convolution for Reinforcement Learning-Based Small Object Detection

Although object detection has achieved significant progress in the past decade, detecting small objects is still far from satisfactory due to the high variability of object scales and complex backgrounds. The common way to enhance small object detection is to use high-resolution (HR) images. However, this method incurs huge computational resources which grow squarely with the resolution of images. To achieve both accuracy and efficiency, we propose a novel reinforcement learning framework that employs an efficient policy network consisting of a Spatial Transformation Network to enhance the state representation learning and a Transformer model with early convolution to improve feature extraction. Our method has two main steps: (1) coarse location query (CLQ), where an RL agent is trained to predict the locations of small objects on low-resolution (LR) (down-sampled version of HR) images; (2) context-sensitive object detection where HR image patches are used to detect objects on the selected coarse locations and LR image patches on background areas (containing no small objects). In this way, we can obtain high detection performance on small objects while avoiding unnecessary computation on background areas. The proposed method has been tested and benchmarked on various datasets. On the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Caltech Pedestrians Detection</i> and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Web Pedestrians</i> datasets, the proposed method improves the detection accuracy by 2%, while reducing the number of processed pixels. On the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Vision meets Drone object detection</i> dataset and the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Oil and Gas Storage Tank</i> dataset, the proposed method outperforms the state-of-the-art (SotA) methods. On <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">MS COCO mini-val</i> set, our method outperforms SotA methods on small object detection, while also achieving comparable performance on medium and large objects.

Super-Resolution of Images using Contrastive Self-Learning

Abstract: Super-resolution (SR) techniques aim to enhance the resolution and quality of low-resolution images. In recent years, deep learning-based approaches have achieved remarkable success in this field. However, training deep SR models often requires large-scale datasets and computationally expensive operations, limiting their practicality. Different approaches have been proposed to tackle computation and time constraints. Our novel approach involves a combination of knowledge distillation and contrastive loss function to train a compact and efficient contrastive self-distillation (CSD) model. In the framework, a teacher network is initially trained on a large dataset using a traditional supervised learning approach. The teacher network learns to generate high-resolution images from low-resolution inputs. Subsequently, a student network is initialized with the same architecture as the teacher network but with fewer parameters. The student network is then trained to mimic the behavior of the teacher network by utilizing a contrastive loss. The contrastive loss is formulated by constructing positive and negative pairs of low-resolution and high-resolution image patches. The proposed CSD method achieves competitive performance compared to state-of-the-art SR methods while requiring significantly fewer parameters and computational resources. Furthermore, it demonstrates improved generalization capability by effectively reconstructing details in real-world images.

Single image super‐resolution based on sparse representation using edge‐preserving regularization and a low‐rank constraint

Sparse representation-based non-local self-similarity approaches have demonstrated promising performance in single image super-resolution reconstruction. This type of method, however, cannot always effectively preserve the key details of an image, resulting in edge artifacts and local structure blurring. To better preserve the image edge information, in this paper, a sparse representation super-resolution method based on non-local self-similarity is proposed. First, we impose slide window gradient domain guided filtering on both the low-resolution input image and the degraded restored image. Then, we utilize their difference as an edge-preserving regularization term and incorporate this regularization term into the non-local self-similarity-based sparse representation model to build a sparse coding model, which can enhance the restored high-resolution image patches’ detail information. Finally, the iterative threshold algorithm is used to calculate the sparse representation coefficients so that a high-resolution image can be estimated. Furthermore, to explore the potential structures of the subspaces spanned by similar patches, we enforce a low-rank matrix recovery technique on the generated super-resolution image, which can further refine the reconstruction quality. Experimental results prove that the new approach preserves the critical edge structures while suppressing noise and exceeding some popular methods both quantitatively and qualitatively.

Super resolution image reconstruction via dual dictionary learning in sparse environment

&lt;span lang="EN-US"&gt;Patch-based super resolution is a method in which spatial features from a low-resolution (LR) patch are used as references for the reconstruction of high-resolution (HR) image patches. Sparse representation for each patch is extracted. These coefficients obtained are used to recover HR patch. One dictionary is trained for LR image patches, and another dictionary is trained for HR image patches and both dictionaries are jointly trained. In the proposed method, high frequency (HF) details required are treated as combination of main high frequency (MHF) and residual high frequency (RHF). Hence, dual-dictionary learning is proposed for main dictionary learning and residual dictionary learning. This is required to recover MHF and RHF respectively for recovering finer image details. Experiments are carried out to test the proposed technique on different test images. The results illustrate the efficacy of the proposed algorithm.&lt;/span&gt;

Super-Resolution Thermal Imaging Using Uncooled Infrared Sensors for Non-Destructive Testing of Adhesively Bonded Joints

To compensate for the inability of inexpensive uncooled infrared (IR) sensors to perform high-speed thermography detection with low noise and high sensitivity in the field of non-destructive testing (NDT), a post-processing method for uncooled IR sensors based on compressive sensing is proposed in this paper to perform super-resolution NDT thermography. Super-resolution IR phase images can be generated by first using a sparse representation of the low-resolution (LR) IR images and a sparse dictionary generated from randomly sampled high-resolution (HR) raw training image patches to generate an HR IR image sequence, and, subsequently, implementing IR thermography NDT. To verify the reliability of the proposed method, super-resolution IR images were combined with pulsed phase thermography (PPT) to verify bonded joint structures in carbon fibre-reinforced polymer (CFRP). Compared to other methods, the proposed method reduces the mean square error (MSE) by 11.13% and yields a better visual effect on the PPT phase image reconstructed using three-fold super-resolution. The proposed method not only produces sharp defect edges, but also retains the original texture, which is expected to enable wider application of uncooled IR sensors in NDT applications in the future.

Self-Organized Maps and High-Frequency Image Detail for MRI Image Enhancement

Magnetic resonance imaging (MRI) is a medical technology that uses powerful magnets, radio waves, and a computer to produce images of the body’s internal organs. The patient should be quiet and motionless during the scanning period, as unavoidable movements, such as breathing and heart rate, cause motion artifacts in the image, which cause contrast instability and low-resolution MRI images. Imaging in the clinical setting is performed at low resolution because the scanning time for high-resolution MR imaging would be very long and cumbersome, and imaging is also very expensive. Learning-based image superresolution methods can reconstruct MR images with optimal quality and resolution. However, these methods have problems such as the inability to find the intrinsic relationship between low- and high-resolution image patches in the training dictionary, specification of the proper amount of error in the training and testing stage due to variability in MR image contrast, inability to reconstruct objects by smoothed edges, and use of the backpropagation method in updating their weight vectors. In this paper, we propose a new superresolution method according to competitive learning-based approaches to overcome the problems of previous superresolution methods, which do not have the problems and complexities of those methods. The proposed method includes self-organizing maps as a preprocessor, the nearest neighbor algorithm as a classifier, and a high-frequency filter as a high-frequency image detail extractor. We constructed a single external dictionary from a combination of low-resolution and high-resolution feature patches and trained our SOM network. Next, we reconstruct the high-resolution image by converting the low-resolution input image into feature patch vectors, and for each vector, we find all corresponding neurons in the network and retrieve all their training feature vectors. Next, we train the nearest neighbor algorithm with the recovered vectors plus the input vector and find the best similarity vector to the input vector. After finding all the best similarity vectors to the input vectors, we reconstruct our high superresolution image. The proposed image superresolution method in practical experiments was trained, tested, and evaluated by the Div2k dataset and compared with other traditional and state-of-the-art image enhancement methods on various datasets, such as SET5, SET14, BSDS100, and URBAN100, and presented better results with higher accuracy and quality than traditional and state-of-the-art methods, both visually compared to each other by human and computational benchmarks, such as the peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM), to compare image superresolution algorithms. This method is best for reconstructing high-resolution magnetic resonance images that require high-frequency details and sharp edges with a smooth slope of the imaging objects in their structures. The execution time of the proposed algorithm is slower than that of the other algorithms, so we use GPU hardware and parallel programming techniques to increase the algorithm speed.

Hypergraph-regularized sparse representation for single color image super resolution

Sparsity-based single image super resolution method generates the High-Resolution (HR) output via a corresponding dictionary from the Low-Resolution (LR) input. However, most of these existing methods ignore the complementary information from color channels, which causes the loss of a valid prior and the limitation of HR image quality improvement. In this paper, hypergraph regularization is first incorporated with Joint Color Dictionary Training (JCDT) model and HR image reconstruction (HRIR) model. A novel Hypergraph-regularized Sparse coding-based Super Resolution (HG-ScSR) is proposed. This regularization can not only focus on the illuminance information, but also exploit the self-channel and cross-channel information of three color RGB channels from high-resolution image patches. Especially, the complex relationship is explored among every color image patch pixel and the consistency of the similar pixels is enforced. Both simulated and real data experiments verify the higher performance of the proposed HG-ScSR.

A Real-Time Learning-Based Super-Resolution System on FPGA

This paper proposes a real-time super-resolution (SR) system. The proposed system performs a fast SR algorithm that generates a high-resolution image from a low-resolution image using direct regression functions with an up-scaling factor of 2. This algorithm contained two processes: feature learning and SR image prediction. The feature learning stage is performed offline, in which several regression functions were trained. The SR image prediction stage is implemented on the proposed system to generate high-resolution image patches. The system implemented on a Xilinx Virtex 7 field-programmable gate array achieves output resolution of [Formula: see text] (UHD) at 85 fps and 700Mpixels/s throughput. Structure similarity (SSIM) is measured for image quality. Experimental results show that the proposed system provides high image quality for real-time applications. And the proposed system possesses high scalability for resolution.

Fast multi‐spectral image super‐resolution via sparse representation

Sparse reconstruction is used to solve the inverse problem of single image super-resolution (SR) as a patch-based sparsity promoting regularisation problem. A coupled trained overcomplete dictionary from high-resolution (HR) and low-resolution (LR) image patches containing significant features is proposed using sparse representation to produce HR patches from their LR counterparts. In this study, the authors develop a multi-core single image SR technique for LR multi-spectral images based on patch-wise sparse representation coupled with morphological component analysis driven feature extraction. Simulations are carried out to evaluate the proposed method using real remote sensing images of a few Indian satellites, RESOURCESAT-2 and CARTOSAT-2, as well as other satellites, such as QuickBird etc. Results are also compared with other existing SR methods to establish the superiority of the proposed method in terms of both objective metrics and visual analysis.

Sparse representation based computed tomography images reconstruction by coupled dictionary learning algorithm

It is very interesting to reconstruct high-resolution computed tomography (CT) medical images that are very useful for clinicians to analyse the diseases. This study proposes an improved super-resolution method for CT medical images in the sparse representation domain with dictionary learning. The sparse coupled K-singular value decomposition (KSVD) algorithm is employed for dictionary learning purposes. Images are divided into two sets of low resolution (LR) and high resolution (HR), to improve the quality of low-resolution images, the authors prepare dictionaries over LR and HR image patches using the KSVD algorithm. The main idea behind the proposed method is that sparse coupled dictionaries learn about each patch and establish the relationship between sparse coefficients of LR and HR image patches to recover the HR image patch for LR image. The proposed method is compared to conventional algorithms in terms of mean peak signal-to-noise ratio and structural similarity index measurements by using three different data set images, including CT chest, CT dental and CT brain images. The authors also analysed the proposed improved method for different dictionary sizes and patch size to obtain a similar high-resolution image. These parameters play an essential role in the reconstruction of the HR images.

Image interpolation with adaptive k ‐nearest neighbours search and random non‐linear regression

Learning-based image interpolation methods have been proved to be effective in image interpolation. In this study, the authors propose an accurate image interpolation with adaptive k-nearest neighbour searching and non-linear regression. The proposed method aims to find k-nearest neighbours of the input image patch and use them to learn the non-linear mapping between low-resolution and high-resolution image patches. To be specific, they first divide the training image patches into many subspaces, then they utilise an adaptive robust and precise k nearest neighbour searching scheme with proposed normalised Gaussian similarity to find the k nearest neighbours in the matched subspace. The selected k image patch pairs are then used to learn the non-linear regression model through an extreme learning machine. Furthermore, the proposed interpolation method is a cascade framework that consists of two stages. Stage 2 takes the results of Stage 1 as input to further improve the performance. Extensive experimental results on commonly used test images and image datasets indicate that their proposed algorithm obtains competitive performance against the state-of-the-art methods both in terms of objective evaluation values and the subjective effect of reconstructed images.

Cost-Efficient Super-Resolution Hardware Using Local Binary Pattern Classification and Linear Mapping for Real-Time 4K Conversion

We propose a new hardware-friendly super-resolution (SR) algorithm using computationally simple feature extraction and regression methods, i.e., local binary pattern (LBP) and linear mapping, respectively. The proposed method pre-trains dedicated linear mapping kernels for different texture types of low-resolution (LR) image patches where the texture type is classified based on LBP features. On inference operation, a high-resolution (HR) image patch is reconstructed by multiplying an LR image patch with a linear mapping kernel, which is inferred by the LBP feature class of the corresponding LR patch. Since, the LBP is a highly efficient feature extraction operator for local texture classification, our method is extremely fast and power-efficient while showing competitive reconstruction quality to the latest machine learning-based SR techniques. We also present a fully pipe-lined hardware architecture and its implementation for real-time operations of the proposed SR method. The proposed SR algorithm has been implemented on a field-programmable-gate-array (FPGA) platform including Xilinx KCU105 that can process 63 frames-per-second (fps) while converting full-high-definition (FHD) images to 4K ultra-high-definition (UHD) images. Extensive experimental results show that the proposed proposed algorithm and its hardware implementation can achieve high reconstruction performance compared to the latest machine-learning-based SR methods while utilizing minimum hardware resources, thereby having remarkably less computational complexity. Sometimes, the latest deep-learning-based SR approaches offer slightly higher reconstruction quality, but they require significantly larger amount of hardware resources than the proposed method.

Single‐image super‐resolution using online kernel adaptive filters

The online kernel adaptive filters are non-linear filters which provide impulse response and are more efficient compared to other kernel algorithms. The performance of kernel adaptive filters depends on dictionary size. Here the single-image super-resolution using online kernel adaptive filters is a learning-based method. The algorithm generates a sparser solution for obtaining high-resolution image from a low-resolution image. It finds out a dictionary with most significant set of basis vectors using the spatial similarity among the dictionaries created from the low-resolution and high-resolution image patches in the training set. The dictionary is utilised to generate the high-resolution image. The algorithm is analysed on three different kernel adaptive filters, extended kernel recursive least squares, kernel recursive least squares tracker and naive online regularised risk minimisation algorithm. The performance of the super-resolution method is evaluated on a large number of images and is compared with the state-of-the art non-linear solutions to the super-resolution. The results show a better progress in peak signal-to-noise ratio up to 1.2 dB.

A U-net based approach to epidermal tissue segmentation in whole slide histopathological images

Malignant melanoma is a severe and aggressive type of skin cancer, with a rapid decrease in survival rate if not diagnosed and treated at an early stage. Histopathological examination of hematoxylin and eosin stained tissue biopsies under a light microscope is currently the gold standard for diagnosis. However, this manual examination is a difficult and time-consuming task, and diagnosis is often subject to intra- and inter-observer variability. With more pathology departments starting to convert conventional glass slides into digital resources, a Computer Aided Diagnostic (CAD) system that can automate part of the diagnostic process will help address these challenges. It is expected to reduce examination time, increase diagnostic accuracy, and reduce diagnostic variations. An important initial step in developing such a system is an automated epidermis segmentation algorithm, since several important diagnostic factors are within or seen relatively to the epidermis’ location. In this paper, we propose a new epidermis segmentation technique built on Convolutional Neural Networks. We trained an U-net based architecture end-to-end, with sim 380hbox {k} overlapping high resolution image patches at 512 times 512 pixels, extracted and augmented from 36 digitized histopathological images from two different clinical sites, to discriminate pixels as either epidermal or non-epidermal. The proposed technique was evaluated on 33 test images, where we achieved a mean Positive Predictive Value at 0.89pm 0.16, Sensitivity at 0.92pm 0.1, Dice Similarity Coefficient at 0.89pm 0.13 and a Matthews Correlation Coefficient at 0.89pm 0.11, showing a superior performance when compared to existing techniques. Our algorithm also proves to be robust to variations in staining, tissue thickness and laboratory pre-processing.

Learning Shared and Cluster-Specific Dictionaries for Single Image Super-Resolution

Recently, many multi-dictionary-based sparse representation methods have been proposed for single-image super-resolution (SISR) by learning a sub-dictionary for each specific type of visual content (i.e., a cluster). Although promising reconstruction results have been achieved for certain scenarios, due to the similarity shared among different types of visual content, each sub-dictionary is not learned as discriminatively as expected, which could compromise the visual quality of reconstructed high-resolution (HR) images. In this paper, we propose a novel shared and cluster-specific dictionaries learning method for SISR, where cluster-specific dictionaries can be learned as separate as possible and a dictionary shared among clusters is learned explicitly to explore similarity among clusters. First, low-resolution (LR) and HR image patches extracted from training images are jointly grouped into visual clusters. Then, a shared sub-dictionary and a set of cluster-specific sub-dictionaries are learned simultaneously under the sparse representation framework. In addition, the group sparsity constraint, the locality constraint, and the incoherence penalty term are incorporated into a unified framework to preserve the relationship and structure among the training data in dictionary learning. Finally, an anchored neighborhood regression method is devised to pre-calculate a projection matrix of each learned dictionary atom from LR to its HR space so that an HR image can be more efficiently recovered in the reconstruction stage. Comprehensive experimental results on two widely used benchmark datasets demonstrate the effectiveness and robustness of our method against a number of state-of-the-art methods. The reconstruction results of real-world maritime surveillance images further indicate that the proposed method is well suitable for practical applications.

Automated extraction and discrimination of open land areas from IRS-1C LISS III imagery

Availability of open land area is a key indicator in assessing and planning urban environments. However, accurate uninhabited surface extraction is still a challenge. In this paper an attempt is made to combine the advantages of partial differential equation (PDE) and random forest (RF) method for segmentation of IRS-1C LISS III satellite image for mapping and discrimination of open land areas. Spatial variations of the pixels in the image are analyzed to identify open land pixels. PDE method denoise and conserve finer details simultaneously by using correlation between the spectral bands and directions of the edges. Low-resolution resultant image patch is mapped to high-resolution image patch by linear regression model. Variance of classification is reduced by training many classifiers using interpolated RF method. This methods elevates the accuracy of the direct RF method and achieves 3.35 dB improvement in PSNR and 6.47 reduction in MSE. Further, discrimination of open land areas is done into distinct classes, using inherent spatial information. Accuracy assessment indicates an overall accuracy of 87% over direct RF method.

Structural-Correlated Self-Examples Based Superresolution of Single Remote Sensing Image

Image superresolution methods are of great importance to image analysis and interpretation and have been intensively studied and widely applied. The main research works on single-image superresolution are how to construct the training image database and how to learn the mapping relationship between low- and high-resolution images. Considering only a single image, a novel super-resolution method for self-examples learning without depending on any external training images is proposed in this paper. The training self-examples are extracted from the gradually degraded versions of the testing image and their corresponding interpolated counterparts to build internal high- and low-resolution training databases. Inspired by the concept of “coarse-to-fine,” the upscaling process is performed gradually as well. The algorithm includes two steps during each upscaling procedure. For each low-resolution patch, the first step is to find structural-correlated patches by sparse representation throughout the training database to learn global linear mapping function between low- and high-resolution image patches without any assumption on the data, and the second step takes the advantage of sparse representation as a local constraint on super-resolution result. At each upscaling procedure, iterative back projection is applied to guarantee the consistency of the estimated image. Moreover, the internal training database will be updated according to the newly generated upscaled image. Experiments show that the proposed algorithm can achieve good performance on peak signal-to-noise ratio and structural similarity index and produce excellent visual effects compared with other super-resolution methods.

Super-resolution CT Image Reconstruction Based on Dictionary Learning and Sparse Representation

In this paper, a single-computed tomography (CT) image super-resolution (SR) reconstruction scheme is proposed. This SR reconstruction scheme is based on sparse representation theory and dictionary learning of low- and high-resolution image patch pairs to improve the poor quality of low-resolution CT images obtained in clinical practice using low-dose CT technology. The proposed strategy is based on the idea that image patches can be well represented by sparse coding of elements from an overcomplete dictionary. To obtain similarity of the sparse representations, two dictionaries of low- and high-resolution image patches are jointly trained. Then, sparse representation coefficients extracted from the low-resolution input patches are used to reconstruct the high-resolution output. Sparse representation is used such that the trained dictionary pair can reduce computational costs. Combined with several appropriate iteration operations, the reconstructed high-resolution image can attain better image quality. The effectiveness of the proposed method is demonstrated using both clinical CT data and simulation image data. Image quality evaluation indexes (root mean squared error (RMSE) and peak signal-to-noise ratio (PSNR)) indicate that the proposed method can effectively improve the resolution of a single CT image.

The Generalized Gamma-DBN for High-Resolution SAR Image Classification

With the increase of resolution, effective characterization of synthetic aperture radar (SAR) image becomes one of the most critical problems in many earth observation applications. Inspired by deep learning and probability mixture models, a generalized Gamma deep belief network (g &Gamma;-DBN) is proposed for SAR image statistical modeling and land-cover classification in this work. Specifically, a generalized Gamma-Bernoulli restricted Boltzmann machine (g&Gamma;B-RBM) is proposed to capture high-order statistical characterizes from SAR images after introducing the generalized Gamma distribution. After stacking the g &Gamma; B-RBM and several standard binary RBMs in a hierarchical manner, a g&Gamma;-DBN is constructed to learn high-level representation of different SAR land-covers. Finally, a discriminative neural network is constructed by adding an additional predict layer for different land-covers over the constructed deep structure. Performance of the proposed approach is evaluated via several experiments on some high-resolution SAR image patch sets and two large-scale scenes which are captured by ALOS PALSAR-2 and COSMO-SkyMed satellites respectively.

Image Super-resolution with Parametric Sparse Model Learning.

Recovering a high-resolution (HR) image from its low-resolution (LR) version is an ill-posed inverse problem. Learning accurate prior of HR images is of great importance to solve this inverse problem. Existing super-resolution (SR) methods either learn a non-parametric image prior from training data (a large set of LR/HR patch pairs) or estimate a parametric prior from the LR image analytically. Both methods have their limitations: the former lacks flexibility when dealing with different SR settings; while the latter often fails to adapt to spatially varying image structures. In this paper, we propose to take a hybrid approach toward image SR by combining those two lines of ideas - that is, a parametric sparse prior of HR images is learned from the training set as well as the input LR image. By exploiting the strengths of both worlds, we can more accurately recover the sparse codes and therefore HR image patches than conventional sparse coding approaches. Experimental results show that the proposed hybrid SR method significantly outperforms existing model-based SR methods and is highly competitive to current state-of-the-art learning-based SR methods in terms of both subjective and objective image qualities.