High-frequency Detail Information Research Articles

A pluggable single-image super-resolution algorithm based on second-order gradient loss

Convolutional neural networks for single-image super-resolution have been widely used with great success. However, most of these methods use L1 loss to guide network optimization, resulting in blurry restored images with sharp edges smoothed. This is because L1 loss limits the optimization goal of the network to the statistical average of all solutions within the solution space of that task. To go beyond the L1 loss, this paper designs an image super-resolution algorithm based on second-order gradient loss. We impose additional constraints by considering the high-order gradient level of the image so that the network can focus on the recovery of fine details such as texture during the learning process. This helps to alleviate the problem of restored image texture over-smoothing to some extent. During network training, we extract the second-order gradient map of the generated image and the target image of the network by minimizing the distance between them, this guides the network to pay attention to the high-frequency detail information in the image and generate a high-resolution image with clearer edge and texture. Besides, the proposed loss function has good embeddability and can be easily integrated with existing image super-resolution networks. Experimental results show that the second-order gradient loss can significantly improve both Learned Perceptual Image Patch Similarity (LPIPS) and Frechet Inception Distance score (FID) performance over other image super-resolution deep learning models.

Blind Super-Resolution Network with Dual-Channel Attention for Images Captured by Sub-Millimeter-Diameter Fiberscope

A blind super-resolution network with dual-channel attention is proposed for images captured by the 0.37 mm diameter sub-millimeter fiberscope. The fiberscope can used in scenarios where other image acquisition devices cannot be applied based on its flexible, soft, and minimally invasive characteristics. However, the images have black reticulated noise and only 3000 pixels. To improve image quality, the Butterworth band-stop filter is used to reduce the frequency of the reticulated noise. By optimizing the blind super-resolution model, high-quality images can be reconstructed that do not require a lot of synthetic paired fiberscope image data. Perceptual loss is utilized as a loss function, and channel and spatial attention mechanisms are introduced to the model to enhance the high-frequency detail information of the reconstructed image. In the comparative experiment with other methods, our method showed improvements of 2.25 in peak signal-to-noise ratio (PSNR) and 0.09 in structural similarity (SSIM) based on objective evaluation metrics. The learned perceptual image patch similarity (LPIPS) based on learning was reduced by 0.6. Furthermore, four different methods were used to enhance the resolution of the fiberscope images by a factor of four. The results of this paper improve the information entropy and Laplace clarity by 0.44 and 2.54, respectively, compared to the average of other methods. Validation results show that the approach in this paper is more applicable to sub-millimeter-diameter fiberscopes.

Improved EfficientDet Algorithm for Basketball Players’ Upper Limb Movement Trajectory Recognition

ABSTRACT Whether a basketball player’s movements are standard or not is a key indicator to evaluate a basketball player and a key point for basketball player training. In order to improve the training effect of basketball players, effective and fast accurate recognition of basketball players’ action trajectories is crucial. In order to solve the problems of inadequate extraction of limb action features and loss of detail information due to multiple scales of limb action features caused by traditional recognition algorithms, an action trajectory recognition algorithm based on the improved EfficientDet-D0 is proposed. The algorithm firstly adds a spatial attention mechanism in the backbone network of EfficientDet-D0, where can locate the limb action trajectory features in the image more accurately. Secondly, in the feature fusion network, in order to describe the high-frequency detail information lost by downsampling, the idea of Laplace pyramid is used to fuse the detail feature maps in the top-down fusion path, and cross-level connections are added to make full use of the feature information of different resolutions, so that the acquired high-level feature map information is richer. Finally, the whole network is trained using the migration learning technique and Adam optimizer. The experimental results show that the model achieves a PCP value (percentage correct parts) of 95.6% on the upper limb pose dataset and takes only 5.34 s for 1000 tests. Compared with the traditional algorithm, it has a higher accuracy rate and stronger robustness, with strong practical applicability, realizing the effective and accurate recognition of basketball players’ upper limb action trajectories in real time.

WDIG: a wavelet domain image generation framework based on frequency domain optimization

In the end-to-end image generation task, the spatial domain of pixel space cannot explicitly separate the low-frequency general information such as texture and color from the high-frequency detail information such as structure and identity. The loss function calculated in the spatial domain fails to effectively constrain the maintenance of detail information, and the generated image quality is insufficient. In this paper, a wavelet domain image generation (WDIG) framework is proposed to preserve the frequency information of images, in which the loss functions are constructed in the pixel space and wavelet space. In the pixel space, the low-frequency and high-frequency characteristic information of the signal are obtained by setting the appropriate Gaussian kernel and adopting the Gaussian fuzzy method. The loss function of ell_{1} norm spatial domain is constructed for the low-frequency and high-frequency characteristic information. In the wavelet space, the corresponding channel sub-band coefficients are obtained by wavelet transform, and the image is explicitly separated into high-frequency information and low-frequency information. The ell_{1} norm frequency domain loss function is constructed respectively for the sub-band coefficients. The WDIG can constrain model training more accurately and optimize model more precisely, so as to better maintain the details and quality of generated image. The WDIG framework is evaluated in the image generation applications including style transfer, image translation and Generative Adversarial Nets (GAN) Inversion. Experimental results show that the WDIG framework can effectively retain the details of images and generate more realistic images, and improve the image quality of the above applications in image generation.

Multi-image hybrid super-resolution reconstruction via interpolation and multi-scale residual networks

High spatial resolution is necessary for several applications such as visual inspection, and can be achieved using high-resolution (HR) image sensors or through image super-resolution (SR) algorithms. Currently, SR algorithms are applied to either single low-resolution (LR) images or multiple LR image sequences. In this paper, we propose a hybrid super-resolution (HYSR) framework to generate HR images by combining multi-image super-resolution (MISR) and single-image super-resolution (SISR) to obtain high spatial resolution images. This method comprehensively utilizes sub-pixel-level high-frequency detail information between multiple images and co-occurrence prior of a single image to reconstruct SR images with a larger scale factor than the existing methods. Generally, the HYSR reconstruction results have more satisfactory details and visual quality than the SISR or MISR reconstruction results. A large number of qualitative and quantitative evaluation results demonstrate the effectiveness and superiority of the HYSR method over traditional MISR and SISR methods.

Multi-scale hybrid attention graph convolution neural network for remote sensing images super-resolution

The super-resolution (SR) method of remote sensing image based on deep learning has achieved satisfactory performance. However, when faced with remote sensing images with large size difference and high complexity, the image SR algorithm based deep learning cannot restore more accurate information, resulting in the reconstructed images with blur problem. To solve this problem, we propose a multi-scale hybrid attention graph convolution neural network for remote sensing image SR (MAGSR). Specifically, we construct a chain module that is composed of a multi-scale hybrid attention module and graph convolution neural network to extract more multi-scale deep features and multi-scale high-frequency detail information. Moreover, we design a hybrid attention mechanism, which combines the advantages of spatial attention and channel attention to learn the spatial dependence on each channel, obtaining more abundant critical high-frequency information. Our proposed MAGSR algorithm obtains the highest PSNR and SSIM values on the Google dataset of SIRI-WHU_earth, UCMerced and NWPU-RESISC45. Compared with the suboptimal algorithm DRCAN, our proposed MAGSR achieve 0.03 dB improvements for PSNR. All experimental results show that our proposed MAGSR is superior to other mainstream SR methods and can reconstruct clear SR remote sensing image with better visual effect.

Super-resolution reconstruction based on Gaussian transform and attention mechanism.

Image super-resolution reconstruction can reconstruct low resolution blurred images in the same scene into high-resolution images. Combined with multi-scale Gaussian difference transform, attention mechanism and feedback mechanism are introduced to construct a new super-resolution reconstruction network. Three improvements are made. Firstly, its multi-scale Gaussian difference transform can strengthen the details of low resolution blurred images. Secondly, it introduces the attention mechanism and increases the network depth to better express the high-frequency features. Finally, pixel loss function and texture loss function are used together, focusing on the learning of structure and texture respectively. The experimental results show that this method is superior to the existing methods in quantitative and qualitative indexes, and promotes the recovery of high-frequency detail information.

Infrared-visible image fusion method based on sparse and prior joint saliency detection and LatLRR-FPDE

In this paper, a new infrared and visible image fusion method based on saliency detection and LatLRR-FPDE is proposed to ensure the saliency of the fused target and improve the clarity of the texture detail information of the latent low-rank representation (LatLRR) fusion method. In the feature representation stage, source images are decomposed into low-rank and sparse parts based on LatLRR. Then Fourth order Partial Differential Equation (FPDE) is used to extract high-frequency detail information from the low-rank part. In the fusion decision stage, according to the characteristic difference of the decomposed sub-bands, a multi-scale fusion scheme is carried out using rules such as EM (expectation-maximization), average (AVG), and the maximum absolute value. To guarantee the integrity of thermal radiation features in the target area in fusion results, an infrared image saliency detection algorithm is proposed, which utilizes the fusion information obtained from the sparse and prior saliency feature. Specifically, the algorithm uses convex hull center prior and contrast prior to obtain the background-suppressed saliency map. Meanwhile, to make full use of the high-dimensional saliency features in the image decomposition stage, the prior saliency map and the sparse saliency map are fused to improve the accuracy of saliency detection without the demand to extract redundant feature, which largely increases time-efficiency of the model. Finally, infrared feature compensation is performed on the target region of the fused image based on the saliency mapping strategy to make the final fused image having a better infrared target feature representation. The experiments are based on registered public datasets and compared with various popular image fusion algorithms. The subjective and objective results show that the proposed algorithm can effectively keep the integrity of salient infrared features and has superior visual quality.

Efficient Image Super-Resolution via Self-Calibrated Feature Fuse.

Recently, many super-resolution reconstruction (SR) feedforward networks based on deep learning have been proposed. These networks enable the reconstructed images to achieve convincing results. However, due to a large amount of computation and parameters, SR technology is greatly limited in devices with limited computing power. To trade-off the network performance and network parameters. In this paper, we propose the efficient image super-resolution network via Self-Calibrated Feature Fuse, named SCFFN, by constructing the self-calibrated feature fuse block (SCFFB). Specifically, to recover the high-frequency detail information of the image as much as possible, we propose SCFFB by self-transformation and self-fusion of features. In addition, to accelerate the network training while reducing the computational complexity of the network, we employ an attention mechanism to elaborate the reconstruction part of the network, called U-SCA. Compared with the existing transposed convolution, it can greatly reduce the computation burden of the network without reducing the reconstruction effect. We have conducted full quantitative and qualitative experiments on public datasets, and the experimental results show that the network achieves comparable performance to other networks, while we only need fewer parameters and computational resources.

Detail-Enhanced Wavelet Residual Network for Single Image Super-Resolution

Single-image super-resolution (SR) is vital in all areas of computer vision, due to the capability of the technology to generate high-resolution (HR) images. Conventional SR approaches do not consider high-frequency detail information during the reconstruction, resulting in high-frequency details of the image unreal, distorted in the reconstructed SR image. In this study, a novel detail-enhanced wavelet residual network (DeWRNet) is proposed to individually deal with the low- and high-frequency of sub-images and resolve the problem of the details over smooth with a novel low-to-high frequency transmission (L2HFT) and detail enhancement (DE) mechanism. Unlike traditional SR approaches, which directly predict high-resolution images, the proposed DeWRNet decomposes the image into low- and high-frequency ones through stationary wavelet transform, and trains low- and high-frequency sub-images with different models. Furthermore, while reconstructing high-frequency details, low-frequency structure is also provided to further restore and enhance high-frequency details by the proposed L2HFT and DE mechanism. Finally, the joint-loss function is used to optimize low- and high-frequency results in different degree of weighting. In addition to correct restoration, image details are further enhanced by adjusting different hyperparameters during training. Compared with the state-of-the-art approaches, the experimental results indicate that the proposed DeWRNet achieves a better performance and has excellent visual presentation, especially in image edges and texture details.

Low-Light-Level Image Super-Resolution Reconstruction Based on a Multi-Scale Features Extraction Network

Wide field-of-view (FOV) and high-resolution (HR) imaging are essential to many applications where high-content image acquisition is necessary. However, due to the insufficient spatial sampling of the image detector and the trade-off between pixel size and photosensitivity, the ability of current imaging sensors to obtain high spatial resolution is limited, especially under low-light-level (LLL) imaging conditions. To solve these problems, we propose a multi-scale feature extraction (MSFE) network to realize pixel-super-resolved LLL imaging. In order to perform data fusion and information extraction for low resolution (LR) images, the network extracts high-frequency detail information from different dimensions by combining the channel attention mechanism module and skip connection module. In this way, the calculation of the high-frequency components can receive greater attention. Compared with other networks, the peak signal-to-noise ratio of the reconstructed image was increased by 1.67 dB. Extensions of the MSFE network are investigated for scene-based color mapping of the gray image. Most of the color information could be recovered, and the similarity with the real image reached 0.728. The qualitative and quantitative experimental results show that the proposed method achieved superior performance in image fidelity and detail enhancement over the state-of-the-art.

Efficient and Model-Based Infrared and Visible Image Fusion via Algorithm Unrolling

Infrared and visible image fusion (IVIF) expects to obtain images that retain thermal radiation information from infrared images and texture details from visible images. In this paper, a model-based convolutional neural network (CNN) model, referred to as Algorithm Unrolling Image Fusion (AUIF), is proposed to overcome the shortcomings of traditional CNN-based IVIF models. The proposed AUIF model starts with the iterative formulas of two traditional optimization models, which are established to accomplish two-scale decomposition, i.e., separating low-frequency base information and high-frequency detail information from source images. Then the algorithm unrolling is implemented where each iteration is mapped to a CNN layer and each optimization model is transformed into a trainable neural network. Compared with the general network architectures, the proposed framework combines the model-based prior information and is designed more reasonably. After the unrolling operation, our model contains two decomposers (encoders) and an additional reconstructor (decoder). In the training phase, this network is trained to reconstruct the input image. While in the test phase, the base (or detail) decomposed feature maps of infrared/visible images are merged respectively by an extra fusion layer, and then the decoder outputs the fusion image. Qualitative and quantitative comparisons demonstrate the superiority of our model, which can robustly generate fusion images containing highlight targets and legible details, exceeding the state-of-the-art methods. Furthermore, our network has fewer weights and faster speed.

A semantic segmentation method with category boundary for Land Use and Land Cover (LULC) mapping of Very-High Resolution (VHR) remote sensing image

ABSTRACT Convolutional Neural Network (CNN) is widely used for semantic segmentation and land-use and land-cover (LULC) mapping of very high-resolution (VHR) remote sensing images. The convolution operation is a powerful method for VHR classification, but the loss of high-frequency detail information caused during its operation decreases the classification accuracy, particularly in the boundary. Thus, it is necessary to supply additional boundary information to the CNN for alleviating this situation. In the classification task (and in LULC mapping), providing more effective information generates a better classification result. Current methods regard the boundary of images as the same category object and process it uniformly, which loses a notable amount of useful information because of the different properties, such as ambiguity and transition, between remote sensing images and their boundaries. Thus, a semantic segmentation method with category boundary for LULC mapping is proposed in this paper. First, a multi-task CNN called the category boundary detection network (CBDN) is designed to extract the boundary information of different category objects. Second, this category boundary and VHR images are used for initial semantic segmentation. Finally, the category boundary and the initial semantic segmentation result (ISSR) are fused to obtain the final LULC map by a two-step strategy, including the explicit fusion and the boundary attention loss function. To verify whether category boundary improved the classification accuracy, a set of comparative experiments were conducted on the International Society for Photogrammetry and Remote Sensing (ISPRS) Vaihingen and Potsdam datasets. The method in this paper was compared with a semantic segmentation method with no boundary information and a semantic segmentation method with global boundary. The results showed that the proposed method in this paper achieved good performances in the Vaihingen (overall accuracy (OA) = 0.924, Kappa coefficient (K) = 0.898, mean F1 score (mF1) = 0.896 and mean Intersection over Union (mIoU) = 0.817) and Potsdam datasets (OA = 0.890, K = 0.857, mF1 = 0.923, and mIoU = 0.860) based on the eroded labels.

A Super Resolution Algorithm Based on Attention Mechanism and SRGAN Network

Image super-resolution reconstruction uses a specific algorithm to restore the low resolution blurred image in the same scene to a high resolution image. In recent years, with the vigorous development of deep learning, this technology has been widely used in many fields. In the field of image super-resolution reconstruction, more and more methods based on deep learning have been studied. According to the principle of GAN, a pseudo high-resolution image is generated by the generator, and then the discriminator calculates the difference between the image and the real high-resolution image to measure the authenticity of the image. Based on SRGAN (super resolution general adverse network), this paper mainly makes three improvements. First, it introduces the attention channel mechanism, that is, it adds Ca (channel attention) module to SRGAN network, and increases the network depth to better express high frequency features; Second, delete the original BN (batch normalization) layer to improve the network performance; Third, modify the loss function to reduce the impact of noise on the image. The experimental results show that the proposed method is superior to the current methods in both quantitative and qualitative indicators, and promotes the recovery of high-frequency detail information. The experimental results show that the proposed method improves the artifact problem and improves the PSNR (peak signal-to-noise ratio) on set5, set10 and bsd100 test sets.

Quantum image encryption algorithm based on Arnold scrambling and wavelet transforms

Based on the modified flexible representation of quantum images, a novel quantum image encryption algorithm was proposed in this paper. The encryption process performs Arnold scrambling operation to disturb the quantum image information in spatial domain first. Then, quantum wavelet transforms are employed to decompose the scrambled quantum image into multiscale resolution (i.e., a sequence of subimages) in the frequency domain, which are mainly divided into two parts: the low-frequency component (i.e., the approximation) and high-frequency detail information (i.e., the horizontal details, vertical details and diagonal details in each decomposition level). Following that, Arnold scrambling operations are implemented to encrypt the wavelet coefficients within each subimage in the frequency domain once again. Finally, based on inverse quantum wavelet transforms, the encrypted wavelet coefficients can affect the pixel values of the entire reconstructed quantum images. Due to the fact that all the quantum operations are invertible, the decryption process of the encrypted image is performed in a straightforward manner by reversing all of the quantum operations within quantum image encryption process. The proposed encryption algorithm is simulated on a classical computer with MATLAB environments. Experimental results and numerical analysis indicate that the presented algorithm has a good encrypted effect and high security.

Multi-Focus Image Fusion Based on Residual Network in Non-Subsampled Shearlet Domain

In order to obtain a panoramic image which is clearer, and has more layers and texture features, we propose an innovative multi-focus image fusion algorithm by combining with non-subsampled shearlet transform (NSST) and residual network (ResNet). First, NSST decomposes a pair of input images to produce subband coefficients of different frequencies for subsequent feature processing. Then, ResNet is applied to fuse the low frequency subband coefficients, and improved gradient sum of Laplace energy (IGSML) perform high frequency feature information processing. Finally, the inverse NSST is performed on the fused coefficients of different frequencies to obtain the final fused image. In our method, we fully consider the low frequency global features and high frequency detail information in image by using NSST. For low-frequency coefficients fusion, we can also obtain the spatial information features of low-frequency coefficient images by using ResNet, which has a deep network structure. IGSML can use different directional gradients to process high-frequency subband coefficients of different levels and directions, which is more conducive to the fusion of the coefficients. The experiment results show that the proposed method has been improved in the structural features and edge texture in the fusion images.

Noise Suppression Method of Microseismic Signal Based on Complementary Ensemble Empirical Mode Decomposition and Wavelet Packet Threshold

Aiming at the situation that complementary ensemble empirical mode decomposition (CEEMD) noise suppression method may produce redundant noise and wavelet transform easily loses high-frequency detail information, considering wavelet packet transform can be used to perform better time-frequency localization analysis on signals containing a large amount of medium and high frequency information, according to the noise and useful signal components of both the characteristic of self-correlation function is different, the CEEMD and wavelet packet threshold jointed method is proposed. The method uses the energy concentration ratio to find noise and useful signal component demarcation point to denoise the microseismic signals. Firstly, we utilize adaptively decompose the signal from high frequency to low frequency by the CEEMD; Secondly, using the self-correlation method to select the intrinsic mode function (IMF) that needs noise suppression, the wavelet suppression method is used to suppress the noise of several high-frequency components whose self-correlation coefficient is below the critical value K; Finally, the IMF component after the wavelet packet threshold noise suppression is reconstructed with the noise-free IMF component. In order to verify the effectiveness of the proposed method on the noise suppression of microseismic signal, we added a Gaussian white noise to the Ricker wavelet signal similar to the microseismic signal. The experimental results show that the signal-to-noise ratio (SNR) of the signal is raised more than 10dB. The energy percentage is higher than 92%. In practical engineering, our proposal achieves an effective noise suppression effect on the microseismic signal.

BUILT-UP AREA DETECTION FROM HIGH-RESOLUTION SATELLITE IMAGES USING MULTI-SCALE WAVELET TRANSFORM AND LOCAL SPATIAL STATISTICS

Abstract. Recently, built-up area detection from high-resolution satellite images (HRSI) has attracted increasing attention because HRSI can provide more detailed object information. In this paper, multi-resolution wavelet transform and local spatial autocorrelation statistic are introduced to model the spatial patterns of built-up areas. First, the input image is decomposed into high- and low-frequency subbands by wavelet transform at three levels. Then the high-frequency detail information in three directions (horizontal, vertical and diagonal) are extracted followed by a maximization operation to integrate the information in all directions. Afterward, a cross-scale operation is implemented to fuse different levels of information. Finally, local spatial autocorrelation statistic is introduced to enhance the saliency of built-up features and an adaptive threshold algorithm is used to achieve the detection of built-up areas. Experiments are conducted on ZY-3 and Quickbird panchromatic satellite images, and the results show that the proposed method is very effective for built-up area detection.

Image Deformation Based on Wavelet Filter and Control Curves

We provide an image deformation method based on wavelet filter using control curves and moving least squares. The image is preprocessed firstly instead of being directly deformed by the old ways. And the original image will be filtered into a high frequency part and a low frequency part by the wavelet, and then use the moving least squares and the control curves only to deform the low frequency part, but not the high frequency part. The key points are set to create control curves according to shape information or deformation requirement, and moved to new position to deform the image using moving least squares. The final result of image deformation is obtained by adding the deformed low frequency part to the high frequency part of the original image. Experiments show that the high frequency detail information is well preserved and the image shape and contour are also well described, and so the deformation results are satisfactory and realistic. DOI : http://dx.doi.org/10.11591/telkomnika.v12i5.4390

A New Control Curve Method for Image Deformation

Different from the direct deformation on the image,we propose a deformation method using wavelet filter and control curves. Firstly, the original image is filtered into a high-frequency subimage and a low-frequency subimage by the wavelet, and the low-frequency subimage is deformed use the moving least squares and the control curves.Then, the key points are set to create control curves according to shape information or deformation requirement, and moved to new position to deform the image using moving least squares. At last, the final deformation image can be obtained by adding the deformed low-frequency part to the high-frequency part. Experiments show that the proprosed method performs very well to preserve high-frequency detail information and describe the image shape and contour, so can obtain the satisfactory and realistic deformation results.