Multi-scale Transformation Research Articles

FCSwinU: Fourier Convolutions and Swin Transformer UNet for Hyperspectral and Multispectral Image Fusion.

The fusion of low-resolution hyperspectral images (LR-HSI) with high-resolution multispectral images (HR-MSI) provides a cost-effective approach to obtaining high-resolution hyperspectral images (HR-HSI). Existing methods primarily based on convolutional neural networks (CNNs) struggle to capture global features and do not adequately address the significant scale and spectral resolution differences between LR-HSI and HR-MSI. To tackle these challenges, our novel FCSwinU network leverages the spectral fast Fourier convolution (SFFC) module for spectral feature extraction and utilizes the Swin Transformer's self-attention mechanism for multi-scale global feature fusion. FCSwinU employs a UNet-like encoder-decoder framework to effectively merge spatiospectral features. The encoder integrates the Swin Transformer feature abstraction module (SwinTFAM) to encode pixel correlations and perform multi-scale transformations, facilitating the adaptive fusion of hyperspectral and multispectral data. The decoder then employs the Swin Transformer feature reconstruction module (SwinTFRM) to reconstruct the fused features, restoring the original image dimensions and ensuring the precise recovery of spatial and spectral details. Experimental results from three benchmark datasets and a real-world dataset robustly validate the superior performance of our method in both visual representation and quantitative assessment compared to existing fusion methods.

Multimodal medical image fusion based on interval gradients and convolutional neural networks

Many image fusion methods have been proposed to leverage the advantages of functional and anatomical images while compensating for their shortcomings. These methods integrate functional and anatomical images while presenting physiological and metabolic organ information, making their diagnostic efficiency far greater than that of single-modal images. Currently, most existing multimodal medical imaging fusion methods are based on multiscale transformation, which involves obtaining pyramid features through multiscale transformation. Low-resolution images are used to analyse approximate image features, and high-resolution images are used to analyse detailed image features. Different fusion rules are applied to achieve feature fusion at different scales. Although these fusion methods based on multiscale transformation can effectively achieve multimodal medical image fusion, much detailed information is lost during multiscale and inverse transformation, resulting in blurred edges and a loss of detail in the fusion images. A multimodal medical image fusion method based on interval gradients and convolutional neural networks is proposed to overcome this problem. First, this method uses interval gradients for image decomposition to obtain structure and texture images. Second, deep neural networks are used to extract perception images. Three methods are used to fuse structure, texture, and perception images. Last, the images are combined to obtain the final fusion image after colour transformation. Compared with the reference algorithms, the proposed method performs better in multiple objective indicators of QEN\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$Q_{EN}$$\\end{document}, QNIQE\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$Q_{NIQE}$$\\end{document}, QSD\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$Q_{SD}$$\\end{document}, QSSEQ\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$Q_{SSEQ}$$\\end{document} and QTMQI\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$Q_{TMQI}$$\\end{document}.

A Hybrid Image Fusion and Denoising Algorithm based on Multi-scale Transformation and Signal Sparse Representation

A Hybrid Image Fusion and Denoising Algorithm based on Multi-scale Transformation and Signal Sparse Representation

Enhanced Seamless Video Fusion: A Convolutional Pyramid-Based 3D Integration Algorithm.

Video fusion aims to synthesize video footage from different sources into a unified, coherent output. It plays a key role in areas such as video editing and special effects production. The challenge is to ensure the quality and naturalness of synthetic video, especially when dealing with footage of different sources and qualities. Researchers continue to strive to optimize algorithms to adapt to a variety of complex application scenarios and improve the effectiveness and applicability of video fusion. We introduce an algorithm based on a convolution pyramid and propose a 3D video fusion algorithm that looks for the potential function closest to the gradient field in the least square sense. The 3D Poisson equation is solved to realize seamless video editing. This algorithm uses a multi-scale method and wavelet transform to approximate linear time. Through numerical optimization, a small core is designed to deal with large target filters, and multi-scale transformation analysis and synthesis are realized. In terms of seamless video fusion, it shows better performance than existing algorithms. Compared with editing multiple 2D images into video after Poisson fusion, the video quality produced by this method is very close, and the computing speed of the video fusion is improved to a certain extent.

Human Motion Prediction Based on a Multi-Scale Hypergraph for Intangible Cultural Heritage Dance Videos

Compared to traditional dance, intangible cultural heritage dance often involves the isotropic extension of choreographic actions, utilizing both upper and lower limbs. This characteristic choreography style makes the remote joints lack interaction, consequently reducing accuracy in existing human motion prediction methods. Therefore, we propose a human motion prediction method based on the multi-scale hypergraph convolutional network of the intangible cultural heritage dance video. Firstly, this method inputs the 3D human posture sequence from intangible cultural heritage dance videos. The hypergraph is designed according to the synergistic relationship of the human joints in the intangible cultural heritage dance video, which is used to represent the spatial correlation of the 3D human posture. Then, a multi-scale hypergraph convolutional network is constructed, utilizing multi-scale transformation operators to segment the human skeleton into different scales. This network adopts a graph structure to represent the 3D human posture at different scales, which is then used by the single-scalar fusion operator to spatial features in the 3D human posture sequence are extracted by fusing the feature information of the hypergraph and the multi-scale graph. Finally, the Temporal Graph Transformer network is introduced to capture the temporal dependence among adjacent frames within the time domain. This facilitates the extraction of temporal features from the 3D human posture sequence, ultimately enabling the prediction of future 3D human posture sequences. Experiments show that we achieve the best performance in both short-term and long-term human motion prediction when compared to Motion-Mixer and Motion-Attention algorithms on Human3.6M and 3DPW datasets. In addition, ablation experiments show that our method can predict more precise 3D human pose sequences, even in the presence of isotropic extensions of upper and lower limbs in intangible cultural heritage dance videos. This approach effectively addresses the issue of missing segments in intangible cultural heritage dance videos.

Multi-Scale Ensemble Booster for Improving Existing TSD Classifiers

Time Series Classification (TSC) is an essential task in Time Series Data (TSD) analysis. Ensemble-based approaches now achieve the best performance on TSC tasks. However, integrating numerous different models makes them highly suffer from heavy preprocessing. Even worse, non-deep-learning ensemble-based methods suffer from substantial computational costs due to lacking GPU acceleration. Multi-scale information in TSD can improve TSC performance. However, Existing TSD classifiers employing multi-scale information struggle with heavy preprocessing and cannot help other TSD classifiers obtain multi-scale feature extraction capabilities. Inspired by these, we proposed a performance enhancement framework called multi-scale ensemble booster (MEB), helping existing TSD classifiers achieve performance leaps. In MEB, we proposed an easy-to-combine network structure without changing any of their structure and hyperparameters, only needed to set one hyperparameter, consisting of multi-scale transformation and multi-output decision fusion. Then, a probability distribution co-evolution strategy is proposed to attain the optimal label probability distribution. We conducted numerous ablation experiments of MEB on 128 univariate datasets and 29 multivariate datasets and comparative experiments with 11 state-of-the-art methods, which demonstrated the significant performance improvement ability of MEB and the most advanced performance of the model enhanced by MEB, respectively. Furthermore, to figure out why MEB can improve model performance, we provided a chain of interpretability analyses.

Weighted IMF-based denoising and multi-scale kurtosis weighted K singular value decomposition dictionary learning model for bearing fault diagnosis

Generally, bearing early failure-induced impulses is weak and easily submerged by strong background noise. How to extract weak impulses is a big challenge. In this paper, based on the K Singular Value Decomposition (KSVD) algorithm, an improved dictionary learning model (WID-MSKW-KSVD) is proposed, which integrates a special denoising method and a multi-scale transformation into a typical dictionary learning model and it successfully overcomes two major shortcomings of traditional KSVD model including the lack of multi-scale learning process and the ability of dictionary atoms to resist strong noise interference. Furthermore, the kurtosis of the envelope spectrum is weighted into the proposed model as a fault matrix to enhance the fault feature. Finally, the feasibility of the proposed model is demonstrated by numerical simulation and practical experiments, and the superiority and reliability of this model in suppressing noise interference and extracting fault features are verified by comparison with other state-of-the-art methods.

ECFuse: Edge-Consistent and Correlation-Driven Fusion Framework for Infrared and Visible Image Fusion.

Infrared and visible image fusion (IVIF) aims to render fused images that maintain the merits of both modalities. To tackle the challenge in fusing cross-modality information and avoiding texture loss in IVIF, we propose a novel edge-consistent and correlation-driven fusion framework (ECFuse). This framework leverages our proposed edge-consistency fusion module to maintain rich and coherent edges and textures, simultaneously introducing a correlation-driven deep learning network to fuse the cross-modality global features and modality-specific local features. Firstly, the framework employs a multi-scale transformation (MST) to decompose the source images into base and detail layers. Then, the edge-consistent fusion module fuses detail layers while maintaining the coherence of edges through consistency verification. A correlation-driven fusion network is proposed to fuse the base layers containing both modalities' main features in the transformation domain. Finally, the final fused spatial image is reconstructed by inverse MST. We conducted experiments to compare our ECFuse with both conventional and deep leaning approaches on TNO, LLVIP and M3FD datasets. The qualitative and quantitative evaluation results demonstrate the effectiveness of our framework. We also show that ECFuse can boost the performance in downstream infrared-visible object detection in a unified benchmark.

Retracted: Highway Traffic Flow Prediction Algorithm Based on Multiscale Transformation and Convolutional Networks

Retracted: Highway Traffic Flow Prediction Algorithm Based on Multiscale Transformation and Convolutional Networks

Detecting slow eye movements using multi-scale one-dimensional convolutional neural network for driver sleepiness detection

Background:Slow eye movements (SEMs), which occurs during eye-closed periods with high time coverage rate during simulated driving process, indicate drivers’ sleep onset. New Method:For the multi-scale characteristics of slow eye movement waveforms, we propose a multi-scale one-dimensional convolutional neural network (MS-1D-CNN) for classification. The MS-1D-CNN performs multiple down-sampling processing branches on the original signal and uses the local convolutional layer to extract the features for each branch. Results:We evaluate the classification performance of this model on ten subjects’ standard train–test datasets and continuous test datasets by means of subject-subject evaluation and leave-one-subject-out cross validation, respectively. For the standard train–test datasets, the overall average classification accuracies are about 99.1% and 98.6%, in subject-subject evaluation and leave-one-subject-out cross validation, respectively. For the continuous test datasets, the overall average values of accuracy, precision, recall and F1-score are 99.3%, 98.9%, 99.5% and 99.1% in subject-subject evaluation, are 99.2%, 98.8%, 99.3% and 99.0% in leave-one-subject-out cross validation. Comparison with existing method:Results of the standard train–test datasets show that the overall average classification accuracy of the MS-1D-CNN is quite higher than the baseline method based on hand-designed features by 3.5% and 3.5%, in subject-subject evaluation and leave-one-subject-out cross validation, respectively. Conclusions:These results suggest that multi-scale transformation in the MS-1D-CNN model can enhance the representation ability of features, thereby improving classification accuracy. Experimental results verify the good performance of the MS-1D-CNN model, even in leave-one-subject-out cross validation, thus promoting the application of SEMs detection technology for driver sleepiness detection.

PETNet: A YOLO-based prior enhanced transformer network for aerial image detection

Unmanned aerial vehicles (UAVs) have been applied to inspect in various scenarios due to their high efficiency, low cost, and excellent mobility. However, the objects in aerial images are much smaller and denser than general objects, causing it difficult for current object detection methods to achieve the expected results. To solve this issue, a prior enhanced Transformer network (PETNet) based on YOLO is proposed in this paper. Specifically, a novel prior enhanced Transformer (PET) module and a one-to-many feature fusion (OMFF) mechanism are proposed to embed into the network. Two additional detection heads are added to the shallow feature maps. In this work, PET is used to capture enhanced global information to improve the expressive ability of the network. The OMFF aims to fuse multi-type features to minimize the information loss of small objects. In addition, the added detection heads provide more possibility of detecting smaller-scale objects, and the extended multi-head parallel detection is more suitable for the multi-scale transformation of objects in aerial images. On the VisDrone-2021 and UAVDT databases, the proposed PETNet achieves state-of-the-art results with average precision (AP) of 35.3 and 21.5, respectively, which indicates that the proposed network is more suitable for aerial image detection and is of a great reference value.

Fault Diagnosis of Rotating Machinery Based on 1D–2D Joint Convolution Neural Network

Convolutional neural network has been widely used in fault diagnosis of mechanical devices. In particular, two-dimensional convolutional neural network requires manual selection of multi-scale transformation to transform vibration signal into two-dimensional structure. Although one-dimensional convolution neural network can directly use the vibration signal for convolution processing, it can't make full use of the nonlinear information in one-dimensional space. In order to make full use of the advantages of one-dimensional and two-dimensional convolutional neural networks, this paper develops a one-dimension in tandem with two-dimension joint convolutional neural network (1D-2D JCNN) for rotating machinery fault diagnosis. More specifically, one-dimensional convolution is employed to adaptively obtain the multi-scale feature vectors of the vibration signal, and these feature vectors are constructed into two-dimensional maps, and then these two-dimensional vectors are used as the input of the two-dimensional convolutions neural network. Take the cross-entropy loss function as the loss function, and use the error back propagation algorithm to optimize the filter parameters of the 1D-2D JCNN model to obtain the final fault diagnosis model. Using the motor bearing data set and the worm gearbox data set, the experimental results show the excellent classification performance of bearings and gears under different working conditions. The average diagnostic accuracy of 10 runs on CWRU bearing data set is 99.92%, and the variance is 3.96e-6. The average diagnostic accuracy of 10 runs on worm gear data set is 99.82%, and the variance is 7.82e-6. Compared with the traditional fault diagnosis model and the latest convolution neural network method, the 1D-2D JCNN shows better diagnosis performance.

Fusion of near-infrared and visible images based on saliency-map-guided multi-scale transformation decomposition

Fusion of near-infrared and visible images based on saliency-map-guided multi-scale transformation decomposition

Elastic-band transform for visualization and detection

This paper presents a new multiscale transformation for statistical analysis of one-dimensional data such as time series under the concept of the scale-space approach. The proposed method uses regular observations (eye scanning) with a range of different intervals. The new approach, termed ‘elastic-band transform,’ can be considered as a collection of observations over various intervals (length of elastic-band) of viewing. It is motivated by how people look at an object, such as a sequence of data repeatedly to overview a global structure of the object and find some specific features of it. Some measures based on the transformed elastic-bands are discussed for describing characteristics of data, and multiscale visualizations induced by the measures are developed for a better understanding of data. Several numerical experiments are performed to demonstrate the usefulness of the proposed transform for visualization and detection.

Analysis of the Correlation Properties of the Wavelet Transform Coefficients of Typical Images

The increasing flow of photo and video information transmitted through the channels of infocommunication systems and complexes stimulates the search for effective compression algorithms that can significantly reduce the volume of transmitted traffic, while maintaining its quality. In the general case, the compression algorithms are based on the operations of converting the correlated brightness values of the pixels of the image matrix into their uncorrelated parameters, followed by encoding the obtained conversion coefficients. Since the main known decorrelating transformations are quasi-optimal, the task of finding transformations that take into account changes in the statistical characteristics of compressed video data is still relevant. These circumstances determined the direction of the study, related to the analysis of the decorrelating properties of the generated wavelet coefficients obtained as a result of multi-scale image transformation. The main result of the study was to establish the fact that the wavelet coefficients of the multi-scale transformation have the structure of nested matrices defined as submatrices. Therefore, it is advisable to carry out the correlation analysis of the wavelet transformation coefficients separately for the elements of each submatrix at each level of decomposition (decomposition). The main theoretical result is the proof that the core of each subsequent level of the multi-scale transformation is a matrix consisting of the wavelet coefficients of the previous level of decomposition. It is this fact that makes it possible to draw a conclusion about the dependence of the corresponding elements of neighboring levels. In addition, it has been found that there is a linear relationship between the wavelet coefficients within the local area of ​​the image with a size of 8×8 pixels. In this case, the maximum correlation of submatrix elements is directly determined by the form of their representation, and is observed between neighboring elements located, respectively, in a row, column or diagonally, which is confirmed by the nature of the scattering. The obtained results were confirmed by the analysis of samples from more than two hundred typical images. At the same time, it is substantiated that between the low-frequency wavelet coefficients of the multi-scale transformation of the upper level of the expansion, approximately the same dependences are preserved uniformly in all directions. The practical significance of the study is determined by the fact that all the results obtained in the course of its implementation confirm the presence of characteristic dependencies between the wavelet transform coefficients at different levels of image decomposition. This fact indicates the possibility of achieving higher compression ratios of video data in the course of their encoding. The authors associate further research with the development of a mathematical model for adaptive arithmetic coding of video data and images, which takes into account the correlation properties of wavelet coefficients of a multi-scale transformation.

Highway Traffic Flow Prediction Algorithm Based on Multiscale Transformation and Convolutional Networks

In order to solve the problem that the traditional long-term high-speed traffic forecasting algorithm is affected by the approximation ability of the function and easy to fall into the local mass value, we wrote a multivariate-based highway traffic forecasting algorithm scaling and convolutional networks. Because the feedforward wavelet neural network algorithm predicts the short-term traffic flow in different areas, it is necessary to examine the ability to predict the difference between different models. From the standard feedforward wavelet neural network algorithm using global optimization capabilities, we improve the wolf pack algorithm, improve the search accuracy of the algorithm, get the best solution of the estimated value of the work according to the search results when completing the research objectives, and get the ability to predict the work of the model. Feedforward neural network algorithm: we develop and obtain the best short-term high-speed traffic forecast values. The results are as follows: after using the author’s algorithm, the processing time increases by 1.5 seconds, but the average percentage of errors decreases by more than 50%, in fact the error and the root mean square error decreased by about 30%, and the smoothing coefficient increased by about 1%. The prediction of the author’s algorithm for short-term high-speed traffic is better than the wavelet neural network prediction algorithm, and the prediction accuracy and stability of the author’s algorithm are higher.

Human action recognition using a convolutional neural network based on skeleton heatmaps from two-stage pose estimation

Human action recognition based on skeleton information has been extensively used in various areas, such as human–computer interaction. In this paper, we extracted human skeleton data by constructing a two-stage human pose estimation model, which combined the improved single shot detector (SSD) algorithm with convolutional pose machines (CPM) to obtain human skeleton heatmaps. The backbone of the SSD algorithm was replaced with ResNet, which can characterize images effectively. In addition, we designed multiscale transformation rules for CPM to fuse the information of different scales and a convolutional neural network for the classification of the skeleton keypoints heatmaps to complete action recognition. Indoor and outdoor experiments were conducted on the Caster Moma mobile robot platform, and without an external remote control, the real-time movement of the robot was controlled by the leader through command actions.

Multivariate time series forecasting method based on nonlinear spiking neural P systems and non-subsampled shearlet transform

Multivariate time series forecasting remains a challenging task because of its nonlinear, non-stationary, high-dimensional, and spatial–temporal characteristics, along with the dependence between variables. To address this limitation, we propose a novel method for multivariate time series forecasting based on nonlinear spiking neural P (NSNP) systems and non-subsampled shearlet transform (NSST). A multivariate time series is first converted into the NSST domain, and then NSNP systems are automatically constructed, trained, and predicted in the NSST domain. Because NSNP systems are used as nonlinear prediction models and work in the NSST domain, the proposed prediction method is essentially a multiscale transform (MST)–based prediction method. Therefore, the proposed prediction method can process nonlinear and non-stationary time series, and the dependence between variables can be characterized by the multiresolution features of the NSST transform. Five real-life multivariate time series were used to compare the proposed prediction method with five state-of-the-art and 28 baseline prediction methods. The comparison results demonstrate the effectiveness of the proposed method for multivariate time-series forecasting.

SDDA: a method of surface defect data augmentation of hot-rolled strip steel

The detection of surface defects in hot-rolled strip steel is an important step in the production process of hot-rolled strip steel, and it is closely related to the final quality. At present, many researchers have conducted many in-depth studies on the classification of strip steel surface defects and achieved fruitful results. However, current research has always focused on modifying the model; this approach has limited improvement in classification accuracy and often brings a reduction in model efficiency. A data augmentation method named SDDA, which can perform multiscale transformations on the images input to the deep learning models, is proposed and validated on X-SDD dataset by MixNet model. The test results show that after the SDDA data augmentation, the accuracy of the model reached 96.81%, which is 1.71% higher than the previous best result; meanwhile, since the improvement occurs on the data side, it only leads to an increase in training time and does not increase the inference time of the model. Through comparison experiments, it can be found that our data augmentation approach provides significant improvements in multiple metrics such as accuracy and F1 for a variety of deep learning models.

Image Fusion and Stylization Processing Based on Multiscale Transformation and Convolutional Neural Network.

With the continuous development of imaging sensors, images contain more and more information, the images presented by different types of sensors are different, and the images obtained by the same type of sensors under different parameters or conditions are also different. Multisource image fusion technology combines images acquired by different types of sensors or the same type of sensors with different parameter settings, which makes the image information more complete, compensates for the limitations of images of the same type, and also allows you to save information about the characteristics of the original image. Multimodal image mosaic and multifocal image mosaic have been studied in detail in two directions. On the one hand, a method based on frequency domain transformation is used for multiscale image decomposition. On the other hand, image extraction with neural network-based methods is proposed. The technology of convolutional neural networks (CNNs) allows to extract richer texture features. However, when using this method for fusion, it is difficult to obtain an accurate decision map, and there are artifacts in the fusion boundary. Based on this, a multifocal fusion method based on a two-stage CNN is proposed. Train the advanced intensive network to classify input image blocks as focus, and then use the appropriate merge rules to get the ideal decision tree. In addition, several versions of the fuzzy learning set have been developed to improve network performance. Experimental results show that the frames of the first stage proposed by the algorithm make it possible to obtain an accurate decision scheme and that the frames of the second stage make it possible to eliminate the pseudo-shadow of the integration boundary.