Spatial Transformer Network Research Articles

A Study of Artificial Intelligence-Based Poster Layout Design in Visual Communication

In visual communication design, the basic design elements include four types of text, graphics, colour, and layout. While the first three can be called visual elements, layout design is the functional arrangement of visual elements. Layout design is as much about making the viewer receive visual information as it is about making them feel attractive. The combination of artificial intelligence and layout design has now become a popular direction in the field of visual communication design. However, the automatic layout design process achieved through an a priori design framework still requires human involvement and is a semi-intelligent application. To solve the above problems, this study proposes a poster layout design method based on artificial intelligence. The layout composition method consists of a learner and a generator. Firstly, the learner uses the spatial transformation network to learn the classification of layout composition elements and form the initial layout design templates for different composition cases. Secondly, the generator optimises the initial template based on the LeNet architecture using the golden ratio and trilateration parameters to produce multiple optimised templates. The templates are then stored in a library of corresponding templates according to their composition and framing style. The experimental results show that the proposed poster layout composition method achieves a higher accuracy rate than existing methods.

Unsupervised Deep Learning for FOD-Based Susceptibility Distortion Correction in Diffusion MRI.

Susceptibility induced distortion is a major artifact that affects the diffusion MRI (dMRI) data analysis. In the Human Connectome Project (HCP), the state-of-the-art method adopted to correct this kind of distortion is to exploit the displacement field from the B0 image in the reversed phase encoding images. However, both the traditional and learning-based approaches have limitations in achieving high correction accuracy in certain brain regions, such as brainstem. By utilizing the fiber orientation distribution (FOD) computed from the dMRI, we propose a novel deep learning framework named DistoRtion Correction Net (DrC-Net), which consists of the U-Net to capture the latent information from the 4D FOD images and the spatial transformer network to propagate the displacement field and back propagate the losses between the deformed FOD images. The experiments are performed on two datasets acquired with different phase encoding (PE) directions including the HCP and the Human Connectome Low Vision (HCLV) dataset. Compared to two traditional methods topup and FODReg and two deep learning methods S-Net and flow-net, the proposed method achieves significant improvements in terms of the mean squared difference (MSD) of fractional anisotropy (FA) images and minimum angular difference between two PEs in white matter and also brainstem regions. In the meantime, the proposed DrC-Net takes only several seconds to predict a displacement field, which is much faster than the FODReg method.

A hybrid attention mechanism for blind automatic modulation classification

AbstractRecently, deep leaning has been making great progress in automatic modulation classification, just like its success in computer vision. However, radio signals with harsh impairments (oscillator drift, clock drift, noise) would significantly degrade the performance of the existing classifiers. To overcome the problem and explore the depth reason, a hybrid attention convolution network is proposed to enhance the capability of feature extraction. First, a spatial transformer network module with long short‐term memory is introduced to synchronize and normalize radio signals. Second, a channel attention module is constructed to weight and assemble feature maps, exploring global feature representations with more context‐relevant information. By combining these two modules, a relatively lightweight classifier with complex convolution layer for final classification is further researched through visualization. Moreover, different structures of attention module are compared and optimized in detail. Experimental result shows that our proposed hybrid model achieves the best performance among all compared models when SNR is upper than 7 dB, and it peaks at 93.448 at 0 dB, 2.7% higher than that of CLDNN and 97.560 at 20 dB, 8.2% higher than that of ResNet. And our model can be more efficient after a trade‐off between accuracy and model size.

The Self-Supervised Spectral–Spatial Vision Transformer Network for Accurate Prediction of Wheat Nitrogen Status from UAV Imagery

Nitrogen (N) fertilizer is routinely applied by farmers to increase crop yields. At present, farmers often over-apply N fertilizer in some locations or at certain times because they do not have high-resolution crop N status data. N-use efficiency can be low, with the remaining N lost to the environment, resulting in higher production costs and environmental pollution. Accurate and timely estimation of N status in crops is crucial to improving cropping systems’ economic and environmental sustainability. Destructive approaches based on plant tissue analysis are time consuming and impractical over large fields. Recent advances in remote sensing and deep learning have shown promise in addressing the aforementioned challenges in a non-destructive way. In this work, we propose a novel deep learning framework: a self-supervised spectral–spatial attention-based vision transformer (SSVT). The proposed SSVT introduces a Spectral Attention Block (SAB) and a Spatial Interaction Block (SIB), which allows for simultaneous learning of both spatial and spectral features from UAV digital aerial imagery, for accurate N status prediction in wheat fields. Moreover, the proposed framework introduces local-to-global self-supervised learning to help train the model from unlabelled data. The proposed SSVT has been compared with five state-of-the-art models including: ResNet, RegNet, EfficientNet, EfficientNetV2, and the original vision transformer on both testing and independent datasets. The proposed approach achieved high accuracy (0.96) with good generalizability and reproducibility for wheat N status estimation.

Artificial Intelligence Registration of Image Series Based on Multiple Features

Multi-source image series vary in quality. To fuse the feature information of multi-source image series, it is necessary to deeply explore the relevant registration and fusion techniques. The existing techniques of image registration and fusion lack a unified multi-feature-based algorithm framework, and fail to achieve real-time accurate registration. To solve these problems, this paper probes into the artificial intelligence (AI) registration of image series based on multiple features. Firstly, the Harris corner detector was selected to extract the corners of multi-source image series, before explaining and improving the flow of the algorithm. In addition, the deep convolutional neural network (DCNN) VGG16 was improved to extract the features of multi-source image series. Finally, the spatial transformation network was adopted to pre-register the image series, and the image series was deformed and restored based on the region-constrained moving least squares. The proposed registration algorithm was proved effective through experiments.

Micro-Doppler Extraction of Radar Targets With Translational Motion Based on Spatial Transformer Network

The micro-Doppler (MD) signature, an effect that is induced by micro-motion, characterizes the rich motion information of targets, making it a valuable tool for target classification and recognition. However, the MD structure is always destroyed for radar targets with translational motion. In this letter, we propose an end–to–end translational motion compensation network based on a spatial transformer network (STN). First, the effect of the translational motion on the radar echo and its time-frequency graph (TFG) is modeled, so that translational motion compensation is regarded as an image spatial transformation task. Then, a two-branch convolutional network based on residual modules is designed to locate the transformation parameters, and the TFG containing only MD is obtained using a grid generator and sampler. Finally, the simulation results demonstrate that the proposed algorithm has high accuracy and stability.

PFVNet: A Partial Fingerprint Verification Network Learned From Large Fingerprint Matching

With the decreasing size of fingerprint scanners in portable devices, e.g., mobile phone and smart watch, partial fingerprint recognition has become a challenging and urgently needed technique due to the limited features contained in small area as well as the large rotation and translation between query and reference images. Deep learning as a powerful modeling method has advanced the research progress of fingerprint recognition, but it still suffers from the lack of labeled data in the scenario of partial fingerprint matching. In this paper, we propose a novel partial fingerprint verification network (PFVNet) based on spatial transformer network (STN) and the local self-attention mechanism. Our model can be trained end-to-end and learn multi-level fingerprint features automatically. To alleviate the data annotation work, the model is trained in a self-supervision and domain adaptation manner with data generated from large fingerprint image matching. The experimental results compared with other methods on FVC2006 DB1 dataset and in-house datasets (i.e., ZJUPartial database) show that our method achieves state-of-the-art performance, and also robust to different types of scanners.

Recurrent Affine Transform Encoder for Image Representation

This paper proposes a Recurrent Affine Transform Encoder (RATE) that can be used for image representation learning. We propose a learning architecture that enables a CNN encoder to learn the affine transform parameter of images. The proposed learning architecture decomposes an affine transform matrix into two transform matrices and learns them jointly in a self-supervised manner. The proposed RATE is trained by unlabeled image data without any ground truth and infers the affine transform parameter of input images recurrently. The inferred affine transform parameter can be used to represent images in canonical form to greatly reduce the image variations in affine transforms such as rotation, scaling, and translation. Different from the spatial transformer network, the proposed RATE does not need to be embedded into other networks for training with the aid of other learning objectives. We show that the proposed RATE learns the affine transform parameter of images and achieves impressive image representation results in terms of invariance to translation, scaling, and rotation. We also show that the classification performance is enhanced and is more robust against distortion by incorporating the RATE into the existing classification model.

Shape Based Deep Estimation of Future Plant Images

Plants exhibit dynamic changes as they grow. For example, a new leaf may appear suddenly, and rotate and fold over time. Therefore, it is difficult to predict the growth of plants. For accurate growth predictions, it is important to predict the shapes, colors and textures of the leaves. The conventional methods simply use RGB images to predict the next plant at once. In this paper, we propose a novel deep network which is divided into two subnets of shape estimation and color reconstruction. Four gray time-series images are first aligned to a future target using a spatial transformer network (STN) for shape estimation. They are then fused using U-Net with two LSTMs to generate a future shape image. The color reconstruction subnet fuses the predicted shape with a RGB plant image to restore the color information. In addition, we use gray images with texture information for shape estimation instead of binary images with only simple information and RGB images with too much information. The proposed deep network can robustly generate future plant images for plant growth prediction. It is evaluated using our proprietary dataset as well as two public datasets for different types of plants. The experimental results demonstrate that our proposed network predicts the leaf shape more accurately and restores RGB. As a result, our method can create accurate future plant images.

A Lightweight Top-Down Multi-Person Pose Estimation Method Based on Symmetric Transformation and Global Matching

The top-down human pose estimation method usually faces the following problems: (i) The target detection result is not well applied in the pose estimation network. (ii) Difficulty of human detection in the crowded state. (iii) The complicated model leads to a long training time. Aiming at the issues above, a lightweight multi-person pose estimation method based on symmetric transformation and global matching is proposed. Symmetric transformation module adds spatial transformation network(STN) and spatial de-transformer network(SDTN) before and after the single-person pose estimation(SPPE) to extract high-quality single-person pose regions from inaccurate human candidate frames. Global matching method is used to transform the key point prediction problem into the optimal matching problem of the human body-key point graph, and solve the questions of misjudgment and error detection of pose estimation in the crowded state. Finally, depth wise separable convolution and inverted residual model are used to reduce the complexity of model, so as to improve the running speed while balancing the accuracy of the algorithm. Experiments show that the algorithm proposed in this paper not only enhance the overall performance of the multi-person posture estimation network in the crowded state, but also improves the running speed significantly, which further confirms the effectiveness and competitiveness of this algorithm.

Spectral–Spatial Transformer Network for Hyperspectral Image Classification: A Factorized Architecture Search Framework

Neural networks have dominated the research of hyperspectral image classification, attributing to the feature learning capacity of convolution operations. However, the fixed geometric structure of convolution kernels hinders long-range interaction between features from distant locations. In this article, we propose a novel spectral–spatial transformer network (SSTN), which consists of spatial attention and spectral association modules, to overcome the constraints of convolution kernels. Also, we design a factorized architecture search (FAS) framework that involves two independent subprocedures to determine the layer-level operation choices and block-level orders of SSTN. Unlike conventional neural architecture search (NAS) that requires a bilevel optimization of both network parameters and architecture settings, the FAS focuses only on finding out optimal architecture settings to enable a stable and fast architecture search. Extensive experiments conducted on five popular HSI benchmarks demonstrate the versatility of SSTNs over other state-of-the-art (SOTA) methods and justify the FAS strategy. On the University of Houston dataset, SSTN obtains comparable overall accuracy to SOTA methods with a small fraction (1.2%) of multiply-and-accumulate operations compared to a strong baseline spectral–spatial residual network (SSRN). Most importantly, SSTNs outperform other SOTA networks using only 1.2% or fewer MACs of SSRNs on the Indian Pines, the Kennedy Space Center, the University of Pavia, and the Pavia Center datasets.

Fast and Few-Shot Unsupervised Defect Detection on Flexible Packaging Surfaces

In the production of flexible packaging, 200 samples need to be detected per minute, so the speed of detection is very demanding. In this paper, we propose an approach to detect and localize flexible packaging defects with only a small number of defect-free samples for model training. We achieve fast data augmentation by using the spatial transformation network to form an end-to-end detection model. We calculate defect scores by estimating the feature maps density based on the normalizing flow to detect and locate defects. Compared with DifferNet, our approach achieves detection 10 times faster, but AUROC remains almost the same. Thus, our method is more compatible with the requirements of industrial production.

Spatial Transformer Network-Based Automatic Modulation Recognition of Blind Signals

Modulation recognition of communication signals plays an important role in both civil and military uses. Neural network-based modulation recognition methods can extract high-level abstract features which can be adopted for classification of modulation types. Compared with traditional recognition methods based on manually defined features, they have the advantage of higher recognition rate. However, in actual modulation recognition scenarios, due to inaccurate estimation of receiving parameters and other reasons, the input signal samples for modulation recognition may have large phase, frequency offsets, and time scale changes. Existing deep learning-based modulation recognition methods have not considered the influences brought by the above issues, thus resulting in a decreased recognition rate. A modulation recognition method based on the spatial transformation network is proposed in this paper. In the proposed network, some prior models for synchronization in communication are introduced, and the priori models are realized through the spatial transformation subnetwork, so as to reduce the influence of phase, frequency offsets, and time scale differences. Experiments on simulated datasets prove that compared with the traditional CNN, ResNet, and the CLDNN, the recognition rate of the proposed method has increased by 8.0%, 5.8%, and 4.6%, respectively, when the signal-to-noise ratio is greater than 0. Moreover, the proposed network is also easier to train. The training time required for convergence has reduced by 4.5% and 80.7% compared to the ResNet and CLDNN, respectively.

Multimodal Emotion Recognition on RAVDESS Dataset Using Transfer Learning.

Emotion Recognition is attracting the attention of the research community due to the multiple areas where it can be applied, such as in healthcare or in road safety systems. In this paper, we propose a multimodal emotion recognition system that relies on speech and facial information. For the speech-based modality, we evaluated several transfer-learning techniques, more specifically, embedding extraction and Fine-Tuning. The best accuracy results were achieved when we fine-tuned the CNN-14 of the PANNs framework, confirming that the training was more robust when it did not start from scratch and the tasks were similar. Regarding the facial emotion recognizers, we propose a framework that consists of a pre-trained Spatial Transformer Network on saliency maps and facial images followed by a bi-LSTM with an attention mechanism. The error analysis reported that the frame-based systems could present some problems when they were used directly to solve a video-based task despite the domain adaptation, which opens a new line of research to discover new ways to correct this mismatch and take advantage of the embedded knowledge of these pre-trained models. Finally, from the combination of these two modalities with a late fusion strategy, we achieved 80.08% accuracy on the RAVDESS dataset on a subject-wise 5-CV evaluation, classifying eight emotions. The results revealed that these modalities carry relevant information to detect users’ emotional state and their combination enables improvement of system performance.

Pose-guided matching based on deep learning for assessing quality of action on rehabilitation training

The application of pose assessment on rehabilitation training has gradually received attention in recent years. However, current evaluation indicators of these methods are mostly based on the score or scoring function that defined by users, which is too subjective and hard to be used by patients directly. In this paper, we conceptualized a new idea for pose matching, namely pose-guided matching that aims at providing objective and accurate score, feedback and guidance (i.e. guided) to the patients when the pose is compared to the standard pose. More specifically, we proposed a pair-based Siamese Convolutional Neural Network (SCNN) abbreviated ST-AMCNN to realize the idea of pose-guided matching on the eight-section brocade dataset which is one of the most representative traditional rehabilitation exercises in China. We simplified the multi-stages pose matching by merging two standalone modules (i.e. alignment and matching module) into a one-stage task. Such that, only one loss function is required to tune, which reduces the computational complexity. On top of the Spatial Transformer Networks (STN) employed as an alignment module, we proposed a new Attention-based Multi-Scale Convolution (AMC) to match different posture parts (i.e. multi-scale). Furthermore, the proposed AMC can assign more weight to useful pose features as opposed to other irrelevant features e.g. background features for performance gain. Finally, Gradient-weighted Class Activation Mapping (Grad-CAM) is adopted to visualize the matching result for the learner. Experimental results indicate that ST-AMCNN achieves a competitive performance than the state-of-the-art models and can provide accurate feedback for learners on rehabilitation training. Simultaneously, the proposed method is also deployed in client software for testing.

Joint object recognition and pose estimation using multiple-anchor triplet learning of canonical plane

Accurate object recognition and pose estimation models are essential for practical applications of robot arms, such as picking products on a shelf. Training such a model often requires a large-scale dataset with qualified labels for both object classes and pose parameters, and collecting accurate pose labels is particularly costly. A recent paper Ueno et al.(2019)[28] proposed a triplet learning framework for joint object recognition and pose estimation without explicit pose labels by learning a spatial transformer network to estimate the pose difference of an input image from an anchor image depicting the same object in a reference pose. However, our analysis suggests that the pose estimation accuracy is severely degraded for input images with large pose differences. To address this problem, we propose a new learning approach called multiple-anchor triplet learning. The basic idea is to give dense reference poses by preparing multiple anchors so that there is at least one anchor image having a small pose difference to the input image. Our multiple-anchor triplet learning is an extension of the standard single-anchor triplet learning to the multiple-anchor case. Inspired by the idea of multiple instance learning, we introduce a selection layer that automatically chooses the best anchor for each input image and allows the network to be trained end-to-end to minimize triplet-based losses. Experiments with three benchmark datasets in product picking scenarios demonstrate that our method significantly outperforms existing methods in both object recognition and pose estimation accuracy.

Pose Estimation of GIS Pipeline Based on Spatial Transformation Network

The manual flange structure alignment between GIS pipelines in the power system is inefficient and difficult to accurately align. To solve this problem, combined with the research results in the field of deep learning named spatial transformation network, a new pose estimation method based on single camera is proposed. In view of the high similarity between the moving flange and the static flange at the pixel level, the spatial transformation network is used to find the pixel mapping relationship of the two flange images. Thereby establishing the mapping relationship between the pixel coordinates of the two flange images and then using multiple points. In the perspective method, the pixel coordinates are mapped to the world coordinates to obtain the estimation of the position of the key point in the flange, and then the direction vector of the flange is calculated according to the position of the key point. Since there is a pixel coordinate transformation relationship between the static flange and the movable flange. Only the position of the key point in the static flange can be inversely solved by measuring the position of the key point in the static flange. Experiments show that, compared to the traditional method of measuring flange pose based on instrument measurement and linear regression, the method proposed in this paper can accurately measure the pose of the flange structure. And it can rely as little as possible on the measurement of the key points of the moving flange by the instrument.

ChangeGAN: A Deep Network for Change Detection in Coarsely Registered Point Clouds

In this letter we introduce a novel change detection approach called ChangeGAN for coarsely registered point clouds in complex street-level urban environment. Our generative adversarial network-like (GAN) architecture compounds Siamese-style feature extraction, U-net-like use of multiscale features, and Spatial Trans-formation Network (STN) blocks for optimal transformation estimation. The input point clouds are represented by range images, which enables the use of 2D convolutional neural networks. The result is a pair of binary masks showing the change regions on each input range image, which can be backprojected to the input point clouds without loss of information. We have evaluated the proposed method on various challenging scenarios and we have shown its superiority against state-of-the-art change detection methods.

Guided Spatial Transformers for Facial Expression Recognition

Spatial Transformer Networks are considered a powerful algorithm to learn the main areas of an image, but still, they could be more efficient by receiving images with embedded expert knowledge. This paper aims to improve the performance of conventional Spatial Transformers when applied to Facial Expression Recognition. Based on the Spatial Transformers’ capacity of spatial manipulation within networks, we propose different extensions to these models where effective attentional regions are captured employing facial landmarks or facial visual saliency maps. This specific attentional information is then hardcoded to guide the Spatial Transformers to learn the spatial transformations that best fit the proposed regions for better recognition results. For this study, we use two datasets: AffectNet and FER-2013. For AffectNet, we achieve a 0.35% point absolute improvement relative to the traditional Spatial Transformer, whereas for FER-2013, our solution gets an increase of 1.49% when models are fine-tuned with the Affectnet pre-trained weights.

Geometric rectification‐based neural network architecture for image manipulation detection

Determination of image authenticity usually requires the identification and localization of the manipulated regions of images. Hence, image manipulation detection has become one of the most important tasks in the field of multimedia forensics. Recently, Convolutional Neural Networks (CNNs) have achieved promising performance in image manipulation detection. However, it is hard for the existing CNN-based manipulation detection approaches to accurately identify and localize the manipulated regions that have undergone geometric transformations, since CNNs are limited by their inability to be geometrically invariant. To address this issue, we propose a geometric rectification-based neural network architecture for image manipulation detection. In this type of network architecture, following the detection of a set of potential manipulated regions (PMRs) using Region Proposal Network, the Spatial Transformer Network is employed to geometrically rectify the convolutional feature maps (CFMs) of these regions to obtain the geometrically rectified CFMs (GR-CFMs). Subsequently, the residual feature maps (RFMs) are computed to capture the characteristic inconsistency between the CFMs and GR-CFMs of each PMR. Finally, the computed RFMs are automatically integrated with the GR-CFMs by a designed attention module to determine whether each PMR is a manipulated region and to localize the manipulated part at the pixel-level. Extensive experiments on the public data set as well as on our challenging data set demonstrate that the proposed network architecture achieves desirable performance in identifying and localizing regions with common tampering artifacts, which involve geometric transformations.