Spatial Transformer Network Research Articles

Scene Coordinate Regression Network With Global Context-Guided Spatial Feature Transformation for Visual Relocalization

Among visual relocalization from a single RGB image, the scene coordinate regression (SCoRe) based on convolutional neural network (CNN) becomes prevailing, however, it is insufficient to extract invariant features under different viewpoints due to fixed geometric structures of CNN. In this letter, we propose a global context-guided spatial feature transformation (SFT) network to learn invariant feature representation for robustness against viewpoint changes. Specifically, global feature extracted from source feature map is regarded as a dynamic convolutional kernel, which is convolved with source feature map for the prediction of transformation parameters. The predicted parameters are used to transform features of multiple viewpoints to a canonical space with the constraint of maximum likelihood-derived loss, and thus viewpoint invariance is achieved. CoordConv is also employed to further improve the discrimination of features on texture-less or repetitive zones. The proposed SFT network can be easily incorporated into the general SCoRe network. To our best knowledge, features are first decoupled from viewpoints explicitly in SCoRe network by the spatial feature transformation network, which achieves a stable and accurate visual relocalization. The experimental results demonstrate the effectiveness of the proposed method in terms of accuracy and efficiency.

An Improved Deep Residual Network Prediction Model for the Early Diagnosis of Alzheimer's Disease.

The early diagnosis of Alzheimer’s disease (AD) can allow patients to take preventive measures before irreversible brain damage occurs. It can be seen from cross-sectional imaging studies of AD that the features of the lesion areas in AD patients, as observed by magnetic resonance imaging (MRI), show significant variation, and these features are distributed throughout the image space. Since the convolutional layer of the general convolutional neural network (CNN) cannot satisfactorily extract long-distance correlation in the feature space, a deep residual network (ResNet) model, based on spatial transformer networks (STN) and the non-local attention mechanism, is proposed in this study for the early diagnosis of AD. In this ResNet model, a new Mish activation function is selected in the ResNet-50 backbone to replace the Relu function, STN is introduced between the input layer and the improved ResNet-50 backbone, and a non-local attention mechanism is introduced between the fourth and the fifth stages of the improved ResNet-50 backbone. This ResNet model can extract more information from the layers by deepening the network structure through deep ResNet. The introduced STN can transform the spatial information in MRI images of Alzheimer’s patients into another space and retain the key information. The introduced non-local attention mechanism can find the relationship between the lesion areas and normal areas in the feature space. This model can solve the problem of local information loss in traditional CNN and can extract the long-distance correlation in feature space. The proposed method was validated using the ADNI (Alzheimer’s disease neuroimaging initiative) experimental dataset, and compared with several models. The experimental results show that the classification accuracy of the algorithm proposed in this study can reach 97.1%, the macro precision can reach 95.5%, the macro recall can reach 95.3%, and the macro F1 value can reach 95.4%. The proposed model is more effective than other algorithms.

ST-V-Net: incorporating shape prior into convolutional neural networks for proximal femur segmentation

We aim to develop a deep-learning-based method for automatic proximal femur segmentation in quantitative computed tomography (QCT) images. We proposed a spatial transformation V-Net (ST-V-Net), which contains a V-Net and a spatial transform network (STN) to extract the proximal femur from QCT images. The STN incorporates a shape prior into the segmentation network as a constraint and guidance for model training, which improves model performance and accelerates model convergence. Meanwhile, a multi-stage training strategy is adopted to fine-tune the weights of the ST-V-Net. We performed experiments using a QCT dataset which included 397 QCT subjects. During the experiments for the entire cohort and then for male and female subjects separately, 90% of the subjects were used in ten-fold stratified cross-validation for training and the rest of the subjects were used to evaluate the performance of models. In the entire cohort, the proposed model achieved a Dice similarity coefficient (DSC) of 0.9888, a sensitivity of 0.9966 and a specificity of 0.9988. Compared with V-Net, the Hausdorff distance was reduced from 9.144 to 5.917 mm, and the average surface distance was reduced from 0.012 to 0.009 mm using the proposed ST-V-Net. Quantitative evaluation demonstrated excellent performance of the proposed ST-V-Net for automatic proximal femur segmentation in QCT images. In addition, the proposed ST-V-Net sheds light on incorporating shape prior to segmentation to further improve the model performance.

Lung segmentation and automatic detection of COVID-19 using radiomic features from chest CT images

This paper aims to develop an automatic method to segment pulmonary parenchyma in chest CT images and analyze texture features from the segmented pulmonary parenchyma regions to assist radiologists in COVID-19 diagnosis. A new segmentation method, which integrates a three-dimensional (3D) V-Net with a shape deformation module implemented using a spatial transform network (STN), was proposed to segment pulmonary parenchyma in chest CT images. The 3D V-Net was adopted to perform an end-to-end lung extraction while the deformation module was utilized to refine the V-Net output according to the prior shape knowledge. The proposed segmentation method was validated against the manual annotation generated by experienced operators. The radiomic features measured from our segmentation results were further analyzed by sophisticated statistical models with high interpretability to discover significant independent features and detect COVID-19 infection. Experimental results demonstrated that compared with the manual annotation, the proposed segmentation method achieved a Dice similarity coefficient of 0.9796, a sensitivity of 0.9840, a specificity of 0.9954, and a mean surface distance error of 0.0318 mm. Furthermore, our COVID-19 classification model achieved an area under curve (AUC) of 0.9470, a sensitivity of 0.9670, and a specificity of 0.9270 when discriminating lung infection with COVID-19 from community-acquired pneumonia and healthy controls using statistically significant radiomic features. The significant features measured from our segmentation results agreed well with those from the manual annotation. Our approach has great promise for clinical use in facilitating automatic diagnosis of COVID-19 infection on chest CT images.

A 3D medical image registration method based on multi-scale feature fusion

Deformable medical image registration refers to finding a certain transformation so that the corresponding points of two medical images can be aligned in space. This has important clinical applications. In this article, we propose an unsupervised end-to-end medical image registration method. In this method, the fixed image and the moving image are concatenated in series and input into the convolution neural network to obtain the feature images of different scales. In order to improve the ability of neural network to capture global and local information, we fuse feature maps of different scales. The spatial transformation network uses the deformation field to deform the moving image, so as to realize the registration of the two pairs. We validate our method in the ABIDE data set and compare it with some classic state-of-the-art methods. The experimental results show that our method improves the registration accuracy of image pairs.

Skeleton-based action recognition via spatial and temporal transformer networks

Skeleton-based Human Activity Recognition has achieved great interest in recent years as skeleton data has demonstrated being robust to illumination changes, body scales, dynamic camera views, and complex background. In particular, Spatial–Temporal Graph Convolutional Networks (ST-GCN) demonstrated to be effective in learning both spatial and temporal dependencies on non-Euclidean data such as skeleton graphs. Nevertheless, an effective encoding of the latent information underlying the 3D skeleton is still an open problem, especially when it comes to extracting effective information from joint motion patterns and their correlations. In this work, we propose a novel Spatial–Temporal Transformer network (ST-TR) which models dependencies between joints using the Transformer self-attention operator. In our ST-TR model, a Spatial Self-Attention module (SSA) is used to understand intra-frame interactions between different body parts, and a Temporal Self-Attention module (TSA) to model inter-frame correlations. The two are combined in a two-stream network, whose performance is evaluated on three large-scale datasets, NTU-RGB+D 60, NTU-RGB+D 120, and Kinetics Skeleton 400, consistently improving backbone results. Compared with methods that use the same input data, the proposed ST-TR achieves state-of-the-art performance on all datasets when using joints’ coordinates as input, and results on-par with state-of-the-art when adding bones information.

Contralaterally Enhanced Networks for Thoracic Disease Detection.

Identifying and locating diseases in chest X-rays are very challenging, due to the low visual contrast between normal and abnormal regions, and distortions caused by other overlapping tissues. An interesting phenomenon is that there exist many similar structures in the left and right parts of the chest, such as ribs, lung fields and bronchial tubes. This kind of similarities can be used to identify diseases in chest X-rays, according to the experience of broad-certificated radiologists. Aimed at improving the performance of existing detection methods, we propose a deep end-to-end module to exploit the contralateral context information for enhancing feature representations of disease proposals. First of all, under the guidance of the spine line, the spatial transformer network is employed to extract local contralateral patches, which can provide valuable context information for disease proposals. Then, we build up a specific module, based on both additive and subtractive operations, to fuse the features of the disease proposal and the contralateral patch. Our method can be integrated into both fully and weakly supervised disease detection frameworks. It achieves 33.17 AP50 on a carefully annotated private chest X-ray dataset which contains 31,000 images. Experiments on the NIH chest X-ray dataset indicate that our method achieves state-of-the-art performance in weakly-supervised disease localization.

STN-enhanced message passing guided by adversarial learning for human pose estimation

In this paper, an STN (Spatial Transformer Network)-enhanced message passing module guided by adversarial learning is proposed for human pose estimation. Transformations performed on the predicted heatmaps mean to modify the raw predictions and conduct the message passing among human joints in an elegant way. Firstly, we employ STN submodule to transform the predicted heatmap of one human joint to heatmap of its neighboring joint to remove ambiguity of its neighboring predictions. STN submodule automatically learns the geometric information between adjacent joints and thus builds related neighboring associations among them. Nevertheless, it seems difficult for STN submodule to learn inherent geometric information from a single RGB image alone. Secondly, limb guidance is introduced to assist STN in predicting corresponding correlations. Since quality of limb predictions poses great significance for the guidance, we propose to exploit adversarial learning to improve the quality of limb heatmaps which are easier to learn than precise keypoint location. Hence, the precision of STN transformation improves owing to more precise prior instructions. However, STN submodule might be confused when performing the transformation due to massive noises of the heatmaps. To circumvent this dilemma, at last, we propose to utilize Weighted Mean Square Error (WMSE) loss and convolutional random walk (CRW) which improve the performance further. Our method achieves competitive results on both MPII and LSP benchmarks.

Rotation-aware representation learning for remote sensing image retrieval

The rising number and size of remote sensing (RS) image archives makes content-based RS image retrieval (CBRSIR) more important. Convolutional neural networks (CNNs) offer good CBRSIR performance, but the features they extract are not rotation-invariant. This is problematic as objects in RS images appear in arbitrary rotation angles . We develop and investigate two new rotation-aware CNN-based CBRSIR methods: 1) In the Feature Map Transformation Based Rotation-Aware Network (FMT-RAN), the last pooling layer is rotated in four different angles during training. Its outputs are passed through the same fully connected-, coding-, and classification layer, and the resulting losses are added. 2) The Spatial Transformer-based Rotation-Aware Network (ST-RAN) contains a spatial transformer network (STN) and a rotation aware network (RAN). For training, the original and a randomly rotated version of an image are fed into the ST-RAN. The STN generates a transformed version of the original to match the rotated image. The RAN extracts the features of all three images. We apply two-stage training, which first optimizes the STN and then the RAN. Both of our methods are efficient in terms of retrieval accuracy and time, but ST-RAN has the overall best performance. It outperforms the state-of-the-art CBRSIR methods.

A novel deep-learning-based approach for automatic reorientation of 3D cardiac SPECT images.

Reconstructed transaxial cardiac SPECT images need to be reoriented into standard short-axis slices for subsequent accurate processing and analysis. We proposed a novel deep-learning-based method for fully automatic reorientation of cardiac SPECT images and evaluated its performance on data from two clinical centers. We used a convolutional neural network to predict the 6 rigid-body transformation parameters and a spatial transformation network was then implemented to apply these parameters on the input images for image reorientation. A novel compound loss function which balanced the parametric similarity and penalized discrepancy of the prediction and training dataset was utilized in the training stage. Data from a set of 322 patients underwent data augmentation to 6440 groups of images for the network training, and a dataset of 52 patients from the same center and 23 patients from another center were used for evaluation. Similarity of the 6 parameters was analyzed between the proposed and the manual methods. Polar maps were generated from the output images and the averaged count values of the 17 segments were computed from polar maps to evaluate the quantitative accuracy of the proposed method. All the testing patients achieved automatic reorientation successfully. Linear regression results showed the 6 predicted rigid parameters and the average count value of the 17 segments having good agreement with the reference manual method. No significant difference by paired t-test was noticed between the rigid parameters of our method and the manual method (p > 0.05). Average count values of the 17 segments show a smaller difference of the proposed and manual methods than those between the existing and manual methods. The results strongly indicate the feasibility of our method in accurate automatic cardiac SPECT reorientation. This deep-learning-based reorientation method has great promise for clinical application and warrants further investigation.

Unsupervised registration of intravascular ultrasound images combined with attention mechanism

For intravascular ultrasound (IVUS) images, the registration technology is used to calculate coronary arteries displacement to analyze vascular elasticity. It not only provides evidence for the prevention and treatment of cardiovascular diseases, but also has important significance for guiding interventional surgery and monitoring the placement of surgical stents. Aiming at the high computational cost of current traditional registration methods and the insufficient accuracy of common deep learning registration methods for IVUS images, this paper proposes a fast unsupervised registration of IVUS images combined with an attention mechanism. The proposed method directly learns to estimate a displacement vector field (DVF) from a pair of input images of the training set. The spatial transform network (STN) uses the DVF to transform the moving image into the fixed image. Finally, the model is trained by minimizing a similarity metric loss function between the deformed moving image and the fixed image. Compared with the previous deep learning method, the registration performance improved after implementing the proposed method. The proposed method can accurately register the inner and outer membranes of IVU S images and provide a reliable basis for vascular elasticity analysis.

Human attention-inspired volume reconstruction method on serial section electron microscopy images.

The alignment of a 2D microscopic image stack to create a 3D image volume is an indispensable aspect of serial section electron microscopy (EM) technology, which could restore the original 3D integrity of biological tissues destroyed by chemical fixation and physical dissection. However, due to the similar texture intrasection and complex variations intersections of neural images, previous registration methods usually failed to yield reliable correspondences. And this also led to misalignment and impeded restoring the z-axis anatomical continuity of the neuron volume. In this article, inspired by human behaviors in finding correspondences, which use the topological relationship of image contents, we developed a spatial attention-based registration method for serial EM images to improve registration accuracy. Our approach combined the U-Net framework with spatial transformer networks (STN) to regress corresponding transformation maps in an unsupervised training fashion. The spatial attention (SA) module was incorporated into the U-Net architecture to increase the distinctiveness of image features by modeling its topological relationship. Experiments are conducted on both simulated and real data sets (MAS and RegCremi). Quantitative and qualitative comparisons demonstrate that our approach results in state of art accuracy (using the evaluation index of NCC, SSIM, Dice, Landmark error) and providing smooth and reliable transformation with less texture blur and unclear boundary than existing techniques. Our method is able to restore image stacks for visualization and quantitative analysis of EM image sequences.

Multistage Model for Robust Face Alignment Using Deep Neural Networks

The ability to generalize unconstrained conditions such as severe occlusions and large pose variations remains a challenging goal to achieve in face alignment. In this paper, a multistage model based on deep neural networks is proposed which takes advantage of spatial transformer networks, hourglass networks and exemplar-based shape constraints. First, a spatial transformer-generative adversarial network which consists of convolutional layers and residual units is utilized to solve the initialization issues caused by face detectors, such as rotation and scale variations, to obtain improved face bounding boxes for face alignment. Then, stacked hourglass network is employed to obtain preliminary locations of landmarks as well as their corresponding scores. In addition, an exemplar-based shape dictionary is designed to determine landmarks with low scores based on those with high scores. By incorporating face shape constraints, misaligned landmarks caused by occlusions or cluttered backgrounds can be considerably improved. Extensive experiments based on challenging benchmark datasets are performed to demonstrate the superior performance of the proposed method over other state-of-the-art methods.

Automatic deep learning-based pleural effusion classification in lung ultrasound images for respiratory pathology diagnosis.

Lung ultrasound (LUS) imaging as a point-of-care diagnostic tool for lung pathologies has been proven superior to X-ray and comparable to CT, enabling earlier and more accurate diagnosis in real-time at the patient's bedside. The main limitation to widespread use is its dependence on the operator training and experience. COVID-19 lung ultrasound findings predominantly reflect a pneumonitis pattern, with pleural effusion being infrequent. However, pleural effusion is easy to detect and to quantify, therefore it was selected as the subject of this study, which aims to develop an automated system for the interpretation of LUS of pleural effusion. A LUS dataset was collected at the Royal Melbourne Hospital which consisted of 623 videos containing 99,209 2D ultrasound images of 70 patients using a phased array transducer. A standardized protocol was followed that involved scanning six anatomical regions providing complete coverage of the lungs for diagnosis of respiratory pathology. This protocol combined with a deep learning algorithm using a Spatial Transformer Network provides a basis for automatic pathology classification on an image-based level. In this work, the deep learning model was trained using supervised and weakly supervised approaches which used frame- and video-based ground truth labels respectively. The reference was expert clinician image interpretation. Both approaches show comparable accuracy scores on the test set of 92.4% and 91.1%, respectively, not statistically significantly different. However, the video-based labelling approach requires significantly less effort from clinical experts for ground truth labelling.

VPC-Net: Completion of 3D vehicles from MLS point clouds

As a dynamic and essential component in the road environment of urban scenarios, vehicles are the most popular investigation targets. To monitor their behavior and extract their geometric characteristics, an accurate and instant measurement of vehicles plays a vital role in traffic and transportation fields. Point clouds acquired from the mobile laser scanning (MLS) system deliver 3D information of road scenes with unprecedented detail. They have proven to be an adequate data source in the fields of intelligent transportation and autonomous driving, especially for extracting vehicles. However, acquired 3D point clouds of vehicles from MLS systems are inevitably incomplete due to object occlusion or self-occlusion. To tackle this problem, we proposed a neural network to synthesize complete, dense, and uniform point clouds for vehicles from MLS data, named Vehicle Points Completion-Net (VPC-Net). In this network, we introduce a new encoder module to extract global features from the input instance, consisting of a spatial transformer network and point feature enhancement layer. Moreover, a new refiner module is also presented to preserve the vehicle details from inputs and refine the complete outputs with fine-grained information. Given sparse and partial point clouds as inputs, the network can generate complete and realistic vehicle structures and keep the fine-grained details from the partial inputs. We evaluated the proposed VPC-Net in different experiments using synthetic and real-scan datasets and applied the results to 3D vehicle monitoring tasks. Quantitative and qualitative experiments demonstrate the promising performance of the proposed VPC-Net and show state-of-the-art results.

Joint Deep Learning Framework for Image Registration and Segmentation of Late Gadolinium Enhanced MRI and Cine Cardiac MRI.

Late gadolinium enhanced (LGE) cardiac magnetic resonance (CMR) imaging, the current benchmark for assessment of myocardium viability, enables the identification and quantification of the compromised myocardial tissue regions, as they appear hyper-enhanced compared to the surrounding, healthy myocardium. However, in LGE CMR images, the reduced contrast between the left ventricle (LV) myocardium and LV blood-pool hampers the accurate delineation of the LV myocardium. On the other hand, the balanced-Steady State Free Precession (bSSFP) cine CMR imaging provides high resolution images ideal for accurate segmentation of the cardiac chambers. In the interest of generating patient-specific hybrid 3D and 4D anatomical models of the heart, to identify and quantify the compromised myocardial tissue regions for revascularization therapy planning, in our previous work, we presented a spatial transformer network (STN) based convolutional neural network (CNN) architecture for registration of LGE and bSSFP cine CMR image datasets made available through the 2019 Multi-Sequence Cardiac Magnetic Resonance Segmentation Challenge (MS-CMRSeg). We performed a supervised registration by leveraging the region of interest (RoI) information using the manual annotations of the LV blood-pool, LV myocardium and right ventricle (RV) blood-pool provided for both the LGE and the bSSFP cine CMR images. In order to reduce the reliance on the number of manual annotations for training such network, we propose a joint deep learning framework consisting of three branches: a STN based RoI guided CNN for registration of LGE and bSSFP cine CMR images, an U-Net model for segmentation of bSSFP cine CMR images, and an U-Net model for segmentation of LGE CMR images. This results in learning of a joint multi-scale feature encoder by optimizing all three branches of the network architecture simultaneously. Our experiments show that the registration results obtained by training 25 of the available 45 image datasets in a joint deep learning framework is comparable to the registration results obtained by stand-alone STN based CNN model by training 35 of the available 45 image datasets and also shows significant improvement in registration performance when compared to the results achieved by the stand-alone STN based CNN model by training 25 of the available 45 image datasets.

Strong Feature Fusion Networks for Person Re-Identification

<p indent=0mm>With the development of deep learning, the performance of person Re-Identification (Re-ID) has been significantly improved. It’s still a challenging task due to the challenges coming from large variations on persons such as occlusion, background clutter, pose, illumination and detection failure, etc. To retrieve true pedestrians, robust feature expression is significant. Instead of using external cues, this paper takes advantage of robust alignment features and metric learning. First, from the aspect of feature extraction, there were three contributions. (i) Embeded a spatial transformer network in the network architecture, which is called ResNet_STN in this paper, which can solve the problem of local spatial semantic feature inconsistency, accurately express the main characteristics of the target, and achieve pedestrian alignment. (ii) Designed a strong feature fusion network based on the aligned local features, which is named a Strong Feature Fusion Module (SFFM) and can make full use of the connection between semantic information to extract detailed features of images. Then, from the aspect of metric loss function, one contribution was put forward. (iii) Proposed a Ranking Matrix (RM) method to select local triplet samples and compute local triplet loss. We combined a regularized classification loss to train the network to unleash the discrimination ability of the learned strong representations of this network architecture. Finally, the proposed method with the existing re-ranking algorithm to further improves Rank-1 and mAP retrieval accuracy. Experimental results on Market-1501 dataset demonstrate the effectiveness of our proposed method.

GG-Net: Gaze Guided Network for Self-driving Cars

Imitation learning is used massively in autonomous driving for training networks to predict steering commands from frames using annotated data collected by an expert driver. Believing that the frames taken from a front-facing camera are completely mimicking the driver’s eyes raises the question of how eyes and the complex human vision system attention mechanisms perceive the scene. This paper proposes the idea of incorporating eye gaze information with the frames into an end-to-end deep neural network in the lane-following task. The proposed novel architecture, GG-Net, is composed of a spatial transformer network (STN), and a multitask network to predict steering angle as well as the gaze map for the input frame. The experimental results of this architecture show a great improvement in steering angle prediction accuracy of 36% over the baseline with inference time of 0.015 seconds per frame (66 fps) using NVIDIA K80 GPU enabling the proposed model to operate in real-time. We argue that incorporating gaze maps enhances the model generalization capability to the unseen environments. Additionally, a novel course-steering angle conversion algorithm with a complementing mathematical proof is proposed.

Robust license plate detection and recognition with automatic rectification

We propose a robust license plate detection and recognition (LPDR) framework with automatic rectification. We explore the YOLOv2 object detector based on deep learning and train it to detect license plates (LPs) effectively. The LPs in natural scene images tend to be tilted and distorted because of the shooting angle or the geometric deformation of LPs. To solve the problem in which the LP tilt and distortion affect recognition accuracy, we introduce a spatial transformation network with thin-plate-spline transformation and propose a neural network called inverse compositional spatial transformer network-hierarchical spatial transformer network (ICSTN-CRNN). ICSTN-CRNN can automatically rectify and recognize LPs. Furthermore, we manually supplement the LP character annotations in PKUData. Our LPDR method achieves satisfactory results on three datasets, including Chinese City Parking Dataset, PKUData, and application-oriented license plate. Through a series of comparative experiments, we prove that our method is more accurate than other advanced methods.

Transformation guided representation GAN for pose invariant face recognition

Face recognition is an important topic in the field of computer vision and has been a vital biometric technique for identity authentication. It is widely used in areas such as public security, military, and daily life. However, face recognition is inherently a challenging problem due to variations in poses, facial expressions, age, and occlusion. In this work, we propose a generative adversarial network (GAN) architecture that disentangles identity and pose variations to learn generative and discriminative representations for pose-invariant face recognition. We use an iterative warping scheme that achieves better results than with the use of a single generator. The features from the encoder are considered pose-invariant features for face recognition, and evaluations on databases demonstrate the usefulness of this approach over prior methods. For example, we report 97.0% (+ 12.7%) and 90.5% (+ 8.4%) accuracy on the Feret and Caspeal datasets compared to 78.2% achieved by the best method without warping. In particular, there are two notable novelties. First, the disentangled architecture GAN (D-GAN) performs frontal face synthesis via an encoder-decoder structure in the generator with the pose variations provided to the decoder and discriminator. Second, we utilize the generator encoder as a spatial transformer network that seeks realistic image synthesis in the geometric warp parameter space.