Local Feature Learning Research Articles

TPBTE: A model based on convolutional Transformer for predicting the binding of TCR to epitope

T cell receptors (TCRs) selectively bind to antigens to fight pathogens with specific immunity. Current tools focus on the nature of amino acids within sequences and take less into account the nature of amino acids far apart and the relationship between sequences, leading to significant differences in the results from different datasets. We propose TPBTE, a model based on convolutional Transformer for Predicting the Binding of TCR to Epitope. It takes epitope sequences and the complementary decision region 3 (CDR3) sequences of TCRβ chain as inputs. And it uses a convolutional attention mechanism to learn amino acid representations between different positions of the sequences based on learning local features of the sequences. At the same time, it uses cross attention to learn the interaction information between TCR sequences and epitope sequences. A comprehensive evaluation of the TCR-epitope data shows that the average area under the curve of TPBTE outperforms the baseline model, and demonstrate an intentional performance. In addition, TPBTE can give the probability of binding TCR to epitopes, which can be used as the first step of epitope screening, narrowing the scope of epitope search and reducing the time of epitope search.

Learning local contextual features for 3D point clouds semantic segmentation by attentive kernel convolution

Learning local contextual features for 3D point clouds semantic segmentation by attentive kernel convolution

Transformer-based multi-task learning for classification and segmentation of gastrointestinal tract endoscopic images

Despite being widely utilized to help endoscopists identify gastrointestinal (GI) tract diseases using classification and segmentation, models based on convolutional neural network (CNN) have difficulties in distinguishing the similarities among some ambiguous types of lesions presented in endoscopic images, and in the training when lacking labeled datasets. Those will prevent CNN from further improving the accuracy of diagnosis. To address these challenges, we first proposed a Multi-task Network (TransMT-Net) capable of simultaneously learning two tasks (classification and segmentation), which has the transformer designed to learn global features and can combine the advantages of CNN in learning local features so that to achieve a more accurate prediction in identifying the lesion types and regions in GI tract endoscopic images. We further adopted the active learning in TransMT-Net to tackle the labeled image-hungry problem. A dataset was created from the CVC-ClinicDB dataset, Macau Kiang Wu Hospital, and Zhongshan Hospital to evaluate the model performance. Then, the experimental results show that our model not only achieved 96.94% accuracy in the classification task and 77.76% Dice Similarity Coefficient in the segmentation task but also outperformed those of other models on our test set. Meanwhile, active learning also produced positive results for the performance of our model with a small-scale initial training set, and even its performance with 30% of the initial training set was comparable to that of most comparable models with the full training set. Consequently, the proposed TransMT-Net has demonstrated its potential performance in GI tract endoscopic images and it through active learning can alleviate the shortage of labeled images.

Deep Convolutional Compressed Sensing-Based Adaptive 3D Reconstruction of Sparse LiDAR Data: A Case Study for Forests

LiDAR point clouds are characterized by high geometric and radiometric resolution and are therefore of great use for large-scale forest analysis. Although the analysis of 3D geometries and shapes has improved at different resolutions, processing large-scale 3D LiDAR point clouds is difficult due to their enormous volume. From the perspective of using LiDAR point clouds for forests, the challenge lies in learning local and global features, as the number of points in a typical 3D LiDAR point cloud is in the range of millions. In this research, we present a novel end-to-end deep learning framework called ADCoSNet, capable of adaptively reconstructing 3D LiDAR point clouds from a few sparse measurements. ADCoSNet uses empirical mode decomposition (EMD), a data-driven signal processing approach with Deep Learning, to decompose input signals into intrinsic mode functions (IMFs). These IMFs capture hierarchical implicit features in the form of decreasing spatial frequency. This research proposes using the last IMF (least varying component), also known as the Residual function, as a statistical prior for capturing local features, followed by fusing with the hierarchical convolutional features from the deep compressive sensing (CS) network. The central idea is that the Residue approximately represents the overall forest structure considering it is relatively homogenous due to the presence of vegetation. ADCoSNet utilizes this last IMF for generating sparse representation based on a set of CS measurement ratios. The research presents extensive experiments for reconstructing 3D LiDAR point clouds with high fidelity for various CS measurement ratios. Our approach achieves a maximum peak signal-to-noise ratio (PSNR) of 48.96 dB (approx. 8 dB better than reconstruction without data-dependent transforms) with reconstruction root mean square error (RMSE) of 7.21. It is envisaged that the proposed framework finds high potential as an end-to-end learning framework for generating adaptive and sparse representations to capture geometrical features for the 3D reconstruction of forests.

DOA Estimation Based on Convolutional Autoencoder in the Presence of Array Imperfections

Array imperfections may exist in an antenna system subject to non-ideal design and practical limitations. It is difficult to accurately model array imperfections, and thus complicated algorithms are usually inevitable for model-based methods to estimate the direction of arrival (DOA) with imperfect arrays. Deep neural network (DNN)-based methods do not need to rely on pre-modeled antenna array geometries, and have been explored to handle flawed array models because of their better flexibility than model-based methods. The DNN autoencoder (DAE) method has been proposed for the array imperfection problem, which decomposes the input into multiple components in different spatial subregions. These components have more concentrated distributions than the original input, which avoid a large number of connections and nodes used in the layers to realize DOA estimation classifiers. In this paper, we study the convolutional AE (CAE) method that substantially focuses on the learning of local features in a different manner from the previous DAE method. The advantage of the convolutional operation using a kernel in CAE is to capture features in a more efficient manner than the DAE, and thus be able to reduce the number of parameters that are required to be trained in the neural networks. From the numerical evaluation of DOA estimation accuracy, the proposed CAE method is also more resistant to the noise effect than the DAE method such that the CAE method has better accuracy at a lower signal-to-noise ratio.

Study on the spatialization of anthropogenic carbon emissions in China based on SVR-ZSSR

The spatialization of anthropogenic carbon emissions is of great significance for achieving the goal of "carbon peaking and neutrality" and promoting the development of carbon trading market. The SVR-ZSSR spatialization model is proposed to solve the problems of "insufficient local feature learning" and "scale dependence of driving factors" existing in the single model, and this model is applied to the study of the spatialization of carbon emissions in China in this paper. The results show that: (1) the simulation results of our proposed model show the distribution characteristics of "high in the East and low in the west"; On the micro-scale, the high carbon emission areas in the simulation results are all concentrated in the built-up land, while the carbon emission in the surrounding areas of the city is significantly lower than that in the center, which is similar to the spatial distribution trend of carbon emission in the existing database. (2) Compared with the results of SVR, the results of our proposed model increase the carbon emission ratio of built-up land in each province by 15.9% on average; Compared with the ODIAC database and SVR model, the carbon emission ratio of built-up land in Gansu Province, Qinghai Province, and other low-carbon emission areas has increased by about 25%.

HDA[formula omitted]L: Hierarchical Domain-Augmented Adaptive Learning for sketch-based 3D shape retrieval

The sketch-based 3D shape retrieval has been an active but challenging task for several decades. In this paper, we deeply analyze the challenges and propose a novel Hierarchical Domain-Augmented Adaptive Learning (HDA2L) for sketch-based 3D shape retrieval. The first notable challenge is the vast cross-modality discrepancies between sketches and 3D shapes. To this issue, the existing methods restrict the consistency of the final features by establishing a shared cross-domain loss but ignore the feature extraction process, resulting in a limited effect. We argue that the mutual information of the samples from the same class but different domains can provide an important cue to enhance common features captured in the feature extraction process. Thus, we design an Inter-Domain Augmented Network (Inter-DAN) by employing inter-domain feature correlation learning to capture cross-domain mutual information to learn augmented common global features for both sketches and 3D shapes. Another challenge is that the input sketch is various: it may be particularly abstract and contains only the overall outline of the target model, or it may be too sketchy and only contains some salient local regions of the target model. Though existing methods have demonstrated their capability to capture overall features, they always ignore the learning of local discriminative features and fail to adapt to the various changes of input sketches. To address this issue, we design an Intra-Domain Augmented Network (Intra-DAN) for sketches and 3D shapes, respectively, which learns augmented local discriminative features by adopting cascading cross-layer bilinear pooling operations. In addition, we design a Source-Agnostic Adversarial Network (SAAN) to accomplish the adaptive hierarchical domain features fusion, which forces the network to adaptively focus on more discriminative information from global features and local features and further adapt the diversity of the input sketches. The experiments on three benchmark datasets demonstrate that our method obtains superior retrieval performance than the state-of-the-art sketch-based 3D shape retrieval approaches.

Hyperspectral Image Super-Resolution Network Based on Cross-Scale Nonlocal Attention

Hyperspectral image (HSI) super-resolution generally means the fusion of low-spatial resolution hyperspectral image (LRHSI) and high-spatial resolution multi-spectral (MSI)/panchromatic (PAN) image (HRMPI) to get high-spatial resolution HSI (HRHSI). Existing fusion methods have not sufficiently considered the huge spectral and spatial resolution difference between the LRHSI and HRMPI. In addition, most DL-based methods that adopt the CNN structure are limited by its local feature learning, and it is difficult to exploit the global dependency of image features. To fully adapt to the huge modality difference between LRHSI and HRMPI and release the limitation of local feature learning, we design the cross spectral-scale and shift-window based cross spatial-scale non-local attention networks (CSSNet) to effectively fuse the LRHSI and HRMPI. These two networks could explicitly learn the spectral and spatial correlations between two input images. These correlations are then used to reconstruct the HRHSI feature, which makes the obtained HRHSI feature to maintain the spectral and spatial feature consistency with the input images. Finally, a ‘feature aggregation module’ is designed to aggregate the image features from these two networks and output the fused HRHSI. Extensive experimental results on both HM-fusion (fusion with MSI) and HP-fusion (fusion with PAN image) tasks demonstrate CSSNet’s state-of-the-art (SOTA) performance compared to other fusion methods. (The codes could be available at https://github.com/rs-lsl/CSSNet).

EB-LG Module for 3D Point Cloud Classification and Segmentation

Thanks to the development of deep learning technology and computer science, 3D point cloud analysis is becoming a research hotspot. Based on convolution operator, local feature learning is fundamental but far from perfect regarding point cloud analysis task. Actually, when people see a wing-like part (local features), our brain immediately associates it with a plane model (global features), and then we further trust that it's a wing based on the feedback from our brain. Such hidden features between local features and global features are commonly-existed, but ignored by existing methods. To this end, we propose Error feature Back-projection based Local-Global (EB-LG) feature learning module for better representations of point clouds. Specifically, EB-LG module adequately captures the hidden features firstly; and then borrowing from the successful idea of error-feedback mechanism, the learned hidden features will be back-projected to original local features, so that the enhanced local features are obtained. Serving as a plug-and-play, EB-LG module is lightweight and can be easily integrated into existing state-of-the-art networks to boost their performance. Extensive evaluations on both synthetic and real-world 3D point cloud benchmarks demonstrate the effectiveness and the generalization ability of our method.

AFU-Net: A Novel U-Net Network for Rice Leaf Disease Segmentation

Highlights The attention mechanism enhances the ability of the model to learn specific semantic information in encoder. The redesigned residual structure deepens the network while reducing the number of parameters. The feature extraction module and feature fusion module obtain richer boundary feature information and effectively integrate output results from different levels. The mIoU, mPA, and Precision values of AFU-Net in the self-built dataset are 87.25%, 92.23%, and 99.67%, respectively. Abstract. Rice diseases adversely affect rice growth and yield. Precise spot segmentation helps to assess the severity of the disease so that appropriate control measures can be taken. In this article, we propose a segmentation method called AFU-Net for rice leaf diseases, and its performance is verified through experiments. Based on the traditional UNet, this method incorporates an attention mechanism, a residual module and a feature fusion module (FFM). The attention mechanism is embedded in skip connections, which enhances the learning of particular semantic features in the encoder layer. In addition, the residual module is integrated into the decoder layer, which deepens the network and enables it to extract richer semantic information. The proposed FFM structure effectively enhances the learning of boundary information and local detail features. The experimental results show that the mean intersection over union (mIoU), mean pixel accuracy (mPA) and Precision of the proposed model on the self-built rice leaf disease segmentation dataset are 87.25%, 92.23%, and 99.67%, respectively. All three evaluation indexes were improved over the control group, while the proposed model had the lowest number of parameters and displayed a good segmentation effect for smaller disease points and disease parts with less obvious characteristics. Keywords: Attention mechanism, Feature fusion module, Residual module, Rice leaves, UNet model.

Convolutional Transformer based Local and Global Feature Learning for Speech Enhancement

Speech enhancement (SE) is an important method for improving speech quality and intelligibility in noisy environments where received speech is severely distorted by noise. An efficient speech enhancement system relies on accurately modelling the long-term dependencies of noisy speech. Deep learning has greatly benefited by the use of transformers where long-term dependencies can be modelled more efficiently with multi-head attention (MHA) by using sequence similarity. Transformers frequently outperform recurrent neural network (RNN) and convolutional neural network (CNN) models in many tasks while utilizing parallel processing. In this paper we proposed a two-stage convolutional transformer for speech enhancement in time domain. The transformer considers global information as well as parallel computing, resulting in a reduction of long-term noise. In the proposed work unlike two-stage transformer neural network (TSTNN) different transformer structures for intra and inter transformers are used for extracting the local as well as global features of noisy speech. Moreover, a CNN module is added to the transformer so that short-term noise can be reduced more effectively, based on the ability of CNN to extract local information. The experimental findings demonstrate that the proposed model outperformed the other existing models in terms of STOI (short-time objective intelligibility), and PESQ (perceptual evaluation of the speech quality).

ViTALnet: Anomaly on Industrial Textured Surfaces With Hybrid Transformer

The coexistence of subtle and long-range anomalies in real-world industrial applications brings significant challenges for anomaly localization. Existing methods typically train deep models by utilizing the multi-level patches or layers-fusion approaches for learning the global-local distribution. However, these methods do not consider learning local and global features simultaneously, which suffer from inaccurate localization results. To this end, a hybrid transformer model (ViTALnet) is proposed here, which is established based on fine-grained feature reconstruction. Our ViTALnet first adopts the vision transformer to extract local discriminatory features as feature representation, which leverages the global semantic capturing capability. Then, an anomaly estimation module is proposed by integrating global attention and a pyramidal architecture to enhance contextual information for fine-grain anomaly localization. The experiments were extensively conducted on industrial anomaly localization datasets MVTec AD-Texture, NanoTWICE, and general textured datasets KolektorSDD2, MT Defect, and Dot-patterned Fabric, where our proposed ViTALnet outperformed major state-of-the-art methods.

Unsupervised learning of local features for person re-identification with loss function

Unsupervised learning of local features for person re-identification with loss function

Seeing Through Darkness: Visual Localization at Night via Weakly Supervised Learning of Domain Invariant Features

Long term visual localization has to conquer the problem of robust matching images with dramatic photometric changes caused by different seasons, weather conditions, natural and man-made illumination changes, etc. Due to the practical demand of applying visual localization at night, either for autonomous driving or for augmented reality related applications, how to extract keypoints and descriptors with robustness to day-night illumination changes on images has became the bottleneck. This paper proposes an adversarial learning based solution to harvest from the weakly domain labels of day and night images, along with the point level correspondences among day time images, to achieve robust local feature extraction and description across day-night images. The key idea is to learn a discriminator to distinguish whether a feature map is generated from the day image or night image, and simultaneously to adjust the parameters of feature extraction network so as to fool the discriminator. After adversarial training of the discriminator and feature extraction network, the feature extraction network finally reaches a stable status so that the extracted feature maps are robust to day-night photometric changes, based on which day-night domain invariant keypoints and descriptors can be extracted. Compared to existing local feature learning methods, it only requires an additional set of easily captured night images to improve the domain invariance of learned features. Experiments on two challenging day-night visual localization benchmarks show the effectiveness of the proposed method. In addition, this paper revisits the widely used image matching metrics on HPatches and finds that recall of different methods reflects their relative localization performance, while previously used mean matching accuracy is not essentially related to downstream tasks like visual localization.

SharpFormer: Learning Local Feature Preserving Global Representations for Image Deblurring

The goal of dynamic scene deblurring is to remove the motion blur presented in a given image. To recover the details from the severe blurs, conventional convolutional neural networks (CNNs) based methods typically increase the number of convolution layers, kernel-size, or different scale images to enlarge the receptive field. However, these methods neglect the non-uniform nature of blurs, and cannot extract varied local and global information. Unlike the CNNs-based methods, we propose a Transformer-based model for image deblurring, named SharpFormer, that directly learns long-range dependencies via a novel Transformer module to overcome large blur variations. Transformer is good at learning global information but is poor at capturing local information. To overcome this issue, we design a novel Locality preserving Transformer (LTransformer) block to integrate sufficient local information into global features. In addition, to effectively apply LTransformer to the medium-resolution features, a hybrid block is introduced to capture intermediate mixed features. Furthermore, we use a dynamic convolution (DyConv) block, which aggregates multiple parallel convolution kernels to handle the non-uniform blur of inputs. We leverage a powerful two-stage attentive framework composed of the above blocks to learn the global, hybrid, and local features effectively. Extensive experiments on the GoPro and REDS datasets show that the proposed SharpFormer performs favourably against the state-of-the-art methods in blurred image restoration.

TC-Net: A Transformer Capsule Network for EEG-based emotion recognition

Deep learning has recently achieved remarkable success in emotion recognition based on Electroencephalogram (EEG), in which convolutional neural networks (CNNs) are the mostly used models. However, due to the local feature learning mechanism, CNNs have difficulty in capturing the global contextual information involving temporal domain, frequency domain, intra-channel and inter-channel. In this paper, we propose a Transformer Capsule Network (TC-Net), which mainly contains an EEG Transformer module to extract EEG features and an Emotion Capsule module to refine the features and classify the emotion states. In the EEG Transformer module, EEG signals are partitioned into non-overlapping windows. A Transformer block is adopted to capture global features among different windows, and we propose a novel patch merging strategy named EEG-PatchMerging (EEG-PM) to better extract local features. In the Emotion Capsule module, each channel of the EEG feature maps is encoded into a capsule to better characterize the spatial relationships among multiple features. Experimental results on two popular datasets (i.e., DEAP and DREAMER) demonstrate that the proposed method achieves the state-of-the-art performance in the subject-dependent scenario. Specifically, on DEAP (DREAMER), our TC-Net achieves the average accuracies of 98.76% (98.59%), 98.81% (98.61%) and 98.82% (98.67%) at valence, arousal and dominance dimensions, respectively. Moreover, the proposed TC-Net also shows high effectiveness in multi-state emotion recognition tasks using the popular VA and VAD models. The main limitation of the proposed model is that it tends to obtain relatively low performance in the cross-subject recognition task, which is worthy of further study in the future.

3DCTN: 3D Convolution-Transformer Network for Point Cloud Classification

Point cloud classification is a fundamental task in 3D applications. However, it is challenging to achieve effective feature learning due to the irregularity and unordered nature of point clouds. Lately, 3D Transformers have been adopted to improve point cloud processing. Nevertheless, massive Transformer layers tend to incur huge computational and memory costs. This paper presented a novel hierarchical framework that incorporated convolutions with Transformers for point cloud classification, named 3D Convolution-Transformer Network (3DCTN). It combined the strong local feature learning ability of convolutions with the remarkable global context modeling capability of Transformers. Our method had two main modules operating on the downsampling point sets. Each module consisted of a multi-scale local feature aggregating (LFA) block and a global feature learning (GFL) block, which were implemented by using the Graph Convolution and Transformer respectively. We also conducted a detailed investigation on a series of self-attention variants to explore better performance for our network. Various experiments on ModelNet40 and ScanObjectNN datasets demonstrated that our method achieves state-of-the-art classification performance with a lightweight design. The code is publicly available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/d62lu/3DCTN</uri> .

Lightweight Semantic-Guided Neural Networks Based on Single Head Attention for Action Recognition

Skeleton-based action recognition can achieve a relatively high performance by transforming the human skeleton structure in an image into a graph and applying action recognition based on structural changes in the body. Among the many graph convolutional network (GCN) approaches used in skeleton-based action recognition, semantic-guided neural networks (SGNs) are fast action recognition algorithms that hierarchically learn spatial and temporal features by applying a GCN. However, because an SGN focuses on global feature learning rather than local feature learning owing to the structural characteristics, there is a limit to an action recognition in which the dependency between neighbouring nodes is important. To solve these problems and simultaneously achieve a real-time action recognition in low-end devices, in this study, a single head attention (SHA) that can overcome the limitations of an SGN is proposed, and a new SGN-SHA model that combines SHA with an SGN is presented. In experiments on various action recognition benchmark datasets, the proposed SGN-SHA model significantly reduced the computational complexity while exhibiting a performance similar to that of an existing SGN and other state-of-the-art methods.

Learning Asymmetric and Local Features in Multi-Dimensional Data Through Wavelets With Recursive Partitioning

Effective learning of asymmetric and local features in images and other data observed on multi-dimensional grids is a challenging objective critical for a wide range of image processing applications involving biomedical and natural images. It requires methods that are sensitive to local details while fast enough to handle massive numbers of images of ever increasing sizes. We introduce a probabilistic model-based framework that achieves these objectives by incorporating adaptivity into discrete wavelet transforms (DWT) through Bayesian hierarchical modeling, thereby allowing wavelet bases to adapt to the geometric structure of the data while maintaining the high computational scalability of wavelet methods-linear in the sample size (e.g., the resolution of an image). We derive a recursive representation of the Bayesian posterior model which leads to an exact message passing algorithm to complete learning and inference. While our framework is applicable to a range of problems including multi-dimensional signal processing, compression, and structural learning, we illustrate its work and evaluate its performance in the context of image reconstruction using real images from the ImageNet database, two widely used benchmark datasets, and a dataset from retinal optical coherence tomography and compare its performance to state-of-the-art methods based on basis transforms and deep learning.

Specific Emitter Identification Based on a Hybrid Deep Neural Network for ACARS Authentication

With the increasing variety and quantity of aircraft, there is a potential threat to the security of the Aircraft Communications Addressing and Reporting System (ACARS) due to the lack of reliable authentication measures. This paper proposes a novel specific emitter identification (SEI) algorithm based on a hybrid deep neural network (DNN) for ACARS authentication. Our deep learning architecture is a combination of Deep Residual Shrinkage Network (DRSN), Bidirectional-LSTM (Bi-LSTM), and attention mechanism (AM), which perform the functions of local and global feature learning and feature focusing, respectively, so that the individual information hidden in the signal waveform can be thoroughly mined. We introduce soft thresholding as a nonlinear transformation in the DSRN to enhance robustness against noise and adopt a low-cost training strategy for new data using transfer learning. The proposed SEI algorithm is optimized and evaluated based on real-world ACARS signals captured in the Xianyang airport. Experimental results demonstrate that our algorithm can distinguish authorized entities from unauthorized entities and obtain an identification accuracy of up to 0.980. In addition, the design rationality and the superiority over other algorithms are verified through the experiments.