Attention-based Network Research Articles

U-MSAA-Net: A Multiscale Additive Attention-Based Network for Pixel-Level Identification of Finfish and Krill in Echograms

This paper addresses the detection of finfish and krill in echograms. Finfish, in particular Pacific hake, are used both as human food and fish meal. Krill, harvested for aquaculture and aquariums, are a primary food source for finfish, including hake. Thus, spatial distributions of hake follow that of krill. Stock assessments need an accurate differentiation of krill from finfish (hake) in acoustic echograms. This paper proposes a semantic segmentation paradigm for the pixel-level classification of multi-frequency information to detect co-occurring finfish and krill. This paradigm is highly relevant for identifying cloud-like, diffuse krill aggregations that are intertwined with small, often sparse and sometimes dense schools of finfish. We propose U-MSAA-Net, a deep learning U-Net-like framework with novel multi-scale additive attention (MSAA) modules. MSAA modules allow us to leverage all contextual and local information from feature maps available at any given level of the decoding phase of the network, yielding an efficient suppression of the feature responses from regions with lesser semantic value. Experimental results on a new finfish and krill data set spanning across nine months of acoustic data and covering various situations show that U-MSAA-Net outperforms both traditional, texture-based machine learning methods, and deep learning methods based on state-of-the-art semantic segmentation networks. Additional experiments on a data set containing schools of herring and salmon confirm the versatility of U-MSAA-Net and its superiority in terms of accuracy and ability to detect schools of varying sizes. U-MSAA-Net is the first step in creating a comprehensive tool for stock and ecosystem assessments.

Crysformer: An attention-based graph neural network for properties prediction of crystals

We present a novel approach for the prediction of crystal material properties that is distinct from the computationally complex and expensive density functional theory (DFT)-based calculations. Instead, we utilize an attention-based graph neural network that yields high-accuracy predictions. Our approach employs two attention mechanisms that allow for message passing on the crystal graphs, which in turn enable the model to selectively attend to pertinent atoms and their local environments, thereby improving performance. We conduct comprehensive experiments to validate our approach, which demonstrates that our method surpasses existing methods in terms of predictive accuracy. Our results suggest that deep learning, particularly attention-based networks, holds significant promise for predicting crystal material properties, with implications for material discovery and the refined intelligent systems.

FanoutNet: A Neuralized PCB Fanout Automation Method Using Deep Reinforcement Learning

In modern electronic manufacturing processes, multi-layer Printed Circuit Board (PCB) routing requires connecting more than hundreds of nets with perplexing topology under complex routing constraints and highly limited resources, so that takes intense effort and time of human engineers. PCB fanout as a pre-design of PCB routing has been proved to be an ideal technique to reduce the complexity of PCB routing by pre-allocating resources and pre-routing. However, current PCB fanout design heavily relies on the experience of human engineers, and there is no existing solution for PCB fanout automation in industry, which limits the quality of PCB routing automation. To address the problem, we propose a neuralized PCB fanout method by deep reinforcement learning. To the best of our knowledge, we are the first in the literature to propose the automation method for PCB fanout. We combine with Convolution Neural Network (CNN) and attention-based network to train our fanout policy model and value model. The models learn representations of PCB layout and netlist to make decisions and evaluations in place of human engineers. We employ Proximal Policy Optimization (PPO) to update the parameters of the models. In addition, we apply our PCB fanout method to a PCB router to improve the quality of PCB routing. Extensive experimental results on real-world industrial PCB benchmarks demonstrate that our approach achieves 100% routability in all industrial cases and improves wire length by an average of 6.8%, which makes a significant improvement compared with the state-of-the-art methods.

Automatic endoscopic navigation based on attention-based network for Nasotracheal Intubation

Nasotracheal intubation (NTI) is one of the most commonly performed procedures in anesthesia and is considered the gold standard for securing the airway of patients. Endoscope operation is critical to the success of NTI. However, this operation remains challenging as it requires the surgeon to classify anatomical landmarks and detect heading targets of the endoscope tip in a sequence of monocular images. To alleviate this problem, this study presents a learning-based navigation method that automatically classifies four different anatomical landmarks and detects the heading target of the endoscope tip from endoscopic images. First, an end-to-end multitask network is introduced that consists of a branch for anatomical landmark classification and another for heading target detection. In addition, a convolutional attention module is designed to improve network performance by combining spatial and channel attention. Second, an endoscopic dataset named intuNav is built for network training. The trained network calculates navigation information without image prior knowledge. Finally, extensive experiments on the built dataset and endoscopic videos demonstrate the high performance of our method, achieving classification (94%) and detection (79.4%) accuracies. The results also indicate that the proposed method is effective and efficient in navigation generation during NTI.

UFO-Net: A Linear Attention-Based Network for Point Cloud Classification

Three-dimensional point cloud classification tasks have been a hot topic in recent years. Most existing point cloud processing frameworks lack context-aware features due to the deficiency of sufficient local feature extraction information. Therefore, we designed an augmented sampling and grouping module to efficiently obtain fine-grained features from the original point cloud. In particular, this method strengthens the domain near each centroid and makes reasonable use of the local mean and global standard deviation to extract point cloud’s local and global features. In addition to this, inspired by the transformer structure UFO-ViT in 2D vision tasks, we first tried to use a linearly normalized attention mechanism in point cloud processing tasks, investigating a novel transformer-based point cloud classification architecture UFO-Net. An effective local feature learning module was adopted as a bridging technique to connect different feature extraction modules. Importantly, UFO-Net employs multiple stacked blocks to better capture feature representation of the point cloud. Extensive ablation experiments on public datasets show that this method outperforms other state-of-the-art methods. For instance, our network performed with 93.7% overall accuracy on the ModelNet40 dataset, which is 0.5% higher than PCT. Our network also achieved 83.8% overall accuracy on the ScanObjectNN dataset, which is 3.8% better than PCT.

Contact-conditioned hand-held object reconstruction from single-view images

Reconstructing the shape of hand-held objects from single-view color images is a long-standing problem in computer vision and computer graphics. The task is complicated by the ill-posed nature of single-view reconstruction, as well as potential occlusions due to both the hand and the object. Previous works mostly handled the problem by utilizing known object templates as priors to reduce the complexity. In contrast, our paper proposes a novel approach without knowing the object templates beforehand but by exploiting prior knowledge of contacts in hand-object interactions to train an attention-based network that can perform precise hand-held object reconstructions with only a single forward pass in inference. The network we propose encodes visual features together with contact features using a multi-head attention module as a way to condition the training of a neural field representation. This neural field representation outputs a Signed Distance Field representing the reconstructed object and extensive experiments on three well-known datasets demonstrate that our method achieves superior reconstruction results even under severe occlusion compared to the state-of-the-art techniques.

Distant supervision for relation extraction with hierarchical attention-based networks

Distant supervision for relation extraction with hierarchical attention-based networks

XBound-Former: Toward Cross-scale Boundary Modeling in Transformers.

Skin lesion segmentation from dermoscopy images is of great significance in the quantitative analysis of skin cancers, which is yet challenging even for dermatologists due to the inherent issues, i.e., considerable size, shape and color variation, and ambiguous boundaries. Recent vision transformers have shown promising performance in handling the variation through global context modeling. Still, they have not thoroughly solved the problem of ambiguous boundaries as they ignore the complementary usage of the boundary knowledge and global contexts. In this paper, we propose a novel cross-scale boundary-aware transformer, XBound-Former, to simultaneously address the variation and boundary problems of skin lesion segmentation. XBound-Former is a purely attention-based network and catches boundary knowledge via three specially designed learners. First, we propose an implicit boundary learner (im-Bound) to constrain the network attention on the points with noticeable boundary variation, enhancing the local context modeling while maintaining the global context. Second, we propose an explicit boundary learner (ex-Bound) to extract the boundary knowledge at multiple scales and convert it into embeddings explicitly. Third, based on the learned multi-scale boundary embeddings, we propose a cross-scale boundary learner (X-Bound) to simultaneously address the problem of ambiguous and multi-scale boundaries by using learned boundary embedding from one scale to guide the boundary-aware attention on the other scales. We evaluate the model on two skin lesion datasets and one polyp lesion dataset, where our model consistently outperforms other convolution- and transformer-based models, especially on the boundary-wise metrics. All resources could be found in https://github.com/jcwang123/xboundformer.

AFFNet: An Attention-Based Feature-Fused Network for Surface Defect Segmentation

Recently, deep learning methods have widely been employed for surface defect segmentation in industrial production with remarkable success. Nevertheless, accurate segmentation of various types of defects is still challenging due to their irregular appearance and low contrast with the background. In light of this challenge, we propose an attention-based network with a U-shaped structure, referred to as AFFNet. In the encoder part, we present a newly designed module, Residual-RepGhost-Dblock (RRD), which focuses on the extraction of more representative features using CA attention and dilated convolution with varying expansion rates without a concomitant increase in the parameters. In the decoder part, we introduce a novel global feature attention (GFA) module to selectively fuse low-level and high-level features, suppressing distracting information such as background. Moreover, considering the imbalance of the dataset sampled from actual industrial production and the difficulty of training samples with small defects, we use the online hard sample mining (OHEM) cross-entropy loss function to improve the learning ability of hard samples. Experimental results on the NEU-seg dataset demonstrate the superiority of our method over other state-of-the-art methods.

Pixel-wise detection algorithm for crack structural reconstruction based on rock CT images

This paper describes a pixel-wise detection algorithm to reconstruct rock cracks using CT images. Through reviewing the shortcomings of previous studies, a pixel-level labeled dataset with 1 k CT images covering diverse rocks is first introduced. Then, following four discoveries about how to design crack detection models better, three novel modules are presented, including 1) an inverted residual block with three hierarchical residual-like mixed connected branches, 2) the attention-based upsampling method that simultaneously executes multi-layer feature fusion, and 3) a multi-representative vector classifier. By employing these novel components and exploring the network structure, two attention-based networks (CTRCrack-T/B) with skip connections are presented for crack detection. Compared with previous state-of-the-art models, the proposed approach exhibits the best performance over the rock CT image dataset, achieving 87.8% mean intersection over union at 46.3 frames per second (5122 sizes). And abundant experiments report the superiority of CTRCracks in crack detection for other structures.

Toward surface defect detection in electronics manufacturing by an accurate and lightweight YOLO-style object detector

In electronics manufacturing, surface defect detection is very important for product quality control, and defective products can cause severe customer complaints. At the same time, in the manufacturing process, the cycle time of each product is usually very short. Furthermore, high-resolution input images from high-resolution industrial cameras are necessary to meet the requirements for high quality control standards. Hence, how to design an accurate object detector with real-time inference speed that can accept high-resolution input is an important task. In this work, an accurate YOLO-style object detector was designed, ATT-YOLO, which uses only one self-attention module, many-scale feature extraction and integration in the backbone and feature pyramid, and an improved auto-anchor design to address this problem. There are few datasets for surface detection in electronics manufacturing. Hence, we curated a dataset consisting of 14,478 laptop surface defects, on which ATT-YOLO achieved 92.8% mAP0.5 for the binary-class object detection task. We also further verified our design on the COCO benchmark dataset. Considering both computation costs and the performance of object detectors, ATT-YOLO outperforms several state-of-the-art and lightweight object detectors on the COCO dataset. It achieves a 44.9% mAP score and 21.8 GFLOPs, which is better than the compared models including YOLOv8-small (44.9%, 28.6G), YOLOv7-tiny-SiLU (38.7%, 13.8G), YOLOv6-small (43.1%, 44.2G), pp-YOLOE-small (42.7%, 17.4G), YOLOX-small (39.6%, 26.8G), and YOLOv5-small (36.7%, 17.2G). We hope that this work can serve as a useful reference for the utilization of attention-based networks in real-world situations.

Cross-sensor remote sensing imagery super-resolution via an edge-guided attention-based network

The deep learning based super-resolution (SR) methods have recently achieved remarkable progress in the reconstruction of ideally simulated high-quality remote sensing image datasets. However, due to the large variation in image quality caused by the complex degradation factors, their performance decreases dramatically on real-world images acquired by different satellite sensors. To this end, we propose a cross-sensor SR framework that consists of a cross-sensor degradation modeling strategy for bridging the gap between the images obtained by the source and target sensors, and an edge-guided attention-based SR (EGASR) network to promote the learning of high-frequency feature representation. Specifically, we build a degradation pool on the low-resolution (LR) target sensor to produce a degraded training dataset simulated from the high-resolution (HR) images obtained by the source sensor. Furthermore, the EGASR network, which employs the edge-guided residual attention block (EGRAB) to introduce implicit edge prior to enhance edge-related information, is embedded in the cross-sensor SR framework for reconstructing HR results with sharp details. The proposed method is applied on images from the Chinese Gaofen (GF) satellite sensors and compared to several representative SR methods. An ideally simulated GF-2 LR/HR image set with only downsampling considered is first used to evaluate the effectiveness of the proposed EGASR network. Moreover, GF-2/GF-1 and GF-2/GF-6 cross-sensor SR datasets are constructed by synthesizing GF-2 degraded image pairs with the degradation pools estimated from the GF-1 and GF-6 images, respectively. The results show that: 1) the proposed EGASR model shows superiority in reconstructing textural details and edge features, and achieves the best results among the state-of-art SR methods involved in comparison; 2) the cross-sensor SR framework significantly promotes the model’s ability to super-resolve real-world LR images acquired by the target satellite sensors, e.g., the NIQE values are improved by at least 30% and 34% on average with respect to other comparative methods for GF-2/GF-1 and GF-2/GF-6 datasets in the real experiments, respectively. Code is available at https://github.com/zhonghangqiu/EGASR.

A Dynamic Fusion of Local and Non-Local Features-Based Feedback Network on Super-Resolution

Many Symmetry blocks were proposed in the Single Image Super-Resolution (SISR) task. The Attention-based block is powerful but costly on non-local features, while the Convolutional-based block is good at efficiently handling the local features. However, assembling two different Symmetry blocks will generate an Asymmetry block, making the classic Symmetry-block-based Super-Resolution (SR) architecture fail to deal with these Asymmetry blocks. In this paper, we proposed a new Dynamic fusion of Local and Non-local features-based Feedback Network (DLNFN) for SR, which focus on optimizing the traditional Symmetry-block-based SR architecture to hold two Symmetry blocks in parallel, making two Symmetry-blocks working on what they do best. (1) We introduce the Convolutional-based block for the local features and Attention-based network block for non-local features and propose the Delivery–Adjust–Fusion framework to hold these blocks. (2) we propose a Dynamic Weight block (DW block) which can generate different weight values to fuse the outputs on different feedback iterations. (3) We introduce the MAConv layer to optimize the In block, which is critical for our two blocks-based feedback algorithm. Experiments show our proposed DLNFN can take full advantage of two different blocks and outperform other state-of-the-art algorithms.

SCA-Net: Spatial and channel attention-based network for 3D point clouds

A key challenge in data processing of 3D point cloud data is how to identify key regions in the point cloud from geometric relationships. Previous work has attempted to overcome this deficiency by slicing and dicing data. Such an approach limits the model’s ability to effectively learn global information. In this paper, in order to better extract features and obtain geometric information we propose a Point Attention (PointAT) model and propose Attention Value (AT value) model for feature fusion to apply geometric relationship to the data. Then, we propose a new Spatial and Channel Attention-based network (SCA). The SCA is the overall structure of the network, and the main purpose is to connect PointAT and AT value model, then capturing meaningful geometric information by applying the geometric relationship between point clouds patches to the model, then propose a auto pooling framework to extract to global features. In this work, we concentrate on learning geometric relationship between point cloud data. For this purpose, we introduce a point attention model based on spatial and channel attention to learn the geometric relationship between point clouds, and further combine the geometric relationship with the point cloud data by the AT Value Model. Finally, we introduce a adaptive downsampling structure, Autopooling. This downsampling structure consider each point’s importance weight and picking key points adaptively, which can be used with convolutional networks. Extensive experiments conducted on two benchmark datasets (ModelNet40 and ShapeNet) clearly demonstrate the effectiveness of our SCA and SCA-Auto (SCAA with Auto pooling) methods.

BAFN: Bi-Direction Attention Based Fusion Network for Multimodal Sentiment Analysis

Attention-based networks currently identify their effectiveness in multimodal sentiment analysis. However, existing methods ignore the redundancy of auxiliary modalities. More importantly, existing methods only attend to top-down attention (static process) or down-top attention (implicit process), leading to the coarse-grained multimodal sentiment context. In this paper, during the preprocessing period, we first propose the multimodal dynamic enhanced block to capture the intra-modality sentiment context. This can effectively decrease the intra-modality redundancy of auxiliary modalities. Furthermore, the bi-direction attention block is proposed to capture fine-grained multimodal sentiment context via the novel bi-direction multimodal dynamic routing mechanism. Specifically, the bi-direction attention block first highlights the explicit and low-level multimodal sentiment context. Then, the low-level multimodal context is transmitted to a carefully designed bi-direction multimodal dynamic routing procedure. This allows us to dynamically update and investigate high-level and much more fine-grained multimodal sentiment contexts. The experiments demonstrate that our fusion network can achieve state-of-the-art performance. Notably, our model outperforms the best baseline on the metric ‘Acc-7’ with an improvement of 6.9%.

Contour extraction of medical images using an attention-based network

A comprehensive analysis of medical images is important, as it assists in early screening and clinical treatment as well as subsequent rehabilitation. In general, the contour information can elaborately describe the shape and size of lesions in a medical image, which accurately reflects specific and valuable properties that facilitate the identification of abnormalities, so contour extraction is meaningful. However, the traditional method usually depends on the output of image segmentation, which causes blurred edges and loss of details. To address these issues, an effective attention-based network for contour extraction is proposed, where a model mixed with U-Net and an attention network is utilized to extract image features, and a multilayer perceptron (MLP) is employed to classify those features to obtain a clear contour. Compared with the existing methods, the experimental results on three datasets (Herlev, Drosophila, and ISIC-2017) show that the accuracy reaches approximately 93–98 % by using the proposed network, and the number of parameters is 46.4 % less than the deep active contour network (DACN). Such performances are impressive when considering accuracy and the number of parameters as the key concerns. Therefore, this study reduces the model computation with almost no loss of accuracy, which can satisfy clinical requirements for medical image analysis.

Variety identification of sweet maize seeds based on hyperspectral imaging combined with deep learning

Seed classification and identification is of great importance for cultivar breeding. In this study, hyperspectral imaging technique combined with deep learning approaches are employed to identify the varieties of sweet maize seeds. First, a total of 1000 seeds, including five varieties of sweet maize seeds, are extracted and preprocessed. Secondly, support vector machine (SVM), K-nearest neighbors (KNN), extreme learning machine (ELM), and backpropagation (BP) are used for the establishment of classification models. Finally, convolutional neural networks (CNN), long short-term memory (LSTM), three-dimensional convolutional neural networks (3DCNN), multiple spectral resolution 3D convolutional neural network (MSR-3DCNN), spatial attention-based network (SATNet) and the proposed CNN-LSTM are applied to discriminate spectral images of sweet maize seeds. The experimental analysis results show that the deep learning model performed best with a classification accuracy of over 95% in the training and test sets. Besides, the performance of the proposed CNN-LSTM is slightly better than the other five models. The overall study demonstrated that deep learning combined with the hyperspectral image could potentially be a valuable alternative for identifying the variety of sweet maize seeds.

Meta-Information Fusion of Hierarchical Semantics Dependency and Graph Structure for Structured Text Classification

Structured text with plentiful hierarchical structure information is an important part in real-world complex texts. Structured text classification is attracting more attention in natural language processing due to the increasing complexity of application scenarios. Most existing methods treat structured text from a local hierarchy perspective, focusing on the semantics dependency and the graph structure of the structured text independently. However, structured text has global hierarchical structures with sophisticated dependency when compared to unstructured text. According to the variety of structured texts, it is not appropriate to use the existing methods directly. The function of distinction information within semantics dependency and graph structure for structured text, referred to as meta-information, should be stated more precisely. In this article, we propose HGMETA, a novel meta-information embedding frame network for structured text classification, to obtain the fusion embedding of hierarchical semantics dependency and graph structure in a structured text, and to distill the meta-information from fusion characteristics. To integrate the global hierarchical features with fused structured text information, we design a hierarchical LDA module and a structured text embedding module. Specially, we employ a multi-hop message passing mechanism to explicitly incorporate complex dependency into a meta-graph. The meta-information is constructed from meta-graph via neighborhood-based propagation to distill redundant information. Furthermore, using an attention-based network, we investigate the complementarity of semantics dependency and graph structure based on global hierarchical characteristics and meta-information. Finally, the fusion embedding and the meta-information can be straightforwardly incorporated for structured text classification. Experiments conducted on three real-world datasets show the effectiveness of meta-information and demonstrate the superiority of our method.

Outlier Vehicle Trajectory Detection Using Deep Autoencoders in Santiago, Chile.

In the last decade, a large amount of data from vehicle location sensors has been generated due to the massification of GPS systems to track them. This is because these sensors usually include multiple variables such as position, speed, angular position of the vehicle, etc., and, furthermore, they are also usually recorded in very short time intervals. On the other hand, routes are often generated so that they do not correspond to reality, due to artifacts such as buildings, bridges, or sensor failures and where, due to the large amount of data, visual analysis of human expert is unable to detect genuinely anomalous routes. The presence of such abnormalities can lead to faulty sensors being detected which may allow sensor replacement to reliably track the vehicle. However, given the reliability of the available sensors, there are very few examples of such anomalies, which can make it difficult to apply supervised learning techniques. In this work we propose the use of unsupervised deep neural network models based on stacked autoencoders to detect anomalous routes in vehicles within Santiago de Chile. The results show that the proposed model is capable of effectively detecting anomalous paths in real data considering validation given by an expert user, reaching a performance of 82.1% on average. As future work, we propose to incorporate the use of Long Short-Term Memory (LSTM) and attention-based networks in order to improve the detection of anomalous trajectories.

Vector boson fusion versus gluon-gluon fusion Higgs boson production with full-event deep learning: Toward a decay-agnostic tagger

We study the benefits of jet- and event-level deep learning methods in distinguishing vector boson fusion (VBF) from gluon-gluon fusion (GGF) Higgs production at the LHC. We show that a variety of classifiers (CNNs, attention-based networks) trained on the complete low-level inputs of the full event achieve significant performance gains over shallow machine learning methods (BDTs) trained on jet kinematics and jet shapes, and we elucidate the reasons for these performance gains. Finally, we take initial steps toward the possibility of a VBF vs GGF tagger that is agnostic to the Higgs decay mode, by demonstrating that the performance of our event-level CNN does not change when the Higgs decay products are removed. These results highlight the potentially powerful benefits of event-level deep learning at the LHC.