Position Embedding Research Articles

MonoCAPE: Monocular 3D object detection with coordinate-aware position embeddings

3D monocular detection remains to be a focal point of research, particularly due to its capacity to deliver available precision under conditions of low cost and simplified configurations, making it especially valuable in fields like autonomous driving. Current 3D object detection methods often overlook the spatial information missing from images, which is critical to spatial perception, and optimize bounding box attributes separately, failing to meet the requirements of autonomous driving. We introduce MonoCAPE, a novel 3D detection framework addressing these issues by encoding spatial information and co-optimizing attributes through a Coordinate-Aware Position Encoding (CAPE) Generator and a Task Co-optimization Strategy (TCS). The CAPE Generator produces sparse positional embeddings, enabling spatial awareness with low computational cost, while the TCS utilizes Gaussian modeling to prevent suboptimal outputs. In this way, our framework comprehensively takes into account what existing approaches ignore. Extensive experiments on the KITTI dataset demonstrate MonoCAPE significantly improves AP3D and APBEV metrics compared to existing advanced methods.

Extraction of Winter Wheat Planting Plots with Complex Structures from Multispectral Remote Sensing Images Based on the Modified Segformer Model

As one of the major global food crops, the monitoring and management of the winter wheat planting area is of great significance for agricultural production and food security worldwide. Today, the development of high-resolution remote sensing imaging technology has provided rich sources of data for extracting the visual planting information of winter wheat. However, the existing research mostly focuses on extracting the planting plots that have a simple terrain structure. In the face of diverse terrain features combining mountainous areas, plains, and saline alkali land, as well as small-scale but complex planting structures, the extraction of planting plots through remote sensing imaging is subjected to great challenges in terms of recognition accuracy and model complexity. In this paper, we propose a modified Segformer model for extracting winter wheat planting plots with complex structures in rural areas based on the 0.8 m high-resolution multispectral data obtained from the Gaofen-2 satellite, which significantly improves the extraction accuracy and efficiency under complex conditions. In the encoder and decoder of this method, new modules were developed for the purpose of optimizing the feature extraction and fusion process. Specifically, the improvement measures of the proposed method include: (1) The MixFFN module in the original Segformer model is replaced with the Multi-Scale Feature Fusion Fully-connected Network (MSF-FFN) module, which enhances the model’s representation ability in handling complex terrain features through multi-scale feature extraction and position embedding convolution; furthermore, the DropPath mechanism is introduced to reduce the possibility of overfitting while improving the model’s generalization ability. (2) In the decoder part, after fusing features at four different scales, a CoordAttention module is added, which can precisely locate important regions with enhanced features in the images by utilizing the coordinate attention mechanism, therefore further improving the model’s extraction accuracy. (3) The model’s input data are strengthened by incorporating multispectral indices, which are also conducive to the improvement of the overall extraction accuracy. The experimental results show that the accuracy rate of the modified Segformer model in extracting winter wheat planting plots is significantly increased compared to traditional segmentation models, with the mean Intersection over Union (mIOU) and mean Pixel Accuracy (mPA) reaching 89.88% and 94.67%, respectively (an increase of 1.93 and 1.23 percentage points, respectively, compared to the baseline model). Meanwhile, the parameter count and computational complexity are significantly reduced compared to other similar models. Furthermore, when multispectral indices are input into the model, the mIOU and mPA reach 90.97% and 95.16%, respectively (an increase of 3.02 and 1.72 percentage points, respectively, compared to the baseline model).

Investigation of Bird Sound Transformer Modeling and Recognition

Birds play a pivotal role in ecosystem and biodiversity research, and accurate bird identification contributes to the monitoring of biodiversity, understanding of ecosystem functionality, and development of effective conservation strategies. Current methods for bird sound recognition often involve processing bird songs into various acoustic features or fusion features for identification, which can result in information loss and complicate the recognition process. At the same time, the recognition method based on raw bird audio has not received widespread attention. Therefore, this study proposes a bird sound recognition method that utilizes multiple one-dimensional convolutional neural networks to directly learn feature representations from raw audio data, simplifying the feature extraction process. We also apply positional embedding convolution and multiple Transformer modules to enhance feature processing and improve accuracy. Additionally, we introduce a trainable weight array to control the importance of each Transformer module for better generalization of the model. Experimental results demonstrate our model’s effectiveness, with an accuracy rate of 99.58% for the public dataset Birds_data, as well as 98.77% for the Birdsonund1 dataset, and 99.03% for the UrbanSound8K environment sound dataset.

HyFormer: a hybrid transformer-CNN architecture for retinal OCT image segmentation

Optical coherence tomography (OCT) has become the leading imaging technique in diagnosing and treatment planning for retinal diseases. Retinal OCT image segmentation involves extracting lesions and/or tissue structures to aid in the decisions of ophthalmologists, and multi-class segmentation is commonly needed. As the target regions often spread widely inside the retina, and the intensities and locations of different categories can be close, good segmentation networks must possess both global modeling capabilities and the ability to capture fine details. To address the challenge in capturing both global and local features simultaneously, we propose HyFormer, an efficient, lightweight, and robust hybrid network architecture. The proposed architecture features parallel Transformer and convolutional encoders for independent feature capture. A multi-scale gated attention block and a group positional embedding block are introduced within the Transformer encoder to enhance feature extraction. Feature integration is achieved in the decoder composed of the proposed three-path fusion modules. A class activation map-based cross-entropy loss function is also proposed to improve segmentation results. Evaluations are performed on a private dataset with myopic traction maculopathy lesions and the public AROI dataset for retinal layer and lesion segmentation with age-related degeneration. The results demonstrate HyFormer's superior segmentation performance and robustness compared to existing methods, showing promise for accurate and efficient OCT image segmentation. .

A Transformer-Based Approach to Leakage Detection in Water Distribution Networks.

The efficient detection of leakages in water distribution networks (WDNs) is crucial to ensuring municipal water supply safety and improving urban operations. Traditionally, machine learning methods such as Convolutional Neural Networks (CNNs) and Autoencoders (AEs) have been used for leakage detection. However, these methods heavily rely on local pressure information and often fail to capture long-term dependencies in pressure series. In this paper, we propose a transformer-based model for detecting leakages in WDNs. The transformer incorporates an attention mechanism to learn data distributions and account for correlations between historical pressure data and data from the same time on different days, thereby emphasizing long-term dependencies in pressure series. Additionally, we apply pressure data normalization across each leakage scenario and concatenate position embeddings with pressure data in the transformer model to avoid feature misleading. The performance of the proposed method is evaluated by using detection accuracy and F1-score. The experimental studies conducted on simulated pressure datasets from three different WDNs demonstrate that the transformer-based model significantly outperforms traditional CNN methods.

Does embeddedness in strategic alliances matter for innovation efficiency? The moderating roles of government R&D subsidies and standardization

PurposeThis paper investigates the impact of strategic alliance network centrality and structural holes on firm innovation efficiency. In addition, the paper aims to explore the moderating effects of government R&D subsidies and firm technology standardization.Design/methodology/approachBased upon the literature on strategic alliance networks, this paper proposes a conceptual model with several hypotheses. The empirical analysis is based on a sample of 736 observations from 92 mechanical manufacturing firms in China from 2010 to 2017. We measured firm innovation efficiency via the DEA model and performed quantitative analysis with GMM estimation.FindingsThe results indicate that strategic alliance network centrality is positively related to firm innovation efficiency, and structural holes have a U-shaped relationship with firm innovation efficiency. Government R&D subsidies positively moderate the relationship between centrality and firm innovation efficiency. Firm technology standardization positively moderates the relationship between centrality and firm innovation efficiency and the U-shaped relationship between structural holes and firm innovation efficiency.Practical implicationsFirms should focus on improving innovation efficiency and maximizing innovation output under limited resources. Furthermore, managers ought to strengthen cooperation between firms and external alliances while promoting the utilization of strategic alliance network position resources to benefit innovation efficiency.Originality/valueThis paper considers that innovation efficiency, including input and output processes, is more representative than innovation performance, and few studies have focused on the relationship between strategic alliance networks and innovation efficiency. To fill this research gap, this paper explores the impact of strategic alliance network position embedding on innovation efficiency.

MAFormer: A cross-channel spatio-temporal feature aggregation method for human action recognition

Human action recognition has been widely used in fields such as human–computer interaction and virtual reality. Despite significant progress, existing approaches still struggle with effectively integrating hierarchical information and processing data beyond a certain frame count. To address these challenges, we introduce the Multi-AxisFormer (MAFormer) model, which is organized in terms of spatial, temporal, and channel dimensions of the action sequence, thereby enhancing the model’s understanding of correlations and intricate structures among and within features. Drawing on the Transformer architecture, we propose the Cross-channel Spatio-temporal Aggregation (CSA) structure for more refined feature extraction and the Multi-Axis Attention (MAA) module for more comprehensive feature aggregation. Moreover, the integration of Rotary Position Embedding (RoPE) boosts the model’s extrapolation and generalization abilities. MAFormer surpasses the known state-of-the-art on multiple skeleton-based action recognition benchmarks with the accuracy of 93.2% on NTU RGB+D 60 cross-subject split, 89.9% on NTU RGB+D 120 cross-subject split, and 97.2% on N-UCLA, offering a novel paradigm for hierarchical modeling in human action recognition.

Deep Location Soft-Embedding-Based Network With Regional Scoring for Mammogram Classification.

Early detection and treatment of breast cancer can significantly reduce patient mortality, and mammogram is an effective method for early screening. Computer-aided diagnosis (CAD) of mammography based on deep learning can assist radiologists in making more objective and accurate judgments. However, existing methods often depend on datasets with manual segmentation annotations. In addition, due to the large image sizes and small lesion proportions, many methods that do not use region of interest (ROI) mostly rely on multi-scale and multi-feature fusion models. These shortcomings increase the labor, money, and computational overhead of applying the model. Therefore, a deep location soft-embedding-based network with regional scoring (DLSEN-RS) is proposed. DLSEN-RS is an end-to-end mammography image classification method containing only one feature extractor and relies on positional embedding (PE) and aggregation pooling (AP) modules to locate lesion areas without bounding boxes, transfer learning, or multi-stage training. In particular, the introduced PE and AP modules exhibit versatility across various CNN models and improve the model's tumor localization and diagnostic accuracy for mammography images. Experiments are conducted on published INbreast and CBIS-DDSM datasets, and compared to previous state-of-the-art mammographic image classification methods, DLSEN-RS performed satisfactorily.

SANAS-Net: spatial attention neural architecture search for breast cancer detection

<p>The utilization of mammography images plays a vital role in the prompt detection and treatment of breast cancer. Breast imaging techniques aid medical professionals in assessing the dimensions, morphology, and spatial orientation of breast lesions, facilitating the differentiation between benign and malignant conditions. Breast tissue can vary widely in terms of density, composition, and structure, leading to complexities in distinguishing between benign and malignant conditions. The primary contribution of this paper is the proposal of a spatial attention-based neural architecture search network (SANAS-Net) technique that incorporates a spatial attention mechanism, enabling the model to learn and prioritize key regions within mammograms (MMs). Multi-head attention is employed within the transformer blocks to effectively capture a wide range of spatial relations and feature interactions. Global contextual information was integrated into the transformer blocks by means of introducing positional embeddings. Several practical studies have been undertaken to verify the effectiveness of our methodology in identifying fully attentive networks that exhibit good performance in distinguishing between malignant and benign breast cancer cases. The experimental study reached a test accuracy of 89.95%, which is way higher than previously proposed algorithms for mammography imagebased breast cancer detection.</p>

Transformer-based Named Entity Recognition for Clinical Cancer Drug Toxicity by Positive-unlabeled Learning and KL Regularizers

Background: With increasing rates of polypharmacy, the vigilant surveillance of clinical drug toxicity has emerged as an important With increasing rates of polypharmacy, the vigilant surveillance of clinical drug toxicity has emerged as an important concern. Named Entity Recognition (NER) stands as an indispensable undertaking, essential for the extraction of valuable insights regarding drug safety from the biomedical literature. In recent years, significant advancements have been achieved in the deep learning models on NER tasks. Nonetheless, the effectiveness of these NER techniques relies on the availability of substantial volumes of annotated data, which is labor-intensive and inefficient. Methods: This study introduces a novel approach that diverges from the conventional reliance on manually annotated data. It employs a transformer-based technique known as Positive-Unlabeled Learning (PULearning), which incorporates adaptive learning and is applied to the clinical cancer drug toxicity corpus. To improve the precision of prediction, we employ relative position embeddings within the transformer encoder. Additionally, we formulate a composite loss function that integrates two Kullback-Leibler (KL) regularizers to align with PULearning assumptions. The outcomes demonstrate that our approach attains the targeted performance for NER tasks, solely relying on unlabeled data and named entity dictionaries. Conclusion: Our model achieves an overall NER performance with an F1 of 0.819. Specifically, it attains F1 of 0.841, 0.801 and 0.815 for DRUG, CANCER, and TOXI entities, respectively. A comprehensive analysis of the results validates the effectiveness of our approach in comparison to existing PULearning methods on biomedical NER tasks. Additionally, a visualization of the associations among three identified entities is provided, offering a valuable reference for querying their interrelationships.

EGST: Enhanced Geometric Structure Transformer for Point Cloud Registration.

We explore the effect of geometric structure descriptors on extracting reliable correspondences and obtaining accurate registration for point cloud registration. The point cloud registration task involves the estimation of rigid transformation motion in unorganized point cloud, hence it is crucial to capture the contextual features of the geometric structure in point cloud. Recent coordinates-only methods ignore numerous geometric information in the point cloud which weaken ability to express the global context. We propose Enhanced Geometric Structure Transformer to learn enhanced contextual features of the geometric structure in point cloud and model the structure consistency between point clouds for extracting reliable correspondences, which encodes three explicit enhanced geometric structures and provides significant cues for point cloud registration. More importantly, we report empirical results that Enhanced Geometric Structure Transformer can learn meaningful geometric structure features using none of the following: (i) explicit positional embeddings, (ii) additional feature exchange module such as cross-attention, which can simplify network structure compared with plain Transformer. Extensive experiments on the synthetic dataset and real-world datasets illustrate that our method can achieve competitive results.

A Dynamic Position Embedding-Based Model for Student Classroom Complete Meta-Action Recognition.

The precise recognition of entire classroom meta-actions is a crucial challenge for the tailored adaptive interpretation of student behavior, given the intricacy of these actions. This paper proposes a Dynamic Position Embedding-based Model for Student Classroom Complete Meta-Action Recognition (DPE-SAR) based on the Video Swin Transformer. The model utilizes a dynamic positional embedding technique to perform conditional positional encoding. Additionally, it incorporates a deep convolutional network to improve the parsing ability of the spatial structure of meta-actions. The full attention mechanism of ViT3D is used to extract the potential spatial features of actions and capture the global spatial-temporal information of meta-actions. The proposed model exhibits exceptional performance compared to baseline models in action recognition as observed in evaluations on public datasets and smart classroom meta-action recognition datasets. The experimental results confirm the superiority of the model in meta-action recognition.

Spatial guided image captioning: Guiding attention with object's spatial interaction

AbstractNowadays relational position embedding is widely used in many large multi‐modal models. It begins with relational captioning (a branch of image captioning) and contains two procedures: geometric modelling and prior attention. However, there are some problems that remain unsolved in the conventional procedures. This paper reviews the shortcomings of geometric modelling and prior attention. Then, a new framework called relational guided transformer (RGT) is proposed to verify the authors' conclusion from the origin of relational position embedding—relational captioning. Specifically, RGT has two simple but effective improvements in geometric modelling and prior attention: (1) A machine‐learned geometric modelling strategy called multi‐task geometric modelling (MTG) is used under multi‐task learning, replacing the original hand‐made geometric feature. (2) The effectiveness of multiple kinds of prior attention is discussed and preserved in a better form, which is called spatial guided attention (SGA) to integrate the geometric prior knowledge. Extensive experiments on MSCOCO and Flickr30k have been performed to investigate the effectiveness of each module and prove our argument. The superiority of the model comparing to the authors' baseline has also been proven on the offline evaluation with the “Karpathy” test split of both datasets.

Unveiling Structure-Enhanced Positional Embedding Module in Transformer-Based 3D Human Pose Estimation

Abstract With the advancement of computational capabilities, the realm of deep learning has expanded from 2D image tasks to encompass 3D video tasks. Human action recognition stands as a typical 3D video task, which emphasizes the accurately capture of local body structures. However, existing methods encounter limitations in precisely capturing these structures, especially in joints within the same body region that exhibits diverse static and dynamic features. To address this complexity, our study innovatively introduces a Structure-Enhanced Positional Embedding Module seamlessly integrated into the PoseFormer model, resulting in the innovative HStruPE-Former model. Experimental validation on the Human3.6M dataset demonstrates a significant performance improvement achieved by the HStruPE-Former model in the task of 3D human pose estimation. Particularly in scenarios involving intricate movements and postures, the module demonstrates improved stability and accuracy. Although the improvements are modest, this technology is expected to facilitate the exploration of untapped potential of structure-enhanced postion embedding modules in diverse neural network applications, thereby promoting further exploration in neural network research. This research outcome not only enhances the performance in current 3D video tasks but also paves the way for novel research directions in the field of deep learning.

Rethinking encoder-decoder architecture using vision transformer for colorectal polyp and surgical instruments segmentation

Accurate polyp segmentation from colonoscopy images is important for the immediate diagnosis and effective treatment of colon cancer. While significant progress has been made in the polyps segmentation task, there are various challenges that need to be addressed. Polyps can vary greatly in size and shape, and often has no clear boundary between surrounding tissues. Furthermore, surgical instrument segmentation can aid surgeons with the precise positioning and orientation of the instruments, helping them to plan the next steps in the robot-assisted surgery. The proposed colorectal polyp segmentation Transformer (CPS-Former) uses innovative attention blocks in a network that encodes-decodes features like classic Semantic Segmentation Network (SegNet). However, it has special self-attention modules with small convolutional kernels that efficiently extract information from different feature-channels. Moreover, it is equipped with an effective positional embedding to capture information from a large area of context for long distance interactions. Additionally, a fusion block is embedded for scaling-attention that combines the outputs from the encoder-decoder blocks to enhance the semantic features and reduce the non-semantic ones. Transformer encoder blocks also modified by adding a local feedforward layer and skips connections, and adjust the channel sizes to reduce the model trainable parameters. We evaluate our colorectal polyp segmentation network (CPS-Former) on four colorectal polyp public datasets and one surgical instrument segmentation dataset, which show its superiority over other state-of-the-art polyp segmentation models. Our implementation source code and network weights are available at GitHub: https://github.com/ahmedeqbal/CPS-Former.

Exploring global context and position-aware representation for group activity recognition

This paper explores the context and position information in the scene for group activity understanding. Firstly, previous group activity recognition methods strive to reason on individual features without considering the information in the scene. Besides correlations among actors, we argue that integrating the scene context simultaneously can afford us more useful and supplementary cues. Therefore, we propose a new network, termed Contextual Transformer Network (CTN), to incorporate global contextual information into individual representations. In addition, the position of individuals also plays a vital role in group activity understanding. Unlike previous methods that explore correlations among individuals semantically, we propose Clustered Position Embedding (CPE) to integrate the spatial structure of actors and produce position-aware representations. Experimental results on two widely used datasets for sports video and social activity (i.e., Volleyball and Collective Activity datasets) show that the proposed method outperforms state-of-the-art approaches. Especially, when using ResNet-18 as the backbone, our method achieves 93.6/93.9% MCA/MPCA on the Volleyball dataset and 95.4/96.3% MCA/MPCA on the Collective Activity dataset.

Bi-directional context-aware network for the nested named entity recognition

The Span-based model can effectively capture the complex entity structure in the text, thus becoming the mainstream model for nested named entity recognition (Nested NER) tasks. However, traditional Span-based models decode each entity span independently. They do not consider the semantic connections between spans or the entities’ positional information, which limits their performance. To address these issues, we propose a Bi-Directional Context-Aware Network (Bi-DCAN) for the Nested NER. Specifically, we first design a new span-level semantic relation model. Then, the Bi-DCAN is implemented to capture this semantic relationship. Furthermore, we incorporate Rotary Position Embedding into the bi-affine mechanism to capture the relative positional information between the head and tail tokens, enabling the model to more accurately determine the position of each entity. Experimental results show that compared to the latest model Diffusion-NER, our model reduces 20M parameters and increases the F1 scores by 0.24 and 0.09 on the ACE2005 and GENIA datasets respectively, which proves that our model has an excellent ability to recognise nested entities.

Continual few-shot patch-based learning for anime-style colorization

The automatic colorization of anime line drawings is a challenging problem in production pipelines. Recent advances in deep neural networks have addressed this problem; however, collectingmany images of colorization targets in novel anime work before the colorization process starts leads to chicken-and-egg problems and has become an obstacle to using them in production pipelines. To overcome this obstacle, we propose a new patch-based learning method for few-shot anime-style colorization. The learning method adopts an efficient patch sampling technique with position embedding according to the characteristics of anime line drawings. We also present a continuous learning strategy that continuously updates our colorization model using new samples colorized by human artists. The advantage of our method is that it can learn our colorization model from scratch or pre-trained weights using only a few pre- and post-colorized line drawings that are created by artists in their usual colorization work. Therefore, our method can be easily incorporated within existing production pipelines. We quantitatively demonstrate that our colorizationmethod outperforms state-of-the-art methods.

Position-aware Interactive Attention Network for multi-intent spoken language understanding

Spoken Language Understanding (SLU) is a crucial component of task-oriented dialog systems. In recent years, there has been increasing attention on multi-intent SLU due to its relevance to complex real-world applications. Most existing joint models only utilize multi-intent information to guide the slot filling, with only a small number of models achieving bidirectional interaction. Additionally, unlike traditional single-intent SLU, multi-intent SLU is scope-dependent, where each intent in a sentence has its specific dependency range. However, current bidirectional joint models often employ a pipeline approach to implement interaction between the two sub-tasks, which fails to fully leverage the semantic information between them and can lead to error propagation. Moreover, attention-based multi-intent joint models do not adequately model the positional relationships between words, resulting in suboptimal overall performance. In this paper, we propose a novel multi-intent SLU model called Position-aware Interactive Attention Network (PIAN), which consists of interactive attention and rotary position embedding. The aim of PIAN is to fully exploit the correlation between the two sub-tasks and capturing the positional dependencies between words. This facilitates mutual guidance and enhancement between the two sub-tasks. Additionally, to address the issue of uncoordinated slot problem generated by traditional slot filling decoders, we employ an MCRF slot filling decoder to constrain the slot labels. We evaluate our model on two public multi-intent SLU datasets, and the experimental results demonstrate that our model achieves state-of-the-art performance on key metrics. Code for this paper is publicly available at https://github.com/puhahahahahaha/SLU_with_Co_PRoE.

Transformer with Selective Shuffled Position Embedding and key-patch exchange strategy for early detection of Knee Osteoarthritis

Knee OsteoArthritis (KOA) is a widespread musculoskeletal disorder that can severely impact the mobility of older individuals. Insufficient medical data presents a significant obstacle for effectively training models due to the high cost associated with data labeling. Currently, deep learning-based models extensively utilize data augmentation techniques to improve their generalization ability and alleviate overfitting. However, conventional data augmentation techniques are primarily based on the original data and fail to introduce substantial diversity to the dataset. In our study, we introduce a unique method that utilizes the Vision Transformer (ViT) model, enhanced by our newly developed Selective Shuffled Position Embedding (SSPE) and key-patch exchange techniques. These innovations generate varied input sequences, serving as a data augmentation strategy for improving KOA classification. More specifically, we fix and shuffle the position embeddings of key patches and non-key patches, respectively. Then, for the target image, we randomly select other candidate images from the training set to exchange their key patches and thus obtain different input sequences. Finally, a hybrid loss function is developed by incorporating multiple loss functions for different types of the sequences. According to the experimental results, the generated data are considered valid as they lead to a notable improvement in the model’s classification performance.