Position Encoder Research Articles

A Letter is a Letter and its Co-Occurrences: Cracking the Emergence of Position-Invariance Processing

Visual word recognition requires encoding letter identities and positions (orthographic processing). The present study focuses on the emergence of the mechanism responsible for encoding letter order in a word: position invariance. Reading experience leads to developing a flexible mechanism that encodes the information of the position of letters, explaining why jugde and judge are easily confused. Critically, orthographic regularities (e.g., frequent letter co-occurrences) modulate letter position encoding: the pseudoword mohter is extremely similar to mother because, in middle positions, the bigram TH is much more frequent than HT. Here, we tested whether position invariance emerges rapidly after the exposition to orthographic regularities—bigrams—in a novel script. To that end, we designed a study with two phases. In Phase 1, following Chetail (2017; Experiment 1b, Cognition, 163, 103–120), individuals were first exposed to a flow of artificial words for a few minutes, with four bigrams occurring frequently. Afterward, participants judged the strings with trained bigrams as more wordlike (i.e., readers quickly picked up subtle new orthographic regularities) than the strings with untrained bigrams, replicating Chetail (2017). In Phase 2, participants performed a same–different matching task in which they had to decide whether pairs of five-letter strings were the same or not. The critical comparison was between pairs with a transposition of letters in a frequent (trained) versus infrequent (untrained) bigram. Results showed that participants were more prone to make errors with frequent bigrams than with infrequent bigrams with a letter transposition. These findings reveal that position invariance emerges rapidly, after continuous exposure to orthographic regularities.

Load forecasting of district heating system based on improved FB-Prophet model

Accurate load forecasting of the district heating network (DHN) is essential to guarantee effective energy production, distribution, and rational utilization. An improved Facebook-Prophet (FB-Prophet) model with additional positional encoding layers has been developed to forecast the DHN heat consumption. The accuracy of univariate and multivariate FB-Prophet models is evaluated; this paper also evaluates the optimum training dataset length. To explore the performance of the improved FB-Prophet model in heating load forecasting tasks, another seven machine learning models, namely FB-Prophet, DeepVAR, long-short term memory, extreme gradient boosting, multilayer perceptron, recurrent neural network, and support vector regression are used for comparison. The historical heating load, outside temperature, relative humidity, speed of wind, direction of wind, and weather type of a DHN in Serbia are utilized to extensively investigate the effectiveness of the improved FB-Prophet model. The prediction outcomes of all the models are thoroughly analyzed. The results indicate that the improved FB-Prophet model can generate the most precise and consistent predictions and it showed better results for sparse DHN data. The prediction curve is fitted to the trend of hourly DHN consumption change, which can play an effective guiding function in the distribution of heat.

Improving Arabic Named Entity Recognition with a Modified Transformer Encoder

This article investigates the use of a transformer encoder for Arabic Named Entity Recognition (NER). The classic transformer that was originally proposed for machine translation adopts the absolute sinusoidal position embedding which is aware of distance but unfortunately is not aware of the directionality. However, in the NER task, both distance and orientation are crucial. Therefore, in this study, instead of using absolute sinusoidal position encoding, we employ relative positional encoding and incorporate the directionality information in our NER model. More specifically, our proposed model uses Bidirectional Long Short-Term Memory (BiLSTM) for encoding every input token. Then, the output of the encoder is fed to the multi-head attention where both the distance and directionality information are incorporated. The decoder layer with a simple fully connected layer takes as input, the result of the attention layer, and the prediction layer with Conditional Random Fields (CRF) predicts the tag of each token. We validate our proposed approach on two merged public datasets, namely, ANER corp and AQMAR. Our experiment results demonstrate significant improvements when compare to the vanilla Transformer with absolute sinusoidal position encoding while achieving a state-of-the-art result on a merged two Arabic public datasets.

Multi-Planar Jump Performance in Speed Skating Athletes: Investigating Interlimb Differences in an Asymmetrical Sport

Elite speed skaters are exposed to asymmetric lower limb loading consequent to the unidirectional turns inherent to the sport. This presents a unique model to study the effects of sport-specific loading on interlimb differences in mechanical muscle function. This study, therefore, examined baseline interlimb asymmetries in multi-directional jump tests in elite speed skaters using a cross-sectional design. Thereafter, participants were monitored longitudinally using the bilateral countermovement jump (CMJ) to quantify interlimb differences in mechanical muscle function throughout a competitive season. Pre-season baseline testing included a single leg lateral jump (JumpLat) and a single leg forward horizontal jump (JumpHorz) attached to a robotic linear position encoder, along with a bilateral CMJ on a dual force plate system. From baseline, CMJ monitoring was conducted throughout the 24-week competitive season. Within-limb changes in right vs left CMJ concentric impulse (CMJCon) and eccentric deceleration impulse (CMJEcc) were assessed using a linear mixed effects model. No systematic interlimb differences were found at baseline (p = 0.33–0.98) and the between-test agreement in limb dominance was poor (Kappa = −0.17–0.33). Furthermore, there were no time effects observed for interlimb differences in CMJCon (fixed effect = 0.01 N*s) and a small decrease in CMJEcc (fixed effects = −0.35 N*s, p = 0.01). These data suggest that even in a sport with asymmetrical loading, interlimb differences in mechanical output remain stable at the group level. However, changes occurring at the individual athlete level may be occurring that are meaningful for performance and injury.

Exploring the potential of time-series transformers for process modeling and control in chemical systems: An inevitable paradigm shift?

The last two years have seen groundbreaking advances in natural language processing (NLP) with the advent of applications like ChatGPT, Codex, and ChatSonic. This revolution is powered by the development of cutting-edge transformer models that leverage multiheaded attention mechanisms, positional encoding, and highly efficient transfer learning capabilities. Despite these remarkable advances, there is still work to be done to fully realize the practical applicability of transformers in chemical systems. Thus, we are excited to present our latest work, which highlights the immense potential of transformers for non-trivial multivariate time-series prediction tasks with high-value implications in process monitoring, control, and optimization. Specifically, impressive prediction capabilities of first-generation time-series transformers (TSTs) were demonstrated by developing, testing, and comparing TSTs with existing models. Further, the practical applicability of TSTs was highlighted by developing a first-of-a-kind TST-based model predictive controller (MPC). More importantly, the current work provides a concrete foundation for exploring promising new directions, such as the development of large-scale TSTs leveraging transfer learning for modeling of new process equipment, and plant-level multisource aggregative cognitive models for fault prognosis and prevention. We are excited to see what the future holds as we continue to push the boundaries of what is possible with these ‘transformer-tive’ technologies.

Human activity recognition with self-attention

In this paper, a self-attention based neural network architecture to address human activity recognition is proposed. The dataset used was collected using smartphone. The contribution of this paper is using a multi-layer multi-head self-attention neural network architecture for human activity recognition and compared to two strong baseline architectures, which are convolutional neural network (CNN) and long-short term network (LSTM). The dropout rate, positional encoding and scaling factor are also been investigated to find the best model. The results show that proposed model achieves a test accuracy of 91.75%, which is a comparable result when compared to both the baseline models.

Modeling Spatial Trajectories Using Coarse-Grained Smartphone Logs

Every user carries their smartphones wherever they go – a crucial aspect ignored by the current models for spatial recommendations. In detail, the current approaches learn the points-of-interest (POI) preferences of a user via the standard spatial features, i.e. the POI coordinates and the social network, and thus ignore the features related to the smartphone usage of a user. Moreover, with growing privacy concerns, users refrain from sharing their exact geographical coordinates as well as their social media activity. In this paper, we present <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ReVAMP</small> , a sequential POI recommendation approach that uses smartphone app-usage logs to identify the mobility preferences of a user. Our work aligns with the recent psychological studies of online urban users which show that their spatial mobility behavior is largely influenced by the activity of their smartphone apps. Specifically, our proposal of coarse-grained data refers to data logs collected in a privacy-conscious manner consisting only of (a) category of smartphone app-used and (b) category of check-in location. Thus, <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ReVAMP</small> is not privy to precise geo-coordinates, social networks, and the specific app being used. Buoyed by the efficacy of self-attention models, <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ReVAMP</small> learns the POI preferences of a user using two forms of positional encodings – absolute and relative – with each extracted from the inter-check-in dynamics in the check-in sequence of a user. Extensive experiments across two large-scale datasets from China show that <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">ReVAMP</small> outperforms the state-of-the-art sequential POI recommendation approaches and can be extended to app- and POI-category prediction.

군 등변 컨벌루션을 사용하는 효율적인 군 등변 트랜스포머

Self-attention is a permutation-equivariant operation by definition, and incorporating group-invariant positional encoding allows it to achieve equivariance for larger symmetry groups. We propose efficient group-equivariant transformers that eliminate the cubic complexity caused by group-invariant positional encoding. Instead of using explicit positional encoding, we replace linear projections with group-equivariant convolutional projections, which serve as both a group mixer and an implicit positional encoding. This allows for reducing the group of self-attention to the translation group while maintaining group equivariance, resulting in less computation and memory. A group-equivariant convolutional stem improves performance much more. The proposed method outperforms the existing group-equivariant transformer and CNNs on the rotated MNIST dataset, a standard dataset for evaluating group-equivariant networks.

TransAM: Transformer appending matcher for few-shot knowledge graph completion

Few-shot knowledge graph completion (FSKGC) refers to predicting new facts for a new relation with only few-shot observed entity pairs (triples) as support set. Existing solutions to FSKGC mainly conduct the matching process over entity pair representations. Although effective, a major concern of these models is that the entity interactions are not fully explored, based on the observation that they usually generate the pair representation before the matching stage. Such a design inherently overlooks the fine-grained information from entity interactions, leading to performance decrements in one or three shot, which require matching models to capture more sufficient semantic meanings for prediction. To remedy this issue, in this paper, we explore the entity interactions within and between different instances, i.e., the co-occurrence of two entities, for FSKGC and propose our model named TransAM, Transformer Appending Matcher. TransAM solves the FSKGC problem by computing the probability of entity sequence with a well-designed transformer matching network. Specifically, TransAM appends query entity pair to serialized reference entity sequence and utilizes transformer to calculate the probability by capturing intra- and inter- triple entity interactions. To bridge the gap between transformer and the triple structure, TransAM introduces rotary operation to preserve the head and tail roles of entity within the triple and distinguishes different triples by a separated triple position encoding. Empirical studies on two public benchmark datasets NELL-One and Wiki-One show that TransAM outperforms existing metric-learning solutions in MRR and Hits@1 with both one- and three- shot settings, and achieves comparable results on five-shot setting. Datasets and code will be public available at https://github.com/gawainx/TransAM.

LabanFormer: Multi-scale graph attention network and transformer with gated recurrent positional encoding for labanotation generation

Labanotation is a widely-used notation system for recording human dance movements. Automatically generating Labanotation scores from motion capture data can save significant manual effort and help the preservation of old folk dances in protecting intangible cultural heritages. Existing Labanotation generation methods have limited ability to capture the flexible limb movements as well as the rich periodic, symmetric, or repeated dance steps. In this paper, we present a novel LabanFormer model including a Multi-Scale Graph Attention network (MS-GAT) and a transformer model with Gated Recurrent Positional Encoding (GRPE) to achieve more effective Labanotation generation. First, the proposed MS-GAT can capture flexible limb movements by learning feature correlations between every two joints and aggregating features of neighboring joints over multiple scales. Second, we propose a new GRPE-based transformer to learn global temporal dependencies in the output feature sequences of MS-GAT. The novel GRPE module can encode position information with learnable parameters while handling various sequence lengths. As such, the periodic, symmetric, or repeated steps in dances can be accurately captured. Finally, the corresponding Laban symbols are generated by the decoder of the GRPE-based transformer. Extensive experiments on two real-world datasets show that the proposed LabanFormer model obtains remarkable performance compared with state-of-the-art approaches on the automatic Labanotation generation task.

Enhanced multihead self-attention block network for remote sensing image scene classification

Remote sensing image scene classification has been widely researched with the aim of assigning semantics labels to the land cover. Although convolutional neural networks (CNN), such as VggNet and ResNet, have achieved good performance, the complex background and redundant information of remote sensing images restrict the improvement of final accuracy. We propose an enhanced multihead self-attention block network, which effectively reduces the adverse impact of background and emphasize the main information. In this model, due to the possible redundancy of high-level information of CNN, we only replace the final three bottleneck blocks of ResNet50 with the enhanced multihead self-attention layer to focus on the salient region of each image more effectively. Our enhanced multihead self-attention layer provides the following improvements over the classical module. First, we construct a triple-way convolution to deal with the arbitrary directionality of remote sensing images and get more stable attention information. Then, the improved relative position encodings are used to consider the relative distance between different location features. Finally, we use depthwise convolution and InstanceNorm operation to restore the diversity ability of multiheads. The contrast and ablation experiments carried out on three public datasets show our approach improves upon the baseline significantly and achieves remarkable performance compared with some state-of-the-art methods.

Multi-task Label-wise Transformer for Chinese Named Entity Recognition

Benefiting from the improvement of positional encoding and the introduction of lexical knowledge, Transformer has achieved superior performance than the prevailing BiLSTM-based models in named entity recognition (NER) task. However, existing Transformer-based models for Chinese NER pay less attention to the information captured by the bottom layers of Transformer and the significance of representation subspace where each head of Transformer is projected. In this article, we propose M ulti- T ask L abel- W ise T ransformer (MTLWT). From a global perspective, we assign entity boundary prediction (EBP) and entity type prediction (ETP) tasks to the first two layers. In this way, we stimulate lower layers to participate more in constructing character representation. Besides, in each multi-head self-attention (MHSA) layer, we provide a specific focus for each individual head, making the head project into a significant subspace. Experiments on four datasets from different domains show that our proposed model achieves comparable performance with other state-of-the-art models. In particular, MTLWT outperforms the other frameworks without external knowledge on all the datasets.

Novel architecture for gated recurrent unit autoencoder trained on time series from electronic health records enables detection of ICU patient subgroups

Electronic health records (EHRs) are used in hospitals to store diagnoses, clinician notes, examinations, lab results, and interventions for each patient. Grouping patients into distinct subsets, for example, via clustering, may enable the discovery of unknown disease patterns or comorbidities, which could eventually lead to better treatment through personalized medicine. Patient data derived from EHRs is heterogeneous and temporally irregular. Therefore, traditional machine learning methods like PCA are ill-suited for analysis of EHR-derived patient data. We propose to address these issues with a new methodology based on training a gated recurrent unit (GRU) autoencoder directly on health record data. Our method learns a low-dimensional feature space by training on patient data time series, where the time of each data point is expressed explicitly. We use positional encodings for time, allowing our model to better handle the temporal irregularity of the data. We apply our method to data from the Medical Information Mart for Intensive Care (MIMIC-III). Using our data-derived feature space, we can cluster patients into groups representing major classes of disease patterns. Additionally, we show that our feature space exhibits a rich substructure at multiple scales.

PUnet: A robust framework for phase unwrapping in interferometric SAR

Synthetic Aperture Radar Interferometry (InSAR) has grown significantly over the past few decades, which were mainly used in remote sensing applications. Most InSAR applications (e.g., terrain mapping and monitoring) utilized a key technique called phase unwrapping Phase unwrapping obtained the absolute phase from the wrapped phase for the subsequent application. However, the collected wrapped phase inevitably contains noise due to the influence of factors such as atmosphere and temperature in the InSAR acquisition stage. This noise made it challenging to obtain the absolute phase from the wrapped phase. This study proposed a deep learning framework (PUnet) for phase unwrapping form InSAR data. pUnet was a robust framework using U-net as the basic structure combined with an attention mechanism and positional encoding, facilitating accurate phase unwrapping from the wrapped phase. Through comparative experiments with typical phase unwrapping algorithms, we demonstrated that pUnet could obtain absolute phases with high accuracy and robustness than from the wrapped phase under various levels of noise.

Predictive Business Process Monitoring Approach Based on Hierarchical Transformer

Predictive business process monitoring is predicting the next stage of the business process based on the sequence of events that have occurred in the business process instance, which is positive for promoting the rational allocation of resources and the improvement of execution efficiency. There are drawbacks in modeling business process instances, such as conceptual drift phenomenon and long event sequences. Therefore, we propose a hierarchical Transformer-based business process prediction model to improve the performance of the Transformer-based predictive business process monitoring model. We encode the event features using two different encoding methods to obtain the relationship between activities and attributes. A drift detection algorithm is proposed to segment the business process and calculate the correlation between activities and segments by using cross-attention. Furthermore, learnable position encoding is designed to capture the relative position information of subsequences. Finally, the information of different granularity, such as event attributes, event subsequences, and complete instances, is fused by different weights. Experiments were run on seven real event logs for the next activity prediction and remaining time prediction, and the next activity prediction accuracy improved by 6.32% on average, and the mean absolute error of remaining time prediction reduces by 21% on average.

MM-ConvBERT-LMS: Detecting Malicious Web Pages via Multi-Modal Learning and Pre-Trained Model

In recent years, the number of malicious web pages has increased dramatically, posing a great challenge to network security. While current machine learning-based detection methods have emerged as a promising alternative to traditional detection techniques. However, these methods are commonly based on single-modal features or simple stacking of classifiers built on various features. As a result, these techniques are not capable of effectively fusing features from different modalities, ultimately limiting the detection effectiveness. To address this limitation, we propose a malicious web page detection method based on multi-modal learning and pre-trained models. First, in the input stage, the raw URL and HTML tag sequences of web pages are used as input features. To help the subsequent model learn the relationship between the two modalities and avoid information confusion, modal-type encoding, and positional encoding are introduced. Next, a single-stream neural network based on the ConvBERT pre-trained model is used as the backbone classifier, and it learns the representation of multi-modal features through fine-tuning. For the output part of the model, a linear layer based on large margin softmax is applied to the decision-making. This activation function effectively increases the classification boundary and improves the robustness. In addition, a coarse-grained modal matching loss is added to the model optimization objective to assist the models in learning the cross-modal association features. Experimental results on synthetic datasets show that our proposed method outperforms traditional single-modal detection methods in general, and has advantages over baseline models in terms of accuracy and reliability.

Speed Measurement for Incremental Position Encoder Using Period-Based Method With Sinc3 Filtering

Precise measurement of the rotor speed is mandatory for machine tool spindle drives and other drives having the same demands for the accuracy of speed stabilization. These drives commonly use incremental position encoders for speed measurement as a derivative of the position in time. Currently, the best practice is the implementation of constant elapse time methods, the accuracy of which is limited by the accuracy of the position encoder itself and the discreteness of time. However, it was noted that the constant elapse time method ignores all the data in-between the measurements. An analysis of the periods between pulses from the encoder led to the discovery that these periods have the properties of delta–sigma modulated signals. Thus, it was suggested to process the results of the period-based method with a sinc filter of a high order to obtain extra information from the incoming data. The proposed period-based method with sinc filtering (PBSF) uses third-order filter as the most commonly used type in the measuring equipment. The simulation model was developed to show the error distribution for competing methods, and the experimental results were obtained in the steady state and during acceleration of the drive. The comparison of the methods was performed with identical data at the input and the same measurement time. Both the simulation and experimental results showed that at high speeds the proposed period-based method with sinc3 filtering (PBS3F) provides a much smaller deviation from the actual value than the conventional constant elapsed time (CET) methods. The computational complexity was analyzed as well showing the possibility of method implementation using modern microcontrollers.

Design and Analysis of Wireless Resolver for Wireless Switched Reluctance Motors

In this article, a wireless resolver is proposed and implemented for wireless switched reluctance (SR) motors, which can detect the rotor position and transfer the information to the primary side without involving any battery storage or active components at the secondary side. The key is to integrate wireless power transfer (WPT) into a variable reluctance resolver to detect and transmit the rotor position. Compared with previous wireless SR motors adopting the Hall effect position encoder and Bluetooth communication, the robustness is highly improved, and the communication delay caused by the connection interval can be solved. Therefore, the proposed system takes the definite advantages of better dynamic performance and sealability. Besides, there is no need to specially design the SR motor, as all existing SR motors and other similar kinds of motors can readily employ this wireless resolver. An experimental prototype has been built for verification, which works well from 100 to 900 rpm with 20 mm WPT distance, and the additional power for position detection can be limited to 1.05 W while the motor consumes 54 W.

SP-GNN: Learning structure and position information from graphs

Graph neural network (GNN) is a powerful model for learning from graph data. However, existing GNNs may have limited expressive power, especially in terms of capturing adequate structural and positional information of input graphs. Structure properties and node position information are unique to graph-structured data, but few GNNs are capable of capturing them. This paper proposes Structure- and Position-aware Graph Neural Networks (SP-GNN), a new class of GNNs offering generic and expressive power of graph data. SP-GNN enhances the expressive power of GNN architectures by incorporating a near-isometric proximity-aware position encoder and a scalable structure encoder. Further, given a GNN learning task, SP-GNN can be used to analyze positional and structural awareness of GNN tasks using the corresponding embeddings computed by the encoders. The awareness scores can guide fusion strategies of the extracted positional and structural information with raw features for better performance of GNNs on downstream tasks. We conduct extensive experiments using SP-GNN on various graph datasets and observe significant improvement in classification over existing GNN models.

LGViT: A Local and Global Vision Transformer with Dynamic Contextual Position Bias Using Overlapping Windows

Vision Transformers (ViTs) have shown their superiority in various visual tasks for the capability of self-attention mechanisms to model long-range dependencies. Some recent works try to reduce the high cost of vision transformers by limiting the self-attention module in a local window. As a price, the adopted window-based self-attention also reduces the ability to capture the long-range dependencies compared with the original self-attention in transformers. In this paper, we propose a Local and Global Vision Transformer (LGViT) that incorporates overlapping windows and multi-scale dilated pooling to robust the self-attention locally and globally. Our proposed self-attention mechanism is composed of a local self-attention module (LSA) and a global self-attention module (GSA), which are performed on overlapping windows partitioned from the input image. In LSA, the key and value sets are expanded by the surroundings of windows to increase the receptive field. For GSA, the key and value sets are expanded by multi-scale dilated pooling to promote global interactions. Moreover, a dynamic contextual positional encoding module is exploited to add positional information more efficiently and flexibly. We conduct extensive experiments on various visual tasks and the experimental results strongly demonstrate the outperformance of our proposed LGViT to state-of-the-art approaches.