Embedding Vectors Research Articles

Comprehensive Assessment of BERT-Based Methods for Predicting Antimicrobial Peptides.

In recent years, the prediction of antimicrobial peptides (AMPs) has gained prominence due to their high antibacterial activity and reduced susceptibility to drug resistance, making them potential antibiotic substitutes. To advance the field of AMP recognition, an increasing number of natural language processing methods are being applied. These methods exhibit diversity in terms of pretraining models, pretraining data sets, word vector embeddings, feature encoding methods, and downstream classification models. Here, we provide a comprehensive survey of current BERT-based methods for AMP prediction. An independent benchmark test data set is constructed to evaluate the predictive capabilities of the surveyed tools. Furthermore, we compared the predictive performance of these computational methods based on six different AMP public databases. LM_pred (BFD) outperformed all other surveyed tools due to abundant pretraining data set and the unique vector embedding approach. To avoid the impact of varying training data sets used by different methods on prediction performance, we performed the 5-fold cross-validation experiments using the same data set, involving retraining. Additionally, to explore the applicability and generalization ability of the models, we constructed a short peptide data set and an external data set to test the retrained models. Although these prediction methods based on BERT can achieve good prediction performance, there is still room for improvement in recognition accuracy. With the continuous enhancement of protein language model, we proposed an AMP prediction method based on the ESM-2 pretrained model called iAMP-bert. Experimental results demonstrate that iAMP-bert outperforms other approaches. iAMP-bert is freely accessible to the public at http://iamp.aielab.cc/.

Application of the Semi-Supervised Learning Approach for Pavement Defect Detection.

Road surface quality is essential for driver comfort and safety, making it crucial to monitor pavement conditions and detect defects in real time. However, the diversity of defects and the complexity of ambient conditions make it challenging to develop an effective and robust classification and detection algorithm. In this study, we adopted a semi-supervised learning approach to train ResNet-18 for image feature retrieval and then classification and detection of pavement defects. The resulting feature embedding vectors from image patches were retrieved, concatenated, and randomly sampled to model a multivariate normal distribution based on the only one-class training pavement image dataset. The calibration pavement image dataset was used to determine the defect score threshold based on the receiver operating characteristic curve, with the Mahalanobis distance employed as a metric to evaluate differences between normal and defect pavement images. Finally, a heatmap derived from the defect score map for the testing dataset was overlaid on the original pavement images to provide insight into the network's decisions and guide measures to improve its performance. The results demonstrate that the model's classification accuracy improved from 0.868 to 0.887 using the expanded and augmented pavement image data based on the analysis of heatmaps.

Speaker-Attributed Training for Multi-Speaker Speech Recognition Using Multi-Stage Encoders and Attention-Weighted Speaker Embedding

Automatic speech recognition (ASR) aims at understanding naturally spoken human speech to be used as text inputs to machines. In multi-speaker environments, where multiple speakers are talking simultaneously with a large amount of overlap, a significant performance degradation may occur with conventional ASR systems if they are trained by recordings of single talkers. This paper proposes a multi-speaker ASR method that incorporates speaker embedding information as an additional input. The embedding information for each of the speakers in the training set was extracted as numeric vectors, and all of the embedding vectors were stacked to construct a total speaker profile matrix. The speaker profile matrix from the training dataset enables finding embedding vectors that are close to the speakers of the input recordings in the test conditions, and it helps to recognize the individual speakers’ voices mixed in the input. Furthermore, the proposed method efficiently reuses the decoder from the existing speaker-independent ASR model, eliminating the need for retraining the entire system. Various speaker embedding methods such as i-vector, d-vector, and x-vector were adopted, and the experimental results show 0.33% and 0.95% absolute (3.9% and 11.5% relative) improvements without and with the speaker profile in the word error rate (WER).

FedGST: An Efficient Federated Graph Neural Network for Spatio-temporal PoI Recommendation

With the proliferation of sensor networks in urban areas, vast amounts of data from location-based social network platforms are now available, thus enabling the stakeholders to develop location-aware services for the end users. PoI (Point of Interests) recommendation is one of the most trending services. PoI recommendation models are developed by considering geographical information and historical records. More recently, with the demand for achieving PoI recommendations on mobile devices, embedded devices, or Internet of Things (IoT) devices, the challenges posed by the limited computational resources, malicious clients, and data silos, the model performance gets immensely affected. To address the above challenges and achieve efficient PoI recommendations, we propose an efficient federated graph learning-based model for mining complex spatiotemporal features to generate recommendations. We implement a GRU-based encoder-decoder to learn the temporal hidden state embeddings. Simultaneously, a 2-layer graph network is used to understand spatial embedding vectors. We also introduce an efficient contribution evaluation method to speed up the training process and improve the recommendation performance. Experiments on the PoI recommendation task based on real-life check-in data validate the effectiveness of our proposed model, and the results indicate that our recommendation model can achieve competitive results with lower computational costs.

Dynamic Graph Representation Learning via Coupling-Process Model.

Representation learning based on dynamic graphs has received a lot of attention in recent years due to its wide range of application scenarios. Although many discrete or continuous dynamic graph representation learning methods have been proposed, many of them ignore the role of edge types. Through the observation of dynamic graphs in the real world, it is found that the types of various edges are very different in nature. They are roughly divided into two categories according to the frequency of occurrence: evolutive edges that appear infrequently and interactive edges that appear frequently. For both types of edges, we propose a coupling-process model (DyCPM) to capture the dynamic mechanisms of them. The model not only generates low-dimensional embedding vectors of nodes, but also aggregates the structural information and temporal information of two kinds of edges. In particular, we design a neural network parameterized discrete process to depict the change law of the topology of evolutive edges and a neural network parameterized temporal point process (TPP) to characterize the temporal dynamic rule of interactive edges. More importantly, we propose a coupling mechanism to transfer the information of the two processes through a shared embedding matrix and finally generate an embedding matrix that aggregates the topology information and temporal information of the two kinds of edges for the dynamic link prediction task. We evaluate our model and several baselines on real datasets. The experimental results show that our model can better aggregate the topology information and temporal information of the two kinds of edges according to their properties and outperforms several state-of-the-art baselines in the performance of dynamic link prediction.

Learning Joint Topic Representation for Detecting Drift in Social Media Text

Social media texts like tweets and blogs are collaboratively created by human interaction. Rapidly changing trends are leading to topic drift in the social media text. This drift is usually associated with words and hashtags. However, geotags play an essential part in determining topic distribution with location context. The rate of change in the distribution of words, hashtags and geotags cannot be considered uniform and must be handled accordingly. This paper builds a topic model that associates the topic with a mixture of distributions of words, hashtags and geotags. Stochastic gradient Langevin dynamic model with varying mini-batch sizes is used to capture the changes due to the asynchronous distribution of words and tags. Topic representations with co-occurrence and location contexts are specified as hashtag context vector and geotag context vector respectively. These two vectors are jointly learned to yield topical word embedding vectors over time conditioned on hashtags and geotags that can predict location-based topical variations effectively. When evaluated with Chennai and UK geolocated Twitter data, the proposed joint topical word embedding model enhanced by the social tags context, outperforms other methods.

GCRL: a graph neural network framework for network connectivity robustness learning

Abstract The resilience and adaptability of complex networks is crucial in ensuring their functionality against disruptions. Particularly, maintaining network connectivity under various attack scenarios is a key aspect of such resilience. Network connectivity refers to the degree to which nodes within a network are interconnected and able to exchange information or resources. Its robustness reflects the ability of a network to maintain connectivity under various attacks. Such ability has profound physical significance, ensuring the stability and reliability of real-world systems. Currently, connectivity robustness assessments rely heavily on very time-consuming attack simulations. This paper introduces a graph neural network framework for network connectivity robustness learning (GCRL) to advance the study of network connectivity robustness. GCRL transforms initial degree distributions and network topology into informative embedding vectors, which are then processed by a robustness learning module mainly composed of multi-layer perceptron, achieving both high speed and precision. Our extensive experiments demonstrate the superior performance of GCRL obtained in less time compared to existing methods, especially in tough scenarios where test data distributions significantly differ from the training set. The framework also shows adaptability to networks of different sizes, making it a more generalized solution for complex network robustness learning.

Ionic species representations for materials informatics

High-dimensional representations of the elements have become common within the field of materials informatics to build useful, structure-agnostic models for the chemistry of materials. However, the characteristics of elements change when they adopt a given oxidation state, with distinct structural preferences and physical properties. We explore several methods for developing embedding vectors of elements decorated with oxidation states. Graphs generated from 110 160 crystals are used to train representations of 84 elements that form 336 species. Clustering these learned representations of ionic species in low-dimensional space reproduces expected chemical heuristics, particularly the separation of cations from anions. We show that these representations have enhanced expressive power for property prediction tasks involving inorganic compounds. We expect that ionic representations, necessary for the description of mixed valence and complex magnetic systems, will support more powerful machine learning models for materials.

Deep learning neural network development for the classification of bacteriocin sequences produced by lactic acid bacteria

Background The rise of antibiotic-resistant bacteria presents a pressing need for exploring new natural compounds with innovative mechanisms to replace existing antibiotics. Bacteriocins offer promising alternatives for developing therapeutic and preventive strategies in livestock, aquaculture, and human health. Specifically, those produced by LAB are recognized as GRAS and QPS. Methods In this study was used a deep learning neural network for binary classification of bacteriocin amino acid sequences, distinguishing those produced by LAB. The features were extracted using the k-mer method and vector embedding. Ten different groups were tested, combining embedding vectors and k-mers: EV, ‘EV+3-mers’, ‘EV+5-mers’, ‘EV+7-mers’, ‘EV+15-mers’, ‘EV+20-mers’, ‘EV+3-mers+5-mers’, ‘EV+3-mers+7-mers’, ‘EV+5-mers+7-mers’, and ‘EV+15-mers+20-mers’. Results Five sets of 100 characteristic k-mers unique to bacteriocins produced by LAB were obtained for values of k = 3, 5, 7, 15, and 20. Significant difference was observed between using only and concatenation. Specially, ‘5-mers+7-mers+EV ’ group showed superior accuracy and loss results. Employing k-fold cross-validation with k=30, the average results for loss, accuracy, precision, recall, and F1 score were 9.90%, 90.14%, 90.30%, 90.10%, and 90.10% respectively. Folder 22 stood out with 8.50% loss, 91.47% accuracy, and 91.00% precision, recall, and F1 score. Conclusions The model developed in this study achieved consistent results with those seen in the reviewed literature. It outperformed some studies by 3-10%. The lists of characteristic k-mers pave the way to identify new bacteriocins that could be valuable for therapeutic and preventive strategies within the livestock, aquaculture industries, and potentially in human health.

Urban streets profiling with coupled spatio-temporal characteristics and topological information from the biking perspective

Urban street profiling is the spatio-temporal pattern discovery of street-level urban areas, which plays a vital role in understanding urban structures and dynamics. Due to the natural topology and various geographic characteristics on the streets, it is necessary to combine multi-dimensional spatio-temporal information to understand different profiles of streets. This research aims to develop a street profiling framework according to the coupled characteristics of streets. At the start, a bidirected dual graph and a spatial weighted graph embedding method were used to solve the street representation. Then, the street profiles can be extracted by clustering embedding vectors of streets and feature importance analysis. As the case study, we employed the bike trajectories and street view images in Xiamen, China to depict the geographic attributes of streets. The results can reveal nine spatio-temporal street profiles from the biking perspective, including three spatial distribution patterns and two spatial semantic patterns. Urban streets in the study area show a significant hierarchical pattern because of locations and the spatial lags of the biking behaviors. Meanwhile, the spatio-temporal characteristics of biking behaviors are the main factors of street profiles, though the street environment attributes participate in over half the number of profile types. We further evaluated the profiling ability of the proposed framework and the importance of urban street profiles using coupled characteristics. Overall, this study explored the profiling method for coupling static and dynamic characteristics of urban streets. The profiling results also help understand street usage and experiences by bikers, which have a practical value on the human-oriented classification of streets and further urban development from a geographic view.

Foundation model for efficient biological discovery in single-molecule data.

Modern data-intensive techniques offer ever deeper insights into biology, but render the process of discovery increasingly complex. For example, exploiting the unique ability of single-molecule fluorescence microscopy (SMFM)1-5. to uncover rare but critical intermediates often demands manual inspection of time traces and iterative ad hoc approaches that are difficult to systematize. To facilitate systematic and efficient discovery from SMFM data, we introduce META-SiM, a transformer-based foundation model pre-trained on diverse SMFM analysis tasks. META-SiM achieves high performance-rivaling best-in-class algorithms-on a broad range of analysis tasks including trace selection, classification, segmentation, idealization, and stepwise photobleaching analysis. Additionally, the model produces high-dimensional embedding vectors that encapsulate detailed information about each trace, which the web-based META-SiM Projector (https://www.simol-projector.org) casts into lower-dimensional space for efficient whole-dataset visualization, labeling, comparison, and sharing. Combining this Projector with the objective metric of Local Shannon Entropy enables rapid identification of condition-specific behaviors, even if rare or subtle. As a result, by applying META-SiM to an existing single-molecule Förster resonance energy transfer (smFRET) dataset6, we discover a previously unobserved intermediate state in pre-mRNA splicing. META-SiM thus removes bottlenecks, improves objectivity, and both systematizes and accelerates biological discovery in complex single-molecule data.

Understanding and mitigating dimensional collapse of Graph Contrastive Learning: A non-maximum removal approach

Graph Contrastive Learning (GCL) generates graph-level embeddings by maximizing Mutual Information between different augmented views of the same graph (positive pairs), and shows promising performance in graph representation learning (GRL) without the supervision of manual annotations. However, GCL suffers from a dimensional collapse problem, i.e., embedding vectors reside in a restricted low-dimensional subspace, curtailing the expressiveness of certain embedding dimensions. In this paper, we present a theoretical analysis identifying the smoothing effect of graph pooling and graph convolution’s implicit regularization as principal causes of dimension collapse in graph contrastive learning. To mitigate the above effects, we propose a Non-Maximum Removal Graph Contrastive Learning (nmrGCL) approach, which removes “prominent” dimensions (those significantly contributing to the similarity measure) for positive pairs within the pretext task. Comprehensive experiments on multiple benchmark datasets are conducted and the results show that the proposed nmrGCL outperforms the state-of-the-art methods.

Affective, semantic, frequency, and descriptive norms for 107 face emojis.

We introduce a novel dataset of affective, semantic, and descriptive norms for all facial emojis at the point of data collection. We gathered and examined subjective ratings of emojis from 138 German speakers along five essential dimensions: valence, arousal, familiarity, clarity, and visual complexity. Additionally, we provide absolute frequency counts of emoji use, drawn from an extensive Twitter corpus, as well as a much smaller WhatsApp database. Our results replicate the well-established quadratic relationship between arousal and valence of lexical items, also known for words. We also report associations among the variables: for example, the subjective familiarity of an emoji is strongly correlated with its usage frequency, and positively associated with its emotional valence and clarity of meaning. We establish the meanings associated with face emojis, by asking participants for up to three descriptions for each emoji. Using this linguistic data, we computed vector embeddings for each emoji, enabling an exploration of their distribution within the semantic space. Our description-based emoji vector embeddings not only capture typical meaning components of emojis, such as their valence, but also surpass simple definitions and direct emoji2vec models in reflecting the semantic relationship between emojis and words. Our dataset stands out due to its robust reliability and validity. This new semantic norm for face emojis impacts the future design of highly controlled experiments focused on the cognitive processing of emojis, their lexical representation, and their linguistic properties.

POI recommendation by deep neural matrix factorization integrated attention-aware meta-paths

With the continuous accumulation of massive amounts of mobile data, point-of-interest (POI) recommendation has become a vital task for location-based social networks. Deep neural networks or matrix factorization (MF) alone are challenging to effectively learn user–POI interaction functions. Moreover, the user–POI interaction matrix is sparse, and the heterogeneous characteristics of auxiliary information are underused. Therefore, we propose an innovative POI recommendation method that integrates attention-aware meta-paths based on deep neural matrix factorization (DNMF-AM). First, we develop a multi-relational heterogeneous information network of “user–POI–geographic region–POI category.” Multiple-weighted isomorphic information networks based on meta-paths are employed to obtain node-embedding vectors across different relationships. Attention networks are employed to aggregate node vectors across various relationships and serve as auxiliary information to mitigate the challenges of data sparsity. Subsequently, the internal embedding vectors of the users and POIs are extracted using feature embedding based on the user–POI interaction matrix. Second, these vectors are integrated with the embedding vectors obtained by aggregating the attention networks. Third, deep neural matrix factorization is used to learn linear and nonlinear user–POI interactions to mitigate the implicit feedback problem. This outcome is achieved using generalized matrix factorization and convolution-constrained multi-head self-attention mechanism deep neural networks. Extensive experiments conducted on two real-world datasets demonstrate that the DNMF-AM outperforms the optimal baseline NeuMF-CAA by 4.24% and 5.04% in terms of HR@10 and NDCG@10, respectively.

Anomalous behavior detection based on optimized graph embedding representation in social networks

Anomalous behaviors in social networks can lead to privacy leaks and the spread of false information. In this paper, we propose an anomalous behavior detection method based on optimized graph embedding representation. Specifically, the user behavior logs are first extracted into a social network user behavior temporal knowledge graph, based on which the graph embedding representation method is used to transform the network topology and temporal information in the user behavior knowledge graph into structural embedding vectors and temporal information embedding vectors, and the hybrid attention mechanism is used to merge the two types of vectors to obtain the final entity embedding to complete the prediction and complementation of the temporal knowledge graph of user behavior. We use graph neural networks, which use the temporal information of user behaviors as a time constraint and capture both user behavioral and semantic information. It converts the two parts of information into vectors for concatenation and linear transformation to obtain a comprehensive representation vector of the whole subgraph, as well as joint deep learning models to evaluate abnormal behavior. Finally, we perform experiments on the Yelp dataset to validate that our method achieves a 9.56% improvement in the F1-score.

Process mining embeddings: Learning vector representations for Petri nets

Process Mining offers a powerful framework for uncovering, analyzing, and optimizing real-world business processes. Petri nets provide a versatile means of modeling process behavior. However, traditional methods often struggle to effectively compare complex Petri nets, hindering their potential for process enhancement. To address this challenge, we introduce PetriNet2Vec, an unsupervised methodology inspired by Doc2Vec. This approach converts Petri nets into embedding vectors, facilitating the comparison, clustering, and classification of process models. We validated our approach using the PDC Dataset, comprising 96 diverse Petri net models. The results demonstrate that PetriNet2Vec effectively captures the structural properties of process models, enabling accurate process classification and efficient process retrieval. Specifically, our findings highlight the utility of the learned embeddings in two key downstream tasks: process classification and process retrieval. In process classification, the embeddings allowed for accurate categorization of process models based on their structural properties. In process retrieval, the embeddings enabled efficient retrieval of similar process models using cosine distance. These results demonstrate the potential of PetriNet2Vec to significantly enhance process mining capabilities.

PatchNet: Maximize the Exploration of Congeneric Semantics for Weakly Supervised Semantic Segmentation.

With the increase in the number of image data and the lack of corresponding labels, weakly supervised learning has drawn a lot of attention recently in computer vision tasks, especially in the fine-grained semantic segmentation problem. To alleviate human efforts from expensive pixel-by-pixel annotations, our method focuses on weakly supervised semantic segmentation (WSSS) with image-level labels, which are much easier to obtain. As a considerable gap exists between pixel-level segmentation and image-level labels, how to reflect the image-level semantic information on each pixel is an important question. To explore the congeneric semantic regions from the same class to the maximum, we construct the patch-level semantic augmentation network (PatchNet) based on the self-detected patches from different images that contain the same class labels. Patches can frame the objects as much as possible and include as little background as possible. The patch-level semantic augmentation network that is established with patches as the nodes can maximize the mutual learning of similar objects. We regard the embedding vectors of patches as nodes and use a transformer-based complementary learning module to construct weighted edges according to the embedding similarity between different nodes. Moreover, to better supplement semantic information, we propose softcomplementary loss functions matched with the whole network structure. We conduct experiments on the popular PASCAL VOC 2012 and MS COCO 2014 benchmarks, and our model yields the state-of-the-art performance.

Efficient greenhouse segmentation with visual foundation models: achieving more with fewer samples

Introduction: The Vision Transformer (ViT) model, which leverages self-supervised learning, has shown exceptional performance in natural image segmentation, suggesting its extensive potential in visual tasks. However, its effectiveness diminishes in remote sensing due to the varying perspectives of remote sensing images and unique optical properties of features like the translucency of greenhouses. Additionally, the high cost of training Visual Foundation Models (VFMs) from scratch for specific scenes limits their deployment.Methods: This study investigates the feasibility of rapidly deploying VFMs on new tasks by using embedding vectors generated by VFMs as prior knowledge to enhance traditional segmentation models’ performance. We implemented this approach to improve the accuracy and robustness of segmentation with the same number of trainable parameters. Comparative experiments were conducted to evaluate the efficiency and effectiveness of this method, especially in the context of greenhouse detection and management.Results: Our findings indicate that the use of embedding vectors facilitates rapid convergence and significantly boosts segmentation accuracy and robustness. Notably, our method achieves or exceeds the performance of traditional segmentation models using only about 40% of the annotated samples. This reduction in the reliance on manual annotation has significant implications for remote sensing applications.Discussion: The application of VFMs in remote sensing tasks, particularly for greenhouse detection and management, demonstrated enhanced segmentation accuracy and reduced dependence on annotated samples. This method adapts more swiftly to different lighting conditions, enabling more precise monitoring of agricultural resources. Our study underscores the potential of VFMs in remote sensing tasks and opens new avenues for the expansive application of these models in diverse downstream tasks.

Soviet View of the World. Exploring Long-Term Visual Patterns in “Novosti dnia” Newsreel Journal (1945-1992)

Newsreels, short documentary news films, were an influential channel of mass communication and propaganda in the Soviet Union. They served as an important means of visualizing the world for audiences in the way the Soviet authorities wanted it to be depicted. Studies in Soviet visual culture have recognized both continuities of repeating patterns and changes in the post-World War II period. This understanding is based primarily on temporally limited source selections, while a more systematic study of the developments in Soviet visual culture over a longer period is pending. In this article, we reveal long-term continuities, subtle changes, and sudden shifts in the official visual discourse in the Soviet newsreel series ‘Novosti dnia’ (News of the Day) 1945 to 1992. We study visual patterns in approximately 1,700 digitized newsreel issues, each about ten minutes long, using multidimensional vector embeddings. These embeddings, produced from the central frames of 205,678 shots, help visually evaluate the footage and assess visual similarities based on ResNet50 feature vectors. For this, we use the Collection Space Navigator tool. The article demonstrates how multidimensional vector embeddings can be used to study the internal time of the films, and the external time of the years running by.

Reconstruction of OFDM Signals Using a Dual Discriminator CGAN with BiLSTM and Transformer.

Communication signal reconstruction technology represents a critical area of research within communication countermeasures and signal processing. Considering traditional OFDM signal reconstruction methods' intricacy and suboptimal reconstruction performance, a dual discriminator CGAN model incorporating LSTM and Transformer is proposed. When reconstructing OFDM signals using the traditional CNN network, it becomes challenging to extract intricate temporal information. Therefore, the BiLSTM network is incorporated into the first discriminator to capture timing details of the IQ (In-phase and Quadrature-phase) sequence and constellation map information of the AP (Amplitude and Phase) sequence. Subsequently, following the addition of fixed position coding, these data are fed into the core network constructed based on the Transformer Encoder for further learning. Simultaneously, to capture the correlation between the two IQ signals, the VIT (Vision in Transformer) concept is incorporated into the second discriminator. The IQ sequence is treated as a single-channel two-dimensional image and segmented into pixel blocks containing IQ sequence through Conv2d. Fixed position coding is added and sent to the Transformer core network for learning. The generator network transforms input noise data into a dimensional space aligned with the IQ signal and embedding vector dimensions. It appends identical position encoding information to the IQ sequence before sending it to the Transformer network. The experimental results demonstrate that, under commonly utilized OFDM modulation formats such as BPSK, QPSK, and 16QAM, the time series waveform, constellation diagram, and spectral diagram exhibit high-quality reconstruction. Our algorithm achieves improved signal quality while managing complexity compared to other reconstruction methods.