Embedding Vectors Research Articles

ExplainLFS: Explaining neural architectures for similarity learning from local perturbations in the latent feature space

Despite the increasing development in recent years of explainability techniques for deep neural networks, only some are dedicated to explaining the decisions made by neural networks for similarity learning. While existing approaches can explain classification models, their adaptation to generate visual similarity explanations is not trivial. Neural architectures devoted to this task learn an embedding that maps similar examples to nearby vectors and non-similar examples to distant vectors in the feature space. In this paper, we propose a post-hoc agnostic technique that explains the inference of such architectures on a pair of images. The proposed method establishes a relation between the most important features of the abstract feature space and the input feature space (pixels) of an image. For this purpose, we employ a relevance assignment and a perturbation process based on the most influential latent features in the inference. Then, a reconstruction process of the images of the pair is carried out from the perturbed embedding vectors. This process relates the latent features to the original input features. The results indicate that our method produces “continuous” and “selective” explanations. A sharp drop in the value of the function (summarized by a low value of the area under the curve) indicates its superiority over other explainability approaches when identifying features relevant to similarity learning. In addition, we demonstrate that our technique is agnostic to the specific type of similarity model, e.g., we show its applicability in two similarity learning tasks: face recognition and image retrieval.

FGeo-TP: A Language Model-Enhanced Solver for Euclidean Geometry Problems

The application of contemporary artificial intelligence techniques to address geometric problems and automated deductive proofs has always been a grand challenge to the interdisciplinary field of mathematics and artificial intelligence. This is the fourth article in a series of our works, in our previous work, we established a geometric formalized system known as FormalGeo. Moreover, we annotated approximately 7000 geometric problems, forming the FormalGeo7k dataset. Despite the fact that FGPS (Formal Geometry Problem Solver) can achieve interpretable algebraic equation solving and human-like deductive reasoning, it often experiences timeouts due to the complexity of the search strategy. In this paper, we introduced FGeo-TP (theorem predictor), which utilizes the language model to predict the theorem sequences for solving geometry problems. The encoder and decoder components in the transformer architecture naturally establish a mapping between the sequences and embedding vectors, exhibiting inherent symmetry. We compare the effectiveness of various transformer architectures, such as BART or T5, in theorem prediction, and implement pruning in the search process of FGPS, thereby improving its performance when solving geometry problems. Our results demonstrate a significant increase in the problem-solving rate of the language model-enhanced FGeo-TP on the FormalGeo7k dataset, rising from 39.7% to 80.86%. Furthermore, FGeo-TP exhibits notable reductions in solution times and search steps across problems of varying difficulty levels.

Fast knowledge graph completion using graphics processing units

Knowledge graphs can be used in many areas related to data semantics such as question-answering systems, knowledge based systems. However, the currently constructed knowledge graphs need to be complemented for better knowledge in terms of relations. It is called knowledge graph completion. To add new relations to the existing knowledge graph by using knowledge graph embedding models, we have to evaluate N×N×R vector operations, where N is the number of entities and R is the number of relation types. It is very costly.In this paper, we provide an efficient knowledge graph completion framework on GPUs to get new relations using knowledge graph embedding vectors. In the proposed framework, we first define transformable to a metric space and then provide a method to transform the knowledge graph completion problem into the similarity join problem for a model which is transformable to a metric space. After that, to efficiently process the similarity join problem, we derive formulas using the properties of a metric space. Based on the formulas, we develop a fast knowledge graph completion algorithm. Finally, we experimentally show that our framework can efficiently process the knowledge graph completion problem.

Cigarette tasting Chinese text classification for low-resource scenarios

At present, the Chinese text field is facing challenges from low resource issues such as data scarcity and annotation difficulties. Moreover, in the domain of cigarette tasting, cigarette tasting texts tend to be colloquial, making it difficult to obtain valuable and high-quality tasting texts. Therefore, in this paper, we construct a cigarette tasting dataset (CT2023) and propose a novel Chinese text classification method based on ERNIE and Comparative Learning for Low-Resource scenarios (ECLLR). Firstly, to address the issues of limited vocabulary diversity and sparse features in cigarette tasting text, we utilize Term Frequency-Inverse Document Frequency (TF-IDF) to extract key terms, supplementing the discriminative features of the original text. Secondly, ERNIE is employed to obtain sentence-level vector embedding of the text. Finally, contrastive learning model is used to further refine the text after fusing the keyword features, thereby enhancing the performance of the proposed text classification model. Experiments on the CT2023 dataset demonstrate an accuracy rate of 96.33% for the proposed method, surpassing the baseline model by at least 11 percentage points, and showing good text classification performance. It is thus clear that the proposed approach can effectively provide recommendations and decision support for cigarette production processes in tobacco companies.

Scoring alignments by embedding vector similarity.

Sequence similarity is of paramount importance in biology, as similar sequences tend to have similar function and share common ancestry. Scoring matrices, such as PAM or BLOSUM, play a crucial role in all bioinformatics algorithms for identifying similarities, but have the drawback that they are fixed, independent of context. We propose a new scoring method for amino acid similarity that remedies this weakness, being contextually dependent. It relies on recent advances in deep learning architectures that employ self-supervised learning in order to leverage the power of enormous amounts of unlabelled data to generate contextual embeddings, which are vector representations for words. These ideas have been applied to protein sequences, producing embedding vectors for protein residues. We propose the E-score between two residues as the cosine similarity between their embedding vector representations. Thorough testing on a wide variety of reference multiple sequence alignments indicate that the alignments produced using the new $E$-score method, especially ProtT5-score, are significantly better than those obtained using BLOSUM matrices. The new method proposes to change the way alignments are computed, with far-reaching implications in all areas of textual data that use sequence similarity. The program to compute alignments based on various $E$-scores is available as a web server at e-score.csd.uwo.ca. The source code is freely available for download from github.com/lucian-ilie/E-score.

Neural Machine Translation with CARU-Embedding Layer and CARU-Gated Attention Layer

The attention mechanism performs well for the Neural Machine Translation (NMT) task, but heavily depends on the context vectors generated by the attention network to predict target words. This reliance raises the issue of long-term dependencies. Indeed, it is very common to combine predicates with postpositions in sentences, and the same predicate may have different meanings when combined with different postpositions. This usually poses an additional challenge to the NMT study. In this work, we observe that the embedding vectors of different target tokens can be classified by part-of-speech, thus we analyze the Natural Language Processing (NLP) related Content-Adaptive Recurrent Unit (CARU) unit and apply it to our attention model (CAAtt) and embedding layer (CAEmbed). By encoding the source sentence with the current decoded feature through the CARU, CAAtt is capable of achieving translation content-adaptive representations, which attention weights are contributed and enhanced by our proposed L1expNx normalization. Furthermore, CAEmbed aims to alleviate long-term dependencies in the target language through partial recurrent design, performing the feature extraction in a local perspective. Experiments on the WMT14, WMT17, and Multi30k translation tasks show that the proposed model achieves improvements in BLEU scores and enhancement of convergence over the attention-based plain NMT model. We also investigate the attention weights generated by the proposed approaches, which indicate that refinement over the different combinations of adposition can lead to different interpretations. Specifically, this work provides local attention to some specific phrases translated in our experiment. The results demonstrate that our approach is effective in improving performance and achieving a more reasonable attention distribution compared to the state-of-the-art models.

Guided diffusion for molecular generation with interaction prompt.

Molecular generative models have exhibited promising capabilities in designing molecules from scratch with high binding affinities in a predetermined protein pocket, offering potential synergies with traditional structural-based drug design strategy. However, the generative processes of such models are random and the atomic interaction information between ligand and protein are ignored. On the other hand, the ligand has high propensity to bind with residues called hotspots. Hotspot residues contribute to the majority of the binding free energies and have been recognized as appealing targets for designed molecules. In this work, we develop an interaction prompt guided diffusion model, InterDiff to deal with the challenges. Four kinds of atomic interactions are involved in our model and represented as learnable vector embeddings. These embeddings serve as conditions for individual residue to guide the molecular generative process. Comprehensive in silico experiments evince that our model could generate molecules with desired ligand-protein interactions in a guidable way. Furthermore, we validate InterDiff on two realistic protein-based therapeutic agents. Results show that InterDiff could generate molecules with better or similar binding mode compared to known targeted drugs.

THE NO TRAIN NO GAIN SYSTEM FOR O-COCOSDA AND VLSP 2022 - A-MSV SHARED TASK: ASIAN MULTILINGUAL SPEAKER VERIFICATION

This paper proposes a semi-supervised multilingual speaker verification (MSV) system submitted for the 2 tasks, MSV for the Asian language inside the training set (T01) and outside the training set (T02) in O-COCOSDA and VLSP challenge 2022.To solve the problem, our strategy is training a baseline acoustic model with given labeled data (MSV CommonVoice) andfine-tuning the trained acoustic model with both given labeled data and given unlabeled data (MSV Youtube). To achieve the fine-tuning step, the unlabeled data is converted to labeled data by pseudo labeling technique using the clustering method with the embedding vectors extracted from the trained acoustic model. Besides, we also apply test-time augmentation, back-end scoring, and score normalization with the AS-Norm technique to improve the result. When evaluated on the VLSP 2022 challenge's given test set, our best system with baseline ECAPA-TDNN achieves an equal error rate (EER) of 2.296% in T01 and 3.3296% in T02, which ranks second rank in both two tasks.

Levenshtein Distance Embedding with Poisson Regression for DNA Storage

Efficient computation or approximation of Levenshtein distance, a widely-used metric for evaluating sequence similarity, has attracted significant attention with the emergence of DNA storage and other biological applications. Sequence embedding, which maps Levenshtein distance to a conventional distance between embedding vectors, has emerged as a promising solution. In this paper, a novel neural network-based sequence embedding technique using Poisson regression is proposed. We first provide a theoretical analysis of the impact of embedding dimension on model performance and present a criterion for selecting an appropriate embedding dimension. Under this embedding dimension, the Poisson regression is introduced by assuming the Levenshtein distance between sequences of fixed length following a Poisson distribution, which naturally aligns with the definition of Levenshtein distance. Moreover, from the perspective of the distribution of embedding distances, Poisson regression approximates the negative log likelihood of the chi-squared distribution and offers advancements in removing the skewness. Through comprehensive experiments on real DNA storage data, we demonstrate the superior performance of the proposed method compared to state-of-the-art approaches.

Privacy-preserving data integration and sharing in multi-party IoT environments: An entity embedding perspective

The increasing prevalence of IoT applications highlights the urgency for insightful data fusion and information acquisition, boosting data integration and sharing needs. However, challenges arise in multi-party data sharing due to inherent data heterogeneity and privacy concerns. To address these issues, this paper discusses the feasibility of using embedding vectors as the semantic representation, aiming to enhance interoperability across diverse data sources and lay the foundation for natural language-based data querying. At the specific method level, this paper proposes an improved entity tree embedding algorithm to reduce information loss and ameliorate the representation of entity semantics. Additionally, a privacy preservation mechanism based on the entity embedding approach is introduced to provide privacy protection for text-based data. Experimental results on address data demonstrate the mechanism’s efficacy in achieving privacy protection comparable to the widely adopted 2D Laplace plane noise method. Furthermore, incorporating the entity tree embedding into the privacy mechanism could yield more robust and reasonable results regarding location privacy and service quality, signifying the validity of the entity embedding results.

Fusion of GBDT and neural network for click-through rate estimation

Aiming at the issue that the current click-through rate prediction methods ignore the varying impacts of different input features on prediction accuracy and exhibit low accuracy when dealing with large-scale data, a click-through rate prediction method (GBIFM) which combines Gradient Boosting Decision Tree (GBDT) and Input-aware Factorization Machine (IFM) is proposed in this paper. The proposed GBIFM method employs GBDT for data processing, which can flexibly handle various types of data without the need for one-hot encoding of discrete features. An Input-aware strategy is introduced to refine the weight vector and embedding vector of each feature for different instances, adaptively learning the impact of each input vector on feature representation. Furthermore, a fully connected network is incorporated to capture high-order features in a non-linear manner, enhancing the method’s ability to express and generalize complex structured data. A comprehensive experiment is conducted on the Criteo and Avazu datasets, the results show that compared to typical methods such as DeepFM, AFM, and IFM, the proposed method GBIFM can increase the AUC value by 10% –12% and decrease the Logloss value by 6% –20%, effectively improving the accuracy of click-through rate prediction.

A representation-learning approach for insurance pricing with images

AbstractUnstructured data are a promising new source of information that insurance companies may use to understand their risk portfolio better and improve the customer experience. However, these novel data sources are difficult to incorporate into existing ratemaking frameworks due to the size and format of the unstructured data. This paper proposes a framework to use street view imagery within a generalized linear model. To do so, we use representation learning to extract an embedding vector containing useful information from the image. This embedding is dense and low dimensional, making it appropriate to use within existing ratemaking models. We find that there is useful information included in street view imagery to predict the frequency of claims for certain types of perils. This model can be used as in a ratemaking framework but also opens the door to future empirical research on attempting to extract which characteristics within the image leads to increased or decreased predicted claim frequencies. Throughout, we discuss the practical difficulties (technical and social) of using this type of data for insurance pricing.

A SURVEY ON SUMMARIZATION OF LEGAL TEXTS

The task of summarizing legal texts is of paramount importance in facilitating access to legal knowledge. This survey paper critically reviews and analyzes diverse methodologies applied to the challenging task of summarizing legal texts. In the supervised realm, models such as GIST, SummaRuNNer, and BERTSUM have gained prominence, demonstrating prowess in extractive summarization techniques. Semi-supervised methods, leveraging innovative techniques such as TF-IDF Scores and Sentence Embeddings, have proven effective in handling the intricate structure of court judgments. Unsolicited approaches, CaseSummarizer and exemplified by USLT-LEGAL-BERT, address the challenges of summarization without the luxury of labeled data, showcasing their adaptability to diverse legal domains. Furthermore, the paper explores legal document retrieval strategies, incorporating document vector embeddings and deep learning techniques, underscoring the significance of domain-specific approaches. The study encompasses a variety of models, from Legal-BERT to hierarchical encoder decoder frameworks, showcasing their applicability in handling the complexity inherent in legal texts. Additionally, the survey delves into BERT-based techniques for legal classification tasks, emphasizing the crucial role of context and introducing innovative methods such as "Stride 64." In the realm of legal text simplification, the survey scrutinizes the KIS model and graph-based approaches, revealing nuanced trade- offs between readability, fluency, and meaning preservation. The research culminates with an exploration of the team's participation in LegalEval 2023 shared tasks, underscoring the prowess of deep learning models in tasks such as rhetorical role classification and legal named entity extraction.This survey paper provides a nuanced and comprehensive perspective on the evolving landscape of legal document summarization, offering valuable insights into current methodologies, challenges, and potential future directions for advancing the field. Keywords- Legal text Summarization,Supervised Learning, Extractive Summarization,Semi-supervised methods,Deep learning Techniques,Legal Document retrieval.

Recommendation Approach Based on TwinBERT Model

In order to provide more personalized recommendation services for users as well as improve the platform's traffic and user satisfaction, a recommendation method based on the TwinBERT model has been proposed. As a matter of fact, this method combines not only the basic information of users and items but also user comments or item text information to predict users' behaviors and make more accurate recommendations. The text information in the dataset is encoded in sentences using the BERT model, and the encoded text features are fused with other features to obtain a representation of user and item embedding vectors. These vectors are then used to predict the user's rating of the item and recommend relevant items for the user. The proposed method was tested on the Amazon review dataset and the MovieLens dataset, and the experimental results showed that it improved over traditional recommendation algorithms in terms of accuracy and efficiency.

Run-indexed time-varying Bayesian optimization with positional encoding for auto-tuning of controllers: Application to a plasma-assisted deposition process with run-to-run drifts

Bayesian optimization (BO) has emerged as a useful paradigm for automatic calibration (aka auto-tuning) of advanced optimization- and learning-based controllers whose closed-loop performance depends on the choice of several tuning parameters in highly nonlinear and nonconvex ways. However, BO approaches to controller auto-tuning commonly rely on the assumption that system dynamics remain constant, which does not hold for systems with time-varying dynamics, for example, due to gradual aging or persistent environmental drifts. This challenge can be further compounded when gradual and persistent system drifts occur over a series of process runs. Existing time-varying BO (TVBO) approaches with spatio-temporal kernels fall short of effectively handling an integer run index, which is imperative for capturing run-to-run changes in the system behavior. To this end, this paper presents a run-indexed TVBO (RI-TVBO) approach that can systematically account for run-to-run process drifts as the system is queried over sequential process runs. The proposed approach relies on adapting the non-stationary Wiener process kernel to accommodate an integer run index, instead of time. This is done via positional encoding that incorporates the integer run index and, thus, enables describing run-to-run variations in system dynamics. The positional embedding vector associated with each run index is then mapped onto a scalar value to leverage the relationships between different process runs within the probabilistic surrogate model of the objective function in RI-TVBO. The performance of RI-TVBO is evaluated for auto-tuning of an offset-free model predictive controller for a low-temperature plasma-assisted process for thin film deposition. Simulation results demonstrate the superior performance of RI-TVBO over standard BO and TVBO under different scenarios of run-to-run process drifts encountered in plasma-assisted deposition processes in semiconductor manufacturing.

Margin and Shared Proxies: Advanced Proxy Anchor Loss for Out-of-Domain Intent Classification

Out-of-Domain (OOD) intent classification is an important task for a dialog system, as it allows for appropriate responses to be generated. Previous studies aiming to solve the OOD intent classification task have generally adopted metric learning methods to generate decision boundaries in the embedding space. However, these existing methods struggle to capture the high-dimensional semantic features of data, as they learn decision boundary using scalar distances. They also use generated OOD samples for learning. However, such OOD samples are biased, and they cannot include all real-world OOD intents, thus representing a limitation. In the current paper, we attempt to overcome these challenges by using Advanced Proxy-Anchor loss, which introduces a margin proxy and shared proxy. First, to generate a decision boundary that has the high-dimensional semantic features of training data, we use a margin proxy for learnable embedding vectors. Next, the shared proxy, which is shared by all In-Domain (IND) samples, is introduced to make it possible to learn the discriminative feature between IND intents and OOD intent, ultimately leading to the improved classification of OOD samples. We conduct evaluations of the proposed method using three benchmark datasets. The experimental results demonstrate that our method achieved an improved performance compared to the methods described in previous studies.

DeepMedFeature: An Accurate Feature Extraction and Drug-Drug Interaction Model for Clinical Text in Medical Informatics

Drug-drug interactions (DDIs) are an important biological phenomenon which can result in medical errors from medical practitioners. Drug interactions can change the molecular structure of interacting agents which may prove to be fatal in the worst case. Finding drug interactions early in diagnosis can be pivotal in side-effect prevention. The growth of big data provides a rich source of information for clinical studies to investigate DDIs. We propose a hierarchical classification model which is double-pass in nature. The first pass predicts the occurrence of an interaction and then the second pass further predicts the type of interaction such as effect, advice, mechanism, and int. We applied different deep learning algorithms with Convolutional Bi-LSTM (ConvBLSTM) proving to be the best. The results show that pre-trained vector embeddings prove to be the most appropriate features. The F1-score of the ConvBLSTM algorithm turned out to be 96.39% and 98.37% in Russian and English language respectively which is greater than the state-of-the-art systems. According to the results, it can be concluded that adding a convolution layer before the bi-directional pass improves model performance in the automatic classification and extraction of drug interactions, using pre-trained vector embeddings such as Fasttext and Bio-Bert.

Hierarchical functional differences between gyri and sulci at different scales.

Gyri and sulci are 2 fundamental cortical folding patterns of the human brain. Recent studies have suggested that gyri and sulci may play different functional roles given their structural and functional heterogeneity. However, our understanding of the functional differences between gyri and sulci remains limited due to several factors. Firstly, previous studies have typically focused on either the spatial or temporal domain, neglecting the inherently spatiotemporal nature of brain functions. Secondly, analyses have often been restricted to either local or global scales, leaving the question of hierarchical functional differences unresolved. Lastly, there has been a lack of appropriate analytical tools for interpreting the hierarchical spatiotemporal features that could provide insights into these differences. To overcome these limitations, in this paper, we proposed a novel hierarchical interpretable autoencoder (HIAE) to explore the hierarchical functional difference between gyri and sulci. Central to our approach is its capability to extract hierarchical features via a deep convolutional autoencoder and then to map these features into an embedding vector using a carefully designed feature interpreter. This process transforms the features into interpretable spatiotemporal patterns, which are pivotal in investigating the functional disparities between gyri and sulci. We evaluate the proposed framework on Human Connectome Project task functional magnetic resonance imaging dataset. The experiments demonstrate that the HIAE model can effectively extract and interpret hierarchical spatiotemporal features that are neuroscientifically meaningful. The analyses based on the interpreted features suggest that gyri are more globally activated, whereas sulci are more locally activated, demonstrating a distinct transition in activation patterns as the scale shifts from local to global. Overall, our study provides novel insights into the brain's anatomy-function relationship.

Learning Implicit Functions for Dense 3D Shape Correspondence of Generic Objects.

The objective of this paper is to learn dense 3D shape correspondence for topology-varying generic objects in an unsupervised manner. Conventional implicit functions estimate the occupancy of a 3D point given a shape latent code. Instead, our novel implicit function produces a probabilistic embedding to represent each 3D point in a part embedding space. Assuming the corresponding points are similar in the embedding space, we implement dense correspondence through an inverse function mapping from the part embedding vector to a corresponded 3D point. Both functions are jointly learned with several effective and uncertainty-aware loss functions to realize our assumption, together with the encoder generating the shape latent code. During inference, if a user selects an arbitrary point on the source shape, our algorithm can automatically generate a confidence score indicating whether there is a correspondence on the target shape, as well as the corresponding semantic point if there is one. Such a mechanism inherently benefits man-made objects with different part constitutions. The effectiveness of our approach is demonstrated through unsupervised 3D semantic correspondence and shape segmentation.

Attribute Graph Embedding Based on Multi-Order Adjacency Views and Attention Mechanisms

Graph embedding plays an important role in the analysis and study of typical non-Euclidean data, such as graphs. Graph embedding aims to transform complex graph structures into vector representations for further machine learning or data mining tasks. It helps capture relationships and similarities between nodes, providing better representations for various tasks on graphs. Different orders of neighbors have different impacts on the generation of node embedding vectors. Therefore, this paper proposes a multi-order adjacency view encoder to fuse the feature information of neighbors at different orders. We generate different node views for different orders of neighbor information, consider different orders of neighbor information through different views, and then use attention mechanisms to integrate node embeddings from different views. Finally, we evaluate the effectiveness of our model through downstream tasks on the graph. Experimental results demonstrate that our model achieves improvements in attributed graph clustering and link prediction tasks compared to existing methods, indicating that the generated embedding representations have higher expressiveness.