Different Types Of Embeddings Research Articles

Contextualized dynamic meta embeddings based on Gated CNNs and self-attention for Arabic machine translation

PurposeThe paper aims to enhance Arabic machine translation (MT) by proposing novel approaches: (1) a dimensionality reduction technique for word embeddings tailored for Arabic text, optimizing efficiency while retaining semantic information; (2) a comprehensive comparison of meta-embedding techniques to improve translation quality; and (3) a method leveraging self-attention and Gated CNNs to capture token dependencies, including temporal and hierarchical features within sentences, and interactions between different embedding types. These approaches collectively aim to enhance translation quality by combining different embedding schemes and leveraging advanced modeling techniques.Design/methodology/approachRecent works on MT in general and Arabic MT in particular often pick one type of word embedding model. In this paper, we present a novel approach to enhance Arabic MT by addressing three key aspects. Firstly, we propose a new dimensionality reduction technique for word embeddings, specifically tailored for Arabic text. This technique optimizes the efficiency of embeddings while retaining their semantic information. Secondly, we conduct an extensive comparison of different meta-embedding techniques, exploring the combination of static and contextual embeddings. Through this analysis, we identify the most effective approach to improve translation quality. Lastly, we introduce a novel method that leverages self-attention and Gated convolutional neural networks (CNNs) to capture token dependencies, including temporal and hierarchical features within sentences, as well as interactions between different types of embeddings. Our experimental results demonstrate the effectiveness of our proposed approach in significantly enhancing Arabic MT performance. It outperforms baseline models with a BLEU score increase of 2 points and achieves superior results compared to state-of-the-art approaches, with an average improvement of 4.6 points across all evaluation metrics.FindingsThe proposed approaches significantly enhance Arabic MT performance. The dimensionality reduction technique improves the efficiency of word embeddings while preserving semantic information. Comprehensive comparison identifies effective meta-embedding techniques, with the contextualized dynamic meta-embeddings (CDME) model showcasing competitive results. Integration of Gated CNNs with the transformer model surpasses baseline performance, leveraging both architectures' strengths. Overall, these findings demonstrate substantial improvements in translation quality, with a BLEU score increase of 2 points and an average improvement of 4.6 points across all evaluation metrics, outperforming state-of-the-art approaches.Originality/valueThe paper’s originality lies in its departure from simply fine-tuning the transformer model for a specific task. Instead, it introduces modifications to the internal architecture of the transformer, integrating Gated CNNs to enhance translation performance. This departure from traditional fine-tuning approaches demonstrates a novel perspective on model enhancement, offering unique insights into improving translation quality without solely relying on pre-existing architectures. The originality in dimensionality reduction lies in the tailored approach for Arabic text. While dimensionality reduction techniques are not new, the paper introduces a specific method optimized for Arabic word embeddings. By employing independent component analysis (ICA) and a post-processing method, the paper effectively reduces the dimensionality of word embeddings while preserving semantic information which has not been investigated before especially for MT task.

Deep speaker embeddings for Speaker Verification: Review and experimental comparison

The construction of speaker-specific acoustic models for automatic speaker recognition is almost exclusively based on deep neural network-based speaker embeddings. This work aims to review the recent progress in speaker embedding development and to perform an experimental benchmark experimental comparison among the state-of-the-art deep speaker representations for a Speaker Verification (SV) task. The performance evaluation of the existing and proposed models on the VoxCeleb1 benchmark database shows that the SV systems based on r-vectors with a Res2Net convolutional architecture including multi-head attention pooling and additive margin softmax outperform other solutions such as d-vectors, x-vectors and conventional r-vectors. In addition, an ensemble network is proposed that fuses the best-performing speaker embeddings. It was found that different types of embeddings can contain complementary speaker-related information. We show that a concatenation of x-vectors and r-vectors can further improve the performance of the SV system. The best-performing embedding ensemble achieves an Equal Error Rate of 2.52% within the Voxceleb1 benchmark test, which is lower than other published results and obtained on the same dataset using the standard Voxceleb1 evaluation methodology.

ARCHITECTURAL AND MATHEMATICAL MODELING OF NETWORK STRUCTURE SYSTEMS USING S-HYPERNETWORK THEORY

The main concepts of the theory of S-hypernetworks are considered. It is shown that the language of this theory can be used to describe many systems of the network structure with sufficient accuracy to solve the given task. In fact, the analytical formulation of the tasks of analysis and synthesis of the structures of the specified systems is almost a simulation model that can be used to solve the tasks. Accordingly, S-hypernetworks can be named according to the mapping of elements. Obviously, in the same S-hypernetwork, different elements can be displayed simultaneously in different ways. Thus, the dictionary of the theory of hypernetworks can be increased due to the features of the display of hypernetwork elements on the surface and various relationships of the incident branches of the edges of secondary networks. The proposed classification of images will allow to set all kinds of tasks related to the description, analysis and synthesis of networks of various purposes. Generally speaking, as it seems at first glance, the most applications of the theory of hypernetworks and S-hypernetworks probably fall on telecommunications and transport. However, it is already clear that the theory of S-hypernetworks is applicable to the analysis and synthesis of many other network structure systems. Of particular interest are the tasks of analyzing inter-network structural interactions of complex systems of various nature. For ecto-hypernetworks, the study of topological properties, in particular, their flat implementation, is of practical importance. The tasks that arise closely overlap with the tasks of stacking graphs on oriented surfaces. However, the specificity of hypernetworks and, in particular, S - hypernetworks entails new topological tasks in the theory of S - hypernetworks. To understand topological tasks, it is necessary to consider different types of embeddings of secondary networks into primary ones. Examples of mapping graphs of secondary networks into the primary network are given, bearing in mind the practical use of the theory in the description and analysis of real systems of the network structure. Also, for practical purposes, an endo-hypernetwork was considered instead of an ecto-hypernetwork.

Embeddings in Natural Language Processing: Theory and Advances in Vector Representations of Meaning

Embeddings in Natural Language Processing: Theory and Advances in Vector Representations of Meaning

Decoding Word Embeddings with Brain-Based Semantic Features

Word embeddings are vectorial semantic representations built with either counting or predicting techniques aimed at capturing shades of meaning from word co-occurrences. Since their introduction, these representations have been criticized for lacking interpretable dimensions. This property of word embeddings limits our understanding of the semantic features they actually encode. Moreover, it contributes to the “black box” nature of the tasks in which they are used, since the reasons for word embedding performance often remain opaque to humans. In this contribution, we explore the semantic properties encoded in word embeddings by mapping them onto interpretable vectors, consisting of explicit and neurobiologically motivated semantic features (Binder et al. 2016). Our exploration takes into account different types of embeddings, including factorized count vectors and predict models (Skip-Gram, GloVe, etc.), as well as the most recent contextualized representations (i.e., ELMo and BERT).In our analysis, we first evaluate the quality of the mapping in a retrieval task, then we shed light on the semantic features that are better encoded in each embedding type. A large number of probing tasks is finally set to assess how the original and the mapped embeddings perform in discriminating semantic categories. For each probing task, we identify the most relevant semantic features and we show that there is a correlation between the embedding performance and how they encode those features. This study sets itself as a step forward in understanding which aspects of meaning are captured by vector spaces, by proposing a new and simple method to carve human-interpretable semantic representations from distributional vectors.

Extraction of Family History Information From Clinical Notes: Deep Learning and Heuristics Approach

BackgroundElectronic health records store large amounts of patient clinical data. Despite efforts to structure patient data, clinical notes containing rich patient information remain stored as free text, greatly limiting its exploitation. This includes family history, which is highly relevant for applications such as diagnosis and prognosis.ObjectiveThis study aims to develop automatic strategies for annotating family history information in clinical notes, focusing not only on the extraction of relevant entities such as family members and disease mentions but also on the extraction of relations between the identified entities.MethodsThis study extends a previous contribution for the 2019 track on family history extraction from national natural language processing clinical challenges by improving a previously developed rule-based engine, using deep learning (DL) approaches for the extraction of entities from clinical notes, and combining both approaches in a hybrid end-to-end system capable of successfully extracting family member and observation entities and the relations between those entities. Furthermore, this study analyzes the impact of factors such as the use of external resources and different types of embeddings in the performance of DL models.ResultsThe approaches developed were evaluated in a first task regarding entity extraction and in a second task concerning relation extraction. The proposed DL approach improved observation extraction, obtaining F1 scores of 0.8688 and 0.7907 in the training and test sets, respectively. However, DL approaches have limitations in the extraction of family members. The rule-based engine was adjusted to have higher generalizing capability and achieved family member extraction F1 scores of 0.8823 and 0.8092 in the training and test sets, respectively. The resulting hybrid system obtained F1 scores of 0.8743 and 0.7979 in the training and test sets, respectively. For the second task, the original evaluator was adjusted to perform a more exact evaluation than the original one, and the hybrid system obtained F1 scores of 0.6480 and 0.5082 in the training and test sets, respectively.ConclusionsWe evaluated the impact of several factors on the performance of DL models, and we present an end-to-end system for extracting family history information from clinical notes, which can help in the structuring and reuse of this type of information. The final hybrid solution is provided in a publicly available code repository.

Embeddings of the “New Massive Gravity”

Using different types of embeddings of equations of motion we investigate the existence of generalizations of the "New Massive Gravity" (NMG) model with the same particle content (massive gravitons). By using the Weyl symmetry as a guiding principle for the embeddings we show that the Noether gauge embedding approach leads us to a sixth order model in derivatives with either a massive or a massless ghost. If the Weyl symmetry is implemented by means of a Stueckelberg field we obtain a new scalar-tensor model for massive gravitons. It is ghost free and Weyl invariant at linearized level. The model can be nonlinearly completed into a scalar field coupled to the NMG theory. The elimination of the scalar field leads to a nonlocal modification of the NMG. We also prove to all orders in derivatives that there is no local, ghost free embedding of the linearized NMG equations of motion around Minkowski space when written in terms of one symmetric tensor. Regarding that point, NMG differs from the Fierz-Pauli theory, since in later case we can replace the Einstein-Hilbert action by specific $f(R,\Box\, R)$ generalizations and still keep the theory ghost free at linearized level.

Embedding methods for solving variational inequalities

Variational inequality problems (VIP) are an important class of mathematical problems that appear in many practical situations. So, it is important to find efficient and robust numerical solution methods. An appealing idea is to embed the VIP into a one-parametric problem which, then, can be solved numerically by a path-following method. In this article, we study two different types of embeddings and we analyse their generic properties. The non-linear complementarity problem and box-constrained VIP are discussed as special cases.

Innately transitive subgroups of wreath products in product action

A permutation group is innately transitive if it has a transitive minimal normal subgroup, which is referred to as a plinth. We study the class of finite, innately transitive permutation groups that can be embedded into wreath products in product action. This investigation is carried out by observing that such a wreath product preserves a natural Cartesian decomposition of the underlying set. Previously we classified the possible embeddings in the case where the plinth is simple. Here we extend that classification and identify several different types of Cartesian decompositions that can be preserved by an innately transitive group with a non-abelian plinth. These different types of decompositions lead to different types of embeddings of the acting group into wreath products in product action. We also obtain a full characterisation of embeddings of innately transitive groups with diagonal type into such wreath products.

Class two analogue of T. Y. Thomas's theorem and different types of embeddings of static spherically symmetric space-times

A Riemannian space of embedding class two is characterised by two symmetric tensors a ij , b ij and a vector si, satisfying the equations of Gauss, Codazzi and Ricci. It is proved that the Gauss equations together with one set of Codazzi equations imply the other set of Codazzi equations and the Ricci equations, provided that the matrix of the tensor b ij (or a ij ) is nonsingular. (The class m generalisation of the result has also been suggested). The result so proved has further been utilized in finding explicitly the a ij 's and b ij 's in the case of the static spherically symmetric line element. It is further indicated that the a ij 's and b ij 's so obtained are responsible for the different types of embeddings of the spacetime considered.