Types Of Embeddings Research Articles

Representing affect information in word embeddings

A growing body of research in natural language processing (NLP) and natural language understanding (NLU) is investigating human-like knowledge learned or encoded in the word embeddings from large language models. This is a step towards understanding what knowledge language models capture that resembles human understanding of language and communication. Here, we investigated whether and how the affect meaning of a word (i.e., valence, arousal, dominance) is encoded in word embeddings pre-trained in large neural networks. We used the human-labeled dataset (Mohammad 2018) as the ground truth and performed various correlational and classification tests on four types of word embeddings. The embeddings varied in being static or contextualized, and how much affect specific information was prioritized during the pre-training and fine-tuning phase. Our analyses show that word embedding from the vanilla BERT model (Devlin et al. 2019) did not saliently encode the affect information of English words. Only when the BERT model was fine-tuned on emotion related tasks or contained extra contextualized information from emotion-rich contexts could the corresponding embedding encode more relevant affect information.

Exploring Afrikaans word embeddings with analogies and nearest neighbours

This paper presents an exploration of word embeddings for Afrikaans using the analogies and nearest neighbours methodologies. We compare the results on three types of embeddings (fastText, FLAIR and GloVe) on a novel analogy data set for Afrikaans, inspired by the Bigger Analogy Test Set: BATS (Gladkova et al. 2016). Our analysis shows that for Afrikaans, similar to English, the types of embeddings influence the quality of analogies found for different linguistic tasks. Our investigation also demonstrates, however, that these Afrikaans embeddings do not encode as clear a linguistic representation as with English embeddings. The exact reason for this is subject to future work, but the added morphological complexity and the lack of data most likely play a role.

On the maximizing problem associated with critical Sobolev inequality under inhomogeneous constraints

In this paper, we study a maximizing problem associated with the critical Sobolev inequality under inhomogeneous constraints. The problem of this type was previously studied by Ishiwata and Wadade [On the effect of equivalent constrain on a maximizing problem associated with the Sobolev type embedding in . Math Ann. 2016;364:1043–1068; On the maximizing problem associated with Sobolev type embeddings under inhomogeneous constraints. Appl Anal. 2019;98(10):1916–1934], Ishiwata [On variational problems associated with Sobolev type inequalities and related topics. Available from: http://www.rism.it/doc/Ishiwata.pdf] and Nguyen [Maximizers for the variational problems associated with Sobolev type inequalities under constraints. Math Ann. 2018;372(1–2):229–255]. Our results give a complete picture of the effect of the constraints on the attainability and non-attainability of the problem. The sharp Sobolev inequality plays a crucial role in our argument. Our method also provides a new and simple proof for the recent results of Ishiwata and Wadade [On the maximizing problem associated with Sobolev type embeddings under inhomogeneous constraints. Appl Anal. 2019;98(10):1916–1934] concerning the sub-critical Sobolev-type inequalities (or Gagliardo–Nirenberg inequalities).

ScGMAAE: Gaussian mixture adversarial autoencoders for diversification analysis of scRNA-seq data.

The progress of single-cell RNA sequencing (scRNA-seq) has led to a large number of scRNA-seq data, which are widely used in biomedical research. The noise in the raw data and tens of thousands of genes pose a challenge to capture the real structure and effective information of scRNA-seq data. Most of the existing single-cell analysis methods assume that the low-dimensional embedding of the raw data belongs to a Gaussian distribution or a low-dimensional nonlinear space without any prior information, which limits the flexibility and controllability of the model to a great extent. In addition, many existing methods need high computational cost, which makes them difficult to be used to deal with large-scale datasets. Here, we design and develop a depth generation model named Gaussian mixture adversarial autoencoders (scGMAAE), assuming that the low-dimensional embedding of different types of cells follows different Gaussian distributions, integrating Bayesian variational inference and adversarial training, as to give the interpretable latent representation of complex data and discover the statistical distribution of different types of cells. The scGMAAE is provided with good controllability, interpretability and scalability. Therefore, it can process large-scale datasets in a short time and give competitive results. scGMAAE outperforms existing methods in several ways, including dimensionality reduction visualization, cell clustering, differential expression analysis and batch effect removal. Importantly, compared with most deep learning methods, scGMAAE requires less iterations to generate the best results.

On the role of membrane embedding, protein rigidity and transmembrane length in lipid membrane fusion.

The fusion of biological membranes is ubiquitous in natural processes like exo- and endocytosis, intracellular trafficking and viral entry. Membrane fusion is also utilized in artificial biomimetic fusion systems, e.g. for drug delivery. Both the natural and the biomimetic fusion systems rely on a wide range of (artificial) proteins mediating the fusion process. Although the exact mechanisms of these proteins differ, clear analogies in their general behavior can be observed in bringing the membranes in close proximity and mediating the fusion reaction. In our study, we use molecular dynamics simulations with coarse grained models, mimicking the general behavior of fusion proteins (spikes), to systematically examine the effects of specific characteristics of these proteins on the fusion process. The protein characteristics considered are (i) the type of membrane embedding, i.e., either transmembrane or not, (ii) the rigidity, and (iii) the transmembrane domain (TMD) length. The results show essential differences in fusion pathway between monotopic and transmembrane spikes, in which transmembrane spikes seem to inhibit the formation of hemifusion diaphragms, leading to a faster fusion development. Furthermore, we observed that an increased rigidity and a decreased TMD length both proved to contribute to a faster fusion development. Finally, we show that a single spike may suffice to successfully induce a fusion reaction, provided that the spike is sufficiently rigid and attractive. Not only does this shed light on biological fusion of membranes, it also provides clear design rules for artificial membrane fusion systems.

ARCHITECTURAL AND MATHEMATICAL MODELING OF NETWORK STRUCTURE SYSTEMS USING S-HYPERNETWORK THEORY

The main concepts of the theory of S-hypernetworks are considered. It is shown that the language of this theory can be used to describe many systems of the network structure with sufficient accuracy to solve the given task. In fact, the analytical formulation of the tasks of analysis and synthesis of the structures of the specified systems is almost a simulation model that can be used to solve the tasks. Accordingly, S-hypernetworks can be named according to the mapping of elements. Obviously, in the same S-hypernetwork, different elements can be displayed simultaneously in different ways. Thus, the dictionary of the theory of hypernetworks can be increased due to the features of the display of hypernetwork elements on the surface and various relationships of the incident branches of the edges of secondary networks. The proposed classification of images will allow to set all kinds of tasks related to the description, analysis and synthesis of networks of various purposes. Generally speaking, as it seems at first glance, the most applications of the theory of hypernetworks and S-hypernetworks probably fall on telecommunications and transport. However, it is already clear that the theory of S-hypernetworks is applicable to the analysis and synthesis of many other network structure systems. Of particular interest are the tasks of analyzing inter-network structural interactions of complex systems of various nature. For ecto-hypernetworks, the study of topological properties, in particular, their flat implementation, is of practical importance. The tasks that arise closely overlap with the tasks of stacking graphs on oriented surfaces. However, the specificity of hypernetworks and, in particular, S - hypernetworks entails new topological tasks in the theory of S - hypernetworks. To understand topological tasks, it is necessary to consider different types of embeddings of secondary networks into primary ones. Examples of mapping graphs of secondary networks into the primary network are given, bearing in mind the practical use of the theory in the description and analysis of real systems of the network structure. Also, for practical purposes, an endo-hypernetwork was considered instead of an ecto-hypernetwork.

A Survey of Text Representation and Embedding Techniques in NLP

Natural Language Processing (NLP) is a research field where a language in consideration is processed to understand its syntactic, semantic, and sentimental aspects. The advancement in the NLP area has helped solve problems in the domains such as Neural Machine Translation, Name Entity Recognition, Sentiment Analysis, and Chatbots, to name a few. The topic of NLP broadly consists of two main parts: the representation of the input text (raw data) into numerical format (vectors or matrix) and the design of models for processing the numerical data. This paper focuses on the former part and surveys how the NLP field has evolved from rule-based, statistical to more context-sensitive learned representations. For each embedding type, we list their representation, issues they addressed, limitations, and applications. This survey covers the history of text representations from the 1970s and onwards, from regular expressions to the latest vector representations used to encode the raw text data. It demonstrates how the NLP field progressed from where it could comprehend just bits and pieces to all the significant aspects of the text over time.

Nonautonomous dynamics: classification, invariants, and implementation

The work is a brief review of the results obtained in nonautonomous dynamics based on the concept of uniform equivalence of nonautonomous systems. This approach to the study of nonautonomous systems was proposed in [10] and further developed in the works of the second author, and recently - jointly by both authors. Such an approach seems to be fruitful and promising, since it allows one to develop a nonautonomous analogue of the theory of dynamical systems for the indicated classes of systems and give a classi cation of some natural classes of nonautonomous systems using combinatorial type invariants. We show this for classes of nonautonomous gradient-like vector elds on closed manifolds of dimensions one, two, and three. In the latter case, a new equivalence invariant appears, the wild embedding type for stable and unstable manifolds [14,17], as shown in a recent paper by the authors [5].

Increase in body temperature in pediatric patients after costal cartilage harvest in microtia reconstruction: A retrospective observational study.

Background:Previous evidence has clearly shown that maintaining normothermia in children undergoing surgery is difficult and is associated with adverse outcomes. Therefore, this study aimed to retrospectively analyze the changes in body temperature over time in 2 different types of microtia reconstruction surgeries, namely, embedding, and elevation surgeries.Methods:We performed a retrospective chart review of patients who underwent microtia reconstruction (embedding and elevation) between July 2012 and February 2015 (n = 38). The changes in body temperature between the 2 types of surgeries were compared.Results:During microtia reconstruction, the body temperature in the embedding surgery group was significantly higher than that in the elevation surgery group from 1 hour after the start of surgery to 1 day after the surgery (P < .001). Time, group, and time-group interaction were associated with an increase in body temperature (P < .001) but not the warming method.Conclusion:We found an increase in body temperature in patients with microtia who underwent embedding surgery (autologous costal cartilage harvest surgery), and this was related to the type of surgery and not to the warming method. Therefore, further research is warranted to determine the cause of the increase in body temperature during this surgery.

A synchronized multimodal neuroimaging dataset for studying brain language processing

We present a synchronized multimodal neuroimaging dataset for studying brain language processing (SMN4Lang) that contains functional magnetic resonance imaging (fMRI) and magnetoencephalography (MEG) data on the same 12 healthy volunteers while the volunteers listened to 6 hours of naturalistic stories, as well as high-resolution structural (T1, T2), diffusion MRI and resting-state fMRI data for each participant. We also provide rich linguistic annotations for the stimuli, including word frequencies, syntactic tree structures, time-aligned characters and words, and various types of word and character embeddings. Quality assessment indicators verify that this is a high-quality neuroimaging dataset. Such synchronized data is separately collected by the same group of participants first listening to story materials in fMRI and then in MEG which are well suited to studying the dynamic processing of language comprehension, such as the time and location of different linguistic features encoded in the brain. In addition, this dataset, comprising a large vocabulary from stories with various topics, can serve as a brain benchmark to evaluate and improve computational language models.

Classification of Articles from Mass Media by Categories and Relevance of the Subject Area

The research is devoted to classification of news articles about P. G. Demidov Yaroslavl State University (YarSU) into 4 categories: “society”, “education”, “science and technologies”, “not relevant”.The proposed approaches are based on using the BERT neural network and methods of machine learning: SVM, Logistic Regression, K-Neighbors, Random Forest, in combination of different embedding types: Word2Vec, FastText, TF-IDF, GPT-3. Also approaches of text preprocessing are considered to achieve higher quality of the classification. The experiments showed that the SVM classifier with TF-IDF embedding and trained on full article texts with titles achieved the best result. Its micro-F-measure and macro-F-measure are 0.8214 and 0.8308 respectively. The BERT neural network trained on fragments of paragraphs with YarSU mentions, from which the first 128 words and the last 384 words were taken, showed comparable results. The resulting micro-F-measure and macro-F-measure are 0.8304 and 0.8181 respectively. Thus, using paragraphs with the target organisation mentions is enough to classify text by categories efficiently.

Comparing Channel Restrictions of Communicating State Machines, High-level Message Sequence Charts, and Multiparty Session Types

Communicating state machines provide a formal foundation for distributed computation. Unfortunately, they are Turing-complete and, thus, challenging to analyse. In this paper, we classify restrictions on channels which have been proposed to work around the undecidability of verification questions. We compare half-duplex communication, existential B-boundedness, and k-synchronisability. These restrictions do not prevent the communication channels from growing arbitrarily large but still restrict the power of the model. Each restriction gives rise to a set of languages so, for every pair of restrictions, we check whether one subsumes the other or if they are incomparable. We investigate their relationship in two different contexts: first, the one of communicating state machines, and, second, the one of communication protocol specifications using high-level message sequence charts. Surprisingly, these two contexts yield different conclusions. In addition, we integrate multiparty session types, another approach to specify communication protocols, into our classification. We show that multiparty session type languages are half-duplex, existentially 1-bounded, and 1-synchronisable. To show this result, we provide the first formal embedding of multiparty session types into high-level message sequence charts.

HIRE: Distilling high-order relational knowledge from heterogeneous graph neural networks

Researchers have recently proposed plenty of heterogeneous graph neural networks (HGNNs) due to the ubiquity of heterogeneous graphs in both academic and industrial areas. Instead of pursuing a more powerful HGNN model, in this paper, we are interested in devising a versatile plug-and-play module, which accounts for distilling relational knowledge from pre-trained HGNNs. To the best of our knowledge, we are the first to propose a HIgh-order RElational (HIRE) knowledge distillation framework on heterogeneous graphs, which can significantly boost the prediction performance regardless of model architectures of HGNNs. Concretely, our HIRE framework initially performs first-order node-level knowledge distillation, which encodes the semantics of the teacher HGNN with its prediction logits. Meanwhile, the second-order relation-level knowledge distillation imitates the relational correlation between node embeddings of different types generated by the teacher HGNN. Extensive experiments on various popular HGNNs models and three real-world heterogeneous graphs demonstrate that our method obtains consistent and considerable performance enhancement, proving its effectiveness and generalization ability.

Transition Information Enhanced Disentangled Graph Neural Networks for session-based recommendation

Session-based recommendation (SBR) is a practical task that predicts the next item based on an anonymous behavior sequence. Most of current methods employ graph neural network to model neighboring item transition information from global and local contexts (i.e, other and current sessions). However, they treat neighbors from other sessions equally without considering its items, which may have different contributions with the target item on varied aspects. In other words, they have not explored finer-granularity transition information in the global context, leading to sub-optimal performance. This paper fills this gap by proposing a novel method called Transition Information Enhanced Disentangled Graph Neural Network (TIE-DGNN) to capture finer-granular transition information between items and try to interpret the transition reason by modeling various factors of items. Specifically, we first propose a position-aware global graph to model neighboring item transition in the global context. Then, we slice item embeddings into blocks, each of which represents a factor, and use global-level disentangling layers to separately learn factor embeddings. Meanwhile, we train local-level item embeddings by using attention mechanisms to capture transition information from the current session. Further, inter-session and intra-session embeddings are generated by two types of item embeddings, respectively. Finally, we use contrastive learning techniques to enhance the robustness of two session embeddings. To this end, our model considers two levels of transition information. Especially in global context, it not only consider finer-granularity transition information between items but also take user intents at factor-level into account to interpret the key reason for the transition. Extensive experiments on three benchmark datasets demonstrate its superiority over state-of-the-art methods.

Delivering Blue-Green Infrastructure: Innovation Pathways for Integrating Multiple Values

Realizing a multifunctional blue-green infrastructure (BGI) as a nature-based solution for the urban water system and built environment within crowded city areas is seen as a promising route for the process of climate adaptation. BGI projects like rain gardens, green roofs, and water squares can be combined to achieve a variety of technical (drainage), environmental (biodiversity), economic (property development) and social (health and wellbeing) goals and values at a local neighborhood level. As integrating such values within local governments' existing fragmented structures and procedures has proved to be challenging, urban governments are increasingly experimenting with innovative governance approaches at different levels to capitalize on the multiple benefits of BGI. Nevertheless, policy actors who try to justify their choices in the face of value conflicts are both constrained and enabled by the institutions they can call on. Using a qualitative comparative case study, this article therefore aims to gain insight into different ways of, or approaches to, organizing value integration. In particular, we compare: (1) a top-down case of programmatic steering to translate value integration into a neighborhood approach; (2) a market-oriented innovative procurement approach to local public-private partnership projects; and (3) a case of invitational governance for a future-proof neighborhood that is striving for a sense of citizen ownership. Our findings demonstrate the conditions, drivers, and barriers to the value integration of different governance innovations in relation to time-related issues, the types of support available, organizational embedding, and stakeholder involvement. Our specific focus is on understanding how social and sustainability and spatial and technical values are integrated. This paper thus helps us to get to grips with different pathways to value integration in the context of urban infrastructures, as well as their applicability and the conditions for success. These insights will enable the further strengthening of our capacity to build climate-proof cities in a value-driven and integrative manner.

Reading the Forum on China–Africa Cooperation (2000-2021): geoeconomics, governance, and embedding ‘creative involvement’

ABSTRACT How China can promote its overseas commercial and strategic interests and protect its citizens and investments while continuing to adhere to a longstanding commitment to non-interference is a subject of intense policy and academic debate. A notion of ‘creative involvement’ has been developed to specify an increasingly flexible and pragmatic approach to projecting and managing China’s expanding global interests. In this research a close reading of the Declarations and Action Plans of the Forum on China–Africa Cooperation (FOCAC) over the period 2000–21 reveals specific degrees and types of embedding of Chinese involvement in Africa’s complex political economic landscape and an evolving reorientation of development cooperation policy practices shaped by growing concerns with African economic policy and local governance issues.

Concept of Placement of Fiber-Optic Sensor in Smart Energy Transport Cable under Tensile Loading.

Due to the exponential growth in offshore renewable energies and structures such as floating offshore wind turbines and wave power converters, the research and engineering in this field is experiencing exceptional development. This emergence of offshore renewable energy requires power cables which are usually made up of copper to transport this energy ashore. These power cables are critical structures that must withstand harsh environmental conditions, handling, and shipping, at high seas which can cause copper wires to deform well above the limit of proportionality and consequently break. Copper, being an excellent electric conductor, has, however, very weak mechanical properties. If plasticity propagates inside copper not only will the mechanical properties be affected, but the electrical properties are also disrupted. Constantly monitoring such large-scale structures can be carried out by providing continuous strain using fiber-optic sensors (FOSs). The embedding of optical fibers within the cables (not within the phase) is practiced. Nevertheless, these optical fibers are first introduced into a cylinder of larger diameter than the optical fiber before this same fiber is embedded within the insulator surrounding the phases. Therefore, this type of embedding can in no way give a precise idea of the true deformation of the copper wires inside the phase. In this article, a set of numerical simulations are carried-out on a single phase (we are not yet working on the whole cable) with the aim of conceptualizing the placement of FOSs that will monitor strain and temperature within the conductor. It is well known that copper wire must never exceed temperatures above 90 °C, as this will result in shutdown of the whole system and therefore result in heavy maintenance, which would be a real catastrophe, economically speaking. This research explores the option of embedding sensors in several areas of the phase and how this can enable obtaining strain values that are representative of what really is happening in the conductor. It is, therefore, the primary objective of the current preliminary model to try to prove that the principle of embedding sensors in between copper wires can be envisaged, in particular to obtain an accurate idea about strain tensor of helical ones (multi-parameter strain sensing). The challenge is to ensure that they are not plastically deformed and hence able to transport electricity without exceeding or even becoming closer to 90 °C (fear of shutdown). The research solely focuses on mechanical aspects of the sensors. There are certainly some others, pertaining to sensors physics, instrumentation, and engineering, that are of prime importance, too. The upstream strategy of this research is to come up with a general concept that can be refined later by including, step by step, all the aspects listed above.

On the global behaviors for defocusing semilinear wave equations in ℝ1+2

In this paper, we study the asymptotic decay properties for defocusing semilinear wave equations in $\mathbb{R}^{1+2}$ with pure power nonlinearity. By applying new vector fields to null hyperplane, we derive improved time decay of the potential energy, with a consequence that the solution scatters both in the critical Sobolev space and energy space for all $p>1+\sqrt{8}$. Moreover combined with Br\'{e}zis-Gallouet-Wainger type of logarithmic Sobolev embedding, we show that the solution decays pointwise with sharp rate $t^{-\frac{1}{2}}$ when $p>\frac{11}{3}$ and with rate $t^{ -\frac{p-1}{8}+\epsilon }$ for all $1<p\leq \frac{11}{3}$. This in particular implies that the solution scatters in energy space when $p>2\sqrt{5}-1$.

Automatic Rail Component Detection Based on AttnConv-Net

The automatic detection of major rail components using railway images is beneficial to ensure the rail transport safety. In this paper, we propose an attention-powered deep convolutional network (AttnConv-net) to detect multiple rail components including the rail, clips, and bolts. The proposed method consists of a deep convolutional neural network (DCNN) as the backbone, cascading attention blocks (CAB), and two feed forward networks (FFN). Two types of positional embedding are applied to enrich information in latent features extracted from the backbone. Based on processed latent features, the CAB aims to learn the local context of rail components including their categories and component boundaries. Final categories and bounding boxes are generated via two FFN implemented in parallel. To enhance the detection of small components, various data augmentation methods are employed in the training process. The effectiveness of the proposed AttnConv-net is validated with one real dataset and another synthesized dataset. Compared with classic convolutional neural network based methods, our proposed method simplifies the detection pipeline by eliminating the need of prior- and post-processing, which offers a new speed-quality solution to enable faster and more accurate image-based rail component detections

A System for Aligning Geographical Entities from Large Heterogeneous Sources

Aligning points of interest (POIs) from heterogeneous geographical data sources is an important task that helps extend map data with information from different datasets. This task poses several challenges, including differences in type hierarchies, labels (different formats, languages, and levels of detail), and deviations in the coordinates. Scalability is another major issue, as global-scale datasets may have tens or hundreds of millions of entities. In this paper, we propose the GeographicaL Entities AligNment (GLEAN) system for efficiently matching large geographical datasets based on spatial partitioning with an adaptable margin. In particular, we introduce a text similarity measure based on the local-context relevance of tokens used in combination with sentence embeddings. We then come up with a scalable type embedding model. Finally, we demonstrate that our proposed system can efficiently handle the alignment of large datasets while improving the quality of alignments using the proposed entity similarity measure.