General Embedding Research Articles

Anti-Leibniz algebras: A non-commutative version of mock-Lie algebras

Leibniz algebras are non skew-symmetric generalization of Lie algebras. In this paper we introduce the notion of anti-Leibniz algebras as a “non commutative version” of mock-Lie algebras. Low dimensional classification of such algebras is given. Then we investigate the notion of averaging operators and more general embedding tensors to build some new algebraic structures, namely anti-associative dialgebras, anti-associative trialgebras and anti-Leibniz trialgebras.

MARML: Motif-Aware Deep Representation Learning in Multilayer Networks.

The rapid increase in high-throughput, complex, and heterogeneous data has led to the adoption of network-structured models and analyses for interpretation. However, these data are inherently complex and challenging to understand, prompting researchers to turn to graph embedding methods to facilitate analysis. While general network embedding techniques have shown promise in improving downstream prediction and classification tasks, real-world data are complicated due to cross-domain interactions between different types of entities. Multilayered networks have been successful in integrating biological data to represent biological systems' hierarchy, but embedding nodes based on different types of interactions remains an unsolved problem. To address this challenge, we propose the Motif-aware deep representation learning in multilayer (MARML) networks for learning network representations. Our method considers recurring motif patterns, topological information, and attributive information from other sources as node features. We validated the MARML method using various multilayer network datasets. In addition, by incorporating motif information, MARML considers higher order connections across different hierarchies. The learned features exhibited excellent accuracy in tasks related to link prediction and link differentiation, enabling us to distinguish between existing and disconnected triplets. Through the integration of both intrinsic node attributes and topological network structures, we enhance our understanding of complex biological systems.

Learning General and Specific Embedding with Transformer for Few-Shot Object Detection

AbstractFew-shot object detection (FSOD) studies how to detect novel objects with few annotated examples effectively. Recently, it has been demonstrated that decent feature embeddings, including the general feature embeddings that are more invariant to visual changes and the specific feature embeddings that are more discriminative for different object classes, are both important for FSOD. However, current methods lack appropriate mechanisms to sensibly cooperate both types of feature embeddings based on their importance to detecting objects of novel classes, which may result in sub-optimal performance. In this paper, to achieve more effective FSOD, we attempt to explicitly encode both general and specific feature embeddings using learnable tensors and apply a Transformer to help better incorporate them in FSOD according to their relations to the input object features. We thus propose a Transformer-based general and specific embedding learning (T-GSEL) method for FSOD. In T-GSEL, learnable tensors are employed in a three-stage pipeline, encoding feature embeddings in general level, intermediate level, and specific level, respectively. In each stage, we apply a Transformer to first model the relations of the corresponding embedding to input object features and then apply the estimated relations to refine the input features. Meanwhile, we further introduce cross-stage connections between embeddings of different stages to make them complement and cooperate with each other, delivering general, intermediate, and specific feature embeddings stage by stage and utilizing them together for feature refinement in FSOD. In practice, a T-GSEL module is easy to inject. Extensive empirical results further show that our proposed T-GSEL method achieves compelling FSOD performance on both PASCAL VOC and MS COCO datasets compared with other state-of-the-art approaches.

Embedding of Markov matrices for d⩽4\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\varvec{d \\leqslant 4}$$\\end{document}

The embedding problem of Markov matrices in Markov semigroups is a classic problem that regained a lot of impetus and activities through recent needs in phylogeny and population genetics. Here, we give an account for dimensions d⩽4\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$d\\leqslant 4$$\\end{document}, including a complete and simplified treatment of the case d=3\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$d=3$$\\end{document}, and derive the results in a systematic fashion, with an eye on the potential applications. Further, we reconsider the setup of the corresponding problem for time-inhomogeneous Markov chains, which is needed for real-world applications because transition rates need not be constant over time. Additional cases of this more general embedding occur for any d⩾3\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$d\\geqslant 3$$\\end{document}. We review the known case of d=3\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$d=3$$\\end{document} and describe the setting for future work on d=4\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$d=4$$\\end{document}.

Standard form of master equations for general non-Markovian jump processes: The Laplace-space embedding framework and asymptotic solution

We present a standard form of master equations (MEs) for general one-dimensional non-Markovian (history-dependent) jump processes, complemented by an asymptotic solution derived from an expanded system-size approach. The ME is obtained by developing a general Markovian embedding using a suitable set of auxiliary field variables. This Markovian embedding uses a Laplace-convolution operation applied to the velocity trajectory. We introduce an asymptotic method tailored for this ME standard, generalizing the system-size expansion for these jump processes. Under specific stability conditions tied to a single noise source, upon coarse graining, the generalized Langevin equation (GLE) emerges as a universal approximate model for point processes in the weak-coupling limit. This methodology offers a unified analytical tool set for general non-Markovian processes, reinforcing the universal applicability of the GLE founded in microdynamics and the principles of statistical physics. Published by the American Physical Society 2024

Advancing language models through domain knowledge integration: a comprehensive approach to training, evaluation, and optimization of social scientific neural word embeddings

This article proposes a comprehensive strategy for training, evaluating, and optimizing domain-specific word2vec-based word embeddings, using social science literature as an example. Our primary objectives are: (1) to train the embeddings utilizing a corpus of social science text, (2) to test their performance against domain-unspecific embeddings using our developed intrinsic and extrinsic evaluation strategy, and (3) to enhance their performance even further by using domain knowledge. As an integral part of this approach, we present SociRel-461, a domain-knowledge dictionary designed for the intrinsic evaluation and subsequent refinement of social science word embeddings. Using a dataset of 100,000 full-text scientific articles in sociology, we train multiple vector space models, which we then benchmark against a larger, pre-trained general language embedding model as part of our extrinsic evaluation. Furthermore, we developed a transfer learning multi-label classification task for extrinsic evaluation. Our findings reveal that domain-specific embeddings outperform their domain-unspecific counterparts in both intrinsic and extrinsic evaluations. We also investigated the retrofitting post-processing method to enhance domain-unspecific embeddings with the domain knowledge embedded in SociRel-461. While retrofitting does not enhance our domain-specific vector space models, it significantly improves the performance of the domain-unspecific embeddings. This highlights the potential of retrofitting for the transfer of domain knowledge to domain-unspecific embeddings. Our results emphasize the importance of utilizing domain-specific word embeddings for better performance in domain specific transfer learning tasks, as they outperform conventional embeddings trained on everyday language.

Bridging dimensions: General embedding algorithm and field-theory reconstruction in 5D braneworld models

Bridging dimensions: General embedding algorithm and field-theory reconstruction in 5D braneworld models

A new sentence embedding framework for the education and professional training domain with application to hierarchical multi-label text classification

In recent years, Natural Language Processing (NLP) has made significant advances through advanced general language embeddings, allowing breakthroughs in NLP tasks such as semantic similarity and text classification. However, complexity increases with hierarchical multi-label classification (HMC), where a single entity can belong to several hierarchically organized classes. In such complex situations, applied on specific-domain texts, such as the Education and professional training domain, general language embedding models often inadequately represent the unique terminologies and contextual nuances of a specialized domain. To tackle this problem, we present HMCCCProbT, a novel hierarchical multi-label text classification approach. This innovative framework chains multiple classifiers, where each individual classifier is built using a novel sentence-embedding method BERTEPro based on existing Transformer models, whose pre-training has been extended on education and professional training texts, before being fine-tuned on several NLP tasks. Each individual classifier is responsible for the predictions of a given hierarchical level and propagates local probability predictions augmented with the input feature vectors to the classifier in charge of the subsequent level. HMCCCProbT tackles issues of model scalability and semantic interpretation, offering a powerful solution to the challenges of domain-specific hierarchical multi-label classification. Experiments over three domain-specific textual HMC datasets indicate the effectiveness of HMCCCProbT to compare favorably to state-of-the-art HMC algorithms in terms of classification accuracy and also the ability of BERTEPro to obtain better probability predictions, well suited to HMCCCProbT, than three other vector representation techniques.

Context-Aware Dynamic Word Embeddings for Aspect Term Extraction

The aspect term extraction (ATE) task aims to extract aspect terms describing a part or an attribute of a product from review sentences. Most existing works rely on either general or domain embedding to address this problem. Despite the promising results, the importance of general and domain embeddings is still ignored by most methods, resulting in degraded performances. Besides, word embedding is also related to downstream tasks, and how to regularize word embeddings to capture context-aware information is an unresolved problem. To solve these issues, we first propose context-aware dynamic word embedding (CDWE), which could simultaneously consider general meanings, domain-specific meanings, and the context information of words. Based on CDWE, we propose an attention-based convolution neural network, called ADWE-CNN for ATE, which could adaptively capture the previous meanings of words by utilizing an attention mechanism to assign different importance to the respective embeddings. The experimental results show that ADWE-CNN achieves a comparable performance with the state-of-the-art approaches. Various ablation studies have been conducted to explore the benefit of each component. Our code is publicly available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">http://github.com/xiejiajia2018/ADWE-CNN</uri> .

Infrared and Visible Image Fusion via General Feature Embedding From CLIP and DINOv2

Infrared and Visible Image Fusion via General Feature Embedding From CLIP and DINOv2

GroomGen: A High-Quality Generative Hair Model Using Hierarchical Latent Representations

Despite recent successes in hair acquisition that fits a high-dimensional hair model to a specific input subject, generative hair models, which establish general embedding spaces for encoding, editing, and sampling diverse hairstyles, are way less explored. In this paper, we present GroomGen , the first generative model designed for hair geometry composed of highly-detailed dense strands. Our approach is motivated by two key ideas. First, we construct hair latent spaces covering both individual strands and hairstyles. The latent spaces are compact, expressive, and well-constrained for high-quality and diverse sampling. Second, we adopt a hierarchical hair representation that parameterizes a complete hair model to three levels: single strands, sparse guide hairs, and complete dense hairs. This representation is critical to the compactness of latent spaces, the robustness of training, and the efficiency of inference. Based on this hierarchical latent representation, our proposed pipeline consists of a strand-VAE and a hairstyle-VAE that encode an individual strand and a set of guide hairs to their respective latent spaces, and a hybrid densification step that populates sparse guide hairs to a dense hair model. GroomGen not only enables novel hairstyle sampling and plausible hairstyle interpolation, but also supports interactive editing of complex hairstyles, or can serve as strong data-driven prior for hairstyle reconstruction from images. We demonstrate the superiority of our approach with qualitative examples of diverse sampled hairstyles and quantitative evaluation of generation quality regarding every single component and the entire pipeline.

Out-of-vocabulary word embedding learning based on reading comprehension mechanism

Currently, most natural language processing tasks use word embeddings as the representation of words. However, when encountering out-of-vocabulary (OOV) words, the performance of downstream models that use word embeddings as input is often quite limited. To solve this problem, the latest methods mainly infer the meaning of OOV words based on two types of information sources: the morphological structure of OOV words and the contexts in which they appear. However, the low frequency of OOV words themselves usually makes them difficult to learn in pre-training tasks by general word embedding models. In addition, this characteristic of OOV word embedding learning also brings the problem of context scarcity. Therefore, we introduce the concept of “similar contexts” based on the classical “distributed hypothesis” in linguistics, by borrowing from the human reading comprehension mechanisms to make up for the deficiency of insufficient contexts in previous OOV word embedding learning work. The experimental results show that our model achieved the highest relative scores in both intrinsic and extrinsic evaluation tasks, which demonstrates the positive effect of the “similar contexts” introduced in our model on OOV word embedding learning.

Relatedness in the era of machine learning

Relatedness is a quantification of how much two human activities are similar in terms of the inputs and contexts needed for their development. Under the idea that it is easier to move between related activities than towards unrelated ones, empirical approaches to quantify relatedness are currently used as predictive tools to inform policies and development strategies in governments, international organizations, and firms. Here we show that the standard, widespread approach of estimating Relatedness through the co-location of activities (e.g. Product Space) generates a measure of relatedness that performs worse than trivial auto-correlation prediction strategies. In this paper, working on data about countries’ trade, technologies, and scientific production, we show two main findings. First, we find that a shift from two-product correlations (network-density based) to many-product correlations (decision trees) can dramatically improve the quality of forecasts, allowing the possibility to assist policymakers in optimizing decisions to promote growth. Then, we propose a new methodology to empirically estimate Relatedness that we call Continuous Projection Space (CPS). CPS, which represents a general network embedding technique, vastly outperforms all the co-location, network-based approaches, while retaining similar interpretability in terms of pairwise distances. Depending on the dataset the best approach is always either CPS or machine learning algorithms based on decision trees.

Task‐oriented feature hallucination for few‐shot image classification

AbstractData hallucination generates additional training examples for novel classes to alleviate the data scarcity problem in few‐shot learning (FSL). Existing hallucination‐based FSL methods normally train a general embedding model first by applying information extracted from base classes that have abundant data. In those methods, hallucinators are then built upon the trained embedding model to generate data for novel classes. However, these hallucination methods usually rely on general‐purpose embeddings, limiting their ability to generate task‐oriented samples for novel classes. Recent studies have shown that task‐specific embedding models, which are adapted to novel tasks, can achieve better classification performance. To improve the performance of example hallucination for tasks, a task‐oriented embedding model is used in the proposed method to perform task‐oriented generation. After the initialization, the hallucinator is finetuned by applying a task‐oriented embedding model with the guidance of a teacher–student mechanism. The proposed task‐oriented hallucination method contains two steps. An initial embedding network and an initial hallucinator are trained with a base dataset in the first step. The second step contains a pseudo‐labelling process where the base dataset is pseudo‐labelled using support data of the few‐shot task and a task‐oriented fine‐tuning process where the embedding network and hallucinator are adjusted simultaneously. Both the embedding network and the hallucinator are updated with the support set and the pseudo‐labelled base dataset using knowledge distillation. The experiments are conducted on four popular few‐shot datasets. The results demonstrate that the proposed approach outperforms state‐of‐the‐art methods with 0.8% to 4.08% increases in classification accuracy for 5‐way 5‐shot tasks. It also achieves comparable accuracy to state‐of‐the‐art methods for 5‐way 1‐shot tasks.

Adaptive search query generation and refinement in systematic literature review

Systematic literature reviews (SLRs) are a central part of evidence-based research, which involves collecting and integrating empirical evidence on specific research questions. A key step in this process is building Boolean search queries, which are at the core of information retrieval systems that support literature search. This involves turning general research aims into specific search terms that can be combined into complex Boolean expressions. Researchers must build and refine search queries to ensure they have sufficient coverage and properly represent the literature. In this paper, we propose an adaptive query generation and refinement pipeline for SLR search that uses reinforcement learning to learn the optimal modifications to a query based on feedback from researchers about its performance. Empirical evaluations with 10 SLR datasets showed our approach achieves comparable performance to queries manually composed by SLR authors. We also investigate the impact of design decisions on the performance of the query generation and refinement pipeline. Specifically, we study the effects of the type of input seed, the use of general versus domain-specific word embedding models, the sampling strategy for relevance feedback, and number of iterations in the refinement process. Our results provide insights into the effects of these choices on the pipeline’s performance.

Edge computing on TPU for brain implant signal analysis

The ever-increasing number of recording sites of silicon-based probes imposes a great challenge for detecting and evaluating single-unit activities in an accurate and efficient manner. Currently separate solutions are available for high precision offline evaluation and separate solutions for embedded systems where computational resources are more limited.We propose a deep learning-based spike sorting system, that utilizes both unsupervised and supervised paradigms to learn a general feature embedding space and detect neural activity in raw data as well as predict the feature vectors for sorting. The unsupervised component uses contrastive learning to extract features from individual waveforms, while the supervised component is based on the MobileNetV2 architecture. One of the key advantages of our system is that it can be trained on multiple, diverse datasets simultaneously, resulting in greater generalizability than previous deep learning-based models.We demonstrate that the proposed model does not only reaches the accuracy of current state-of-art offline spike sorting methods but has the unique potential to run on edge Tensor Processing Units (TPUs), specialized chips designed for artificial intelligence and edge computing. We compare our model performance with state of art solutions on paired datasets as well as on hybrid recordings as well. The herein demonstrated system paves the way to the integration of deep learning-based spike sorting algorithms into wearable electronic devices, which will be a crucial element of high-end brain–computer interfaces.

Contradiction Detection in Financial Reports

Finding and amending contradictions in a financial report is crucial for the publishing company and its financial auditors. To automate this process, we introduce a novel approach that incorporates informed pre-training into its transformer-based architecture to infuse this model with additional Part-Of-Speech knowledge. Furthermore, we fine-tune the model on the public Stanford Natural Language Inference Corpus and our proprietary financial contradiction dataset. It achieves an exceptional contradiction detection F1 score of 89.55% on our real-world financial contradiction dataset, beating our several baselines by a considerable margin. During the model selection process we also test various financial-document-specific transformer models and find that they underperform the more general embedding approaches.

Explicit time embedding based cascade attention network for information popularity prediction

Predicting information cascade popularity is a fundamental problem in social networks. Capturing temporal attributes and cascade role information (e.g., cascade graphs and cascade sequences) is necessary for understanding the information cascade. Current methods rarely focus on unifying this information for popularity predictions, which prevents them from effectively modeling the full properties of cascades to achieve satisfactory prediction performances. In this paper, we propose an explicit Time embedding based Cascade Attention Network (TCAN) as a novel popularity prediction architecture for large-scale information networks. TCAN integrates temporal attributes (i.e., periodicity, linearity, and non-linear scaling) into node features via a general time embedding approach (TE), and then employs a cascade graph attention encoder (CGAT) and a cascade sequence attention encoder (CSAT) to fully learn the representation of cascade graphs and cascade sequences. We use two real-world datasets (i.e., Weibo and APS) with tens of thousands of cascade samples to validate our methods. Experimental results show that TCAN obtains mean logarithm squared errors of 2.007 and 1.201 and running times of 1.76 h and 0.15 h on both datasets, respectively. Furthermore, TCAN outperforms other representative baselines by 10.4%, 3.8%, and 10.4% in terms of MSLE, MAE, and R-squared on average while maintaining good interpretability.

A Topic-Specific Web Crawler using Deep Convolutional Networks

This paper presented a new focused crawler that efficiently supports the Turkish language. The developed architecture was divided into multiple units: a control unit, crawler unit, link extractor unit, link sorter unit, and natural language processing unit. The crawler's units can work in parallel to process the massive amount of published websites. Also, the proposed Convolutional Neural Network (CNN) based natural language processing unit can professionally classifying Turkish text and web pages. Extensive experiments using three datasets have been performed to illustrate the performance of the developed approach. The first dataset contains 50,000 Turkish web pages downloaded by the developed crawler, while the other two are publicly available and consist of “28,567” and “22,431” Turkish web pages, respectively. In addition, the Vector Space Model (VSM) in general and word embedding state-of-the-art techniques, in particular, were investigated to find the most suitable one for the Turkish language. Overall, results indicated that the developed approach had achieved good performance, robustness, and stability when processing the Turkish language. Also, Bidirectional Encoder Representations from Transformer (BERT) was found to be the most appropriate embedding for building an efficient Turkish language classification system. Finally, our experiments showed superior performance of the developed natural language processing unit against seven state-of-the-art CNN classification systems. Where accuracy improvement compared to the second-best is 10% and 47% compared to the lowest performance.

Contrastive learning–based structure preserving projection for hyperspectral images

ABSTRACT Unsupervised feature extraction methods have been widely applied to remove the huge amount of redundancy in hyperspectral images due to their effectiveness when the label information of samples is unreachable. However, because of the lack of label information, unsupervised feature extraction methods are deficient in the discriminant ability compared to supervised methods. When the number of samples is small, the effect of dimension reduction is usually not good enough. To address the problems, an unsupervised structure preserving projection method named contrastive learning based sparsity preserving projection (CL-SPP) is proposed in this paper. Firstly, CL-SPP increases the discriminant ability of samples by introducing the concept of positive and negative pairs, and adjusts the number of positive and negative pairs in the training set through a parameter. Then, by minimizing the contrastive loss function, CL-SPP makes the positive pairs more similar and the negative pairs less similar after projection. Moreover, the proposed contrastive learning-based method is also extended to the supervised case, as well as a general graph embedding model framework based on comparative learning. Experiments on three hyperspectral images demonstrate that the proposed methods have a better performance than related approaches. More impressively, the effect of CL-SPP is comparable to its supervised version.