General Embeddings Research Articles

Pre-Training General Trajectory Embeddings With Maximum Multi-View Entropy Coding

Pre-Training General Trajectory Embeddings With Maximum Multi-View Entropy Coding

Learning General and Specific Embedding with Transformer for Few-Shot Object Detection

AbstractFew-shot object detection (FSOD) studies how to detect novel objects with few annotated examples effectively. Recently, it has been demonstrated that decent feature embeddings, including the general feature embeddings that are more invariant to visual changes and the specific feature embeddings that are more discriminative for different object classes, are both important for FSOD. However, current methods lack appropriate mechanisms to sensibly cooperate both types of feature embeddings based on their importance to detecting objects of novel classes, which may result in sub-optimal performance. In this paper, to achieve more effective FSOD, we attempt to explicitly encode both general and specific feature embeddings using learnable tensors and apply a Transformer to help better incorporate them in FSOD according to their relations to the input object features. We thus propose a Transformer-based general and specific embedding learning (T-GSEL) method for FSOD. In T-GSEL, learnable tensors are employed in a three-stage pipeline, encoding feature embeddings in general level, intermediate level, and specific level, respectively. In each stage, we apply a Transformer to first model the relations of the corresponding embedding to input object features and then apply the estimated relations to refine the input features. Meanwhile, we further introduce cross-stage connections between embeddings of different stages to make them complement and cooperate with each other, delivering general, intermediate, and specific feature embeddings stage by stage and utilizing them together for feature refinement in FSOD. In practice, a T-GSEL module is easy to inject. Extensive empirical results further show that our proposed T-GSEL method achieves compelling FSOD performance on both PASCAL VOC and MS COCO datasets compared with other state-of-the-art approaches.

Aspect-Aware Graph Attention Network for Heterogeneous Information Networks.

Graph Convolutional Networks (GCNs) derive inspiration from recent advances in computer vision, by stacking layers of first-order filters followed by a nonlinear activation function to learn entity or graph embeddings. Although GCNs have been shown to boost the performance of many network analysis tasks, they still face tremendous challenges in learning from Heterogeneous Information Networks (HINs), where relations play a decisive role in knowledge reasoning. What's more, there are multiaspect representations of entities in HINs, and a filter learned in one aspect do not necessarily apply to another. We address these challenges by proposing the Aspect-Aware Graph Attention Network (AGAT), a model that extends GCNs with alternative learnable filters to incorporate entity and relational information. Instead of focusing on learning the general entity embeddings, AGAT learns the adaptive entity embeddings based on prediction scenario. Experiments of link prediction and semi-supervised classification verify the effectiveness of our algorithm.

Visual representation learning using graph-based higher-order heuristic distillation for cell detection in blood smear images

Background and objectiveIn many real-world scenarios, including the blood smear domain, it is difficult for detection networks to achieve good performance because image annotation is usually time consuming and expensive. To address this issue, similarity-based distillation (SD) methods, considered the soft version of contrastive learning, are applied to learn a better visual representation without requiring any supervision of the downstream task. Motivated by our theoretical analysis, we treat standard SD methods as the maximization of common 1-hop neighboring key points between two queries in an attributed graph, where nodes represent query and key data points. However, such first-order graph heuristic methods that calculate the likelihood of an unseen link between target nodes by using up to 1-hop neighborhoods are normally limited by insufficient representation power and even lack of generalization ability. MethodsTherefore, in this paper, we propose a novel higher-order heuristic distillation (H2D) method that distills knowledge about more general and powerful higher-order heuristic features based on a more than 1-hop relationship in the attributed graph. To do this, we utilize a graph neural network model to learn the higher-order heuristic features on the attributed graph constructed by query and key data representations and transfer the knowledge from the teacher to the student encoder. ResultsOur method outperforms the previous state-of-the-art SD methods in the cell detection task on the blood smear dataset as well on open databases (Pascal VOC and MS COCO). Conclusions: Our proposed model allow teacher encoder to transfer the knowledge about more general and powerful higher-order heuristic embeddings to the student and enables better learning for visual representation on cell detection task using blood smear images.

A new sentence embedding framework for the education and professional training domain with application to hierarchical multi-label text classification

In recent years, Natural Language Processing (NLP) has made significant advances through advanced general language embeddings, allowing breakthroughs in NLP tasks such as semantic similarity and text classification. However, complexity increases with hierarchical multi-label classification (HMC), where a single entity can belong to several hierarchically organized classes. In such complex situations, applied on specific-domain texts, such as the Education and professional training domain, general language embedding models often inadequately represent the unique terminologies and contextual nuances of a specialized domain. To tackle this problem, we present HMCCCProbT, a novel hierarchical multi-label text classification approach. This innovative framework chains multiple classifiers, where each individual classifier is built using a novel sentence-embedding method BERTEPro based on existing Transformer models, whose pre-training has been extended on education and professional training texts, before being fine-tuned on several NLP tasks. Each individual classifier is responsible for the predictions of a given hierarchical level and propagates local probability predictions augmented with the input feature vectors to the classifier in charge of the subsequent level. HMCCCProbT tackles issues of model scalability and semantic interpretation, offering a powerful solution to the challenges of domain-specific hierarchical multi-label classification. Experiments over three domain-specific textual HMC datasets indicate the effectiveness of HMCCCProbT to compare favorably to state-of-the-art HMC algorithms in terms of classification accuracy and also the ability of BERTEPro to obtain better probability predictions, well suited to HMCCCProbT, than three other vector representation techniques.

Context-Aware Dynamic Word Embeddings for Aspect Term Extraction

The aspect term extraction (ATE) task aims to extract aspect terms describing a part or an attribute of a product from review sentences. Most existing works rely on either general or domain embedding to address this problem. Despite the promising results, the importance of general and domain embeddings is still ignored by most methods, resulting in degraded performances. Besides, word embedding is also related to downstream tasks, and how to regularize word embeddings to capture context-aware information is an unresolved problem. To solve these issues, we first propose context-aware dynamic word embedding (CDWE), which could simultaneously consider general meanings, domain-specific meanings, and the context information of words. Based on CDWE, we propose an attention-based convolution neural network, called ADWE-CNN for ATE, which could adaptively capture the previous meanings of words by utilizing an attention mechanism to assign different importance to the respective embeddings. The experimental results show that ADWE-CNN achieves a comparable performance with the state-of-the-art approaches. Various ablation studies have been conducted to explore the benefit of each component. Our code is publicly available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">http://github.com/xiejiajia2018/ADWE-CNN</uri> .

Absoluteness for the theory of the inner model constructed from finitely many cofinality quantifiers

We prove that the theory of the models constructible using finitely many cofinality quantifiers – Cλ1,…,λn⁎ and C<λ1,…,<λn⁎ for λ1,…,λn regular cardinals – is set-forcing absolute under the assumption of class many Woodin cardinals, and is independent of the regular cardinals used. Towards this goal we prove some properties of the generic embedding induced from the stationary tower restricted to <μ-closed sets.

Basic functional properties of certain scale of rearrangement‐invariant spaces

AbstractWe define a new scale of function spaces governed by a norm‐like functional based on a combination of a rearrangement‐invariant norm, the elementary maximal function, and powers. A particular instance of such spaces surfaced recently in connection with optimality of target function spaces in general Sobolev embeddings involving upper Ahlfors regular measures; however, a thorough analysis of these structures has not been carried out. We present a variety of results on these spaces including their basic functional properties, their relations to customary function spaces and mutual embeddings, and, in a particular situation, a characterization of their associate structures. We discover a new one‐parameter path of function spaces leading from a Lebesgue space to a Zygmund class and we compare it to the classical one.

Domain-Specific Word Embeddings with Structure Prediction

Abstract Complementary to finding good general word embeddings, an important question for representation learning is to find dynamic word embeddings, for example, across time or domain. Current methods do not offer a way to use or predict information on structure between sub-corpora, time or domain and dynamic embeddings can only be compared after post-alignment. We propose novel word embedding methods that provide general word representations for the whole corpus, domain- specific representations for each sub-corpus, sub-corpus structure, and embedding alignment simultaneously. We present an empirical evaluation on New York Times articles and two English Wikipedia datasets with articles on science and philosophy. Our method, called Word2Vec with Structure Prediction (W2VPred), provides better performance than baselines in terms of the general analogy tests, domain-specific analogy tests, and multiple specific word embedding evaluations as well as structure prediction performance when no structure is given a priori. As a use case in the field of Digital Humanities we demonstrate how to raise novel research questions for high literature from the German Text Archive.

Self-Supervised Learning for Non-Rigid Registration Between Near-Isometric 3D Surfaces in Medical Imaging.

Non-rigid registration between 3D surfaces is an important but notorious problem in medical imaging, because finding correspondences between non-isometric instances is mathematically non-trivial. We propose a novel self-supervised method to learn shape correspondences directly from a group of bone surfaces segmented from CT scans, without any supervision from time-consuming and error-prone manual annotations. Relying on a Siamese architecture, DiffusionNet as the feature extractor is jointly trained with a pair of randomly rotated and scaled copies of the same shape. The learned embeddings are aligned in spectral domain using eigenfunctions of the Laplace-Beltrami Operator. Additional normalization and regularization losses are incorporated to guide the learned embeddings towards a similar uniform representation over spectrum, which promotes the embeddings to encode multiscale features and advocates sparsity and diagonality of the inferred functional maps. Our method achieves state-of-the-art results among the unsupervised methods on several benchmarks, and presents greater robustness and efficacy in registering moderately deformed shapes. A hybrid refinement strategy is proposed to retrieve smooth and close-to-conformal point-to-point correspondences from the inferred functional map. Our method is orientation and discretization-invariant. Given a pair of near-isometric surfaces, our method automatically computes registration in high accuracy, and outputs anatomically meaningful correspondences. In this study, we show that it is possible to use neural networks to learn general embeddings from 3D shapes in a self-supervised way. The learned features are multiscale, informative, and discriminative, which might potentially benefit almost all types of morphology-related downstream tasks, such as diagnostics, data screening and statistical shape analysis in future.

Quantum Particle Swarm Optimization with Deep Learning-Based Arabic Tweets Sentiment Analysis

Sentiment Analysis (SA), a Machine Learning (ML) technique, is often applied in the literature. The SA technique is specifically applied to the data collected from social media sites. The research studies conducted earlier upon the SA of the tweets were mostly aimed at automating the feature extraction process. In this background, the current study introduces a novel method called Quantum Particle Swarm Optimization with Deep Learning-Based Sentiment Analysis on Arabic Tweets (QPSODL-SAAT). The presented QPSODL-SAAT model determines and classifies the sentiments of the tweets written in Arabic. Initially, the data pre-processing is performed to convert the raw tweets into a useful format. Then, the word2vec model is applied to generate the feature vectors. The Bidirectional Gated Recurrent Unit (BiGRU) classifier is utilized to identify and classify the sentiments. Finally, the QPSO algorithm is exploited for the optimal fine-tuning of the hyperparameters involved in the BiGRU model. The proposed QPSODL-SAAT model was experimentally validated using the standard datasets. An extensive comparative analysis was conducted, and the proposed model achieved a maximum accuracy of 98.35%. The outcomes confirmed the supremacy of the proposed QPSODL-SAAT model over the rest of the approaches, such as the Surface Features (SF), Generic Embeddings (GE), Arabic Sentiment Embeddings constructed using the Hybrid (ASEH) model and the Bidirectional Encoder Representations from Transformers (BERT) model.

Cyber threat attribution using unstructured reports in cyber threat intelligence

Cyber-threat attribution is the identification of attacker responsible for a cyber-attack. It is a challenging task as attacker uses different obfuscation and deception techniques to hide its identity. After an attack has occurred, digital forensic investigation is conducted to collect evidence from network/system logs. After investigation and collecting evidence reports are published in multiple formats such as text and PDF. There is no standard format for publishing these reports, so extracting meaningful information from these reports is a challenging task. Manual extraction of features from unstructured cyber-threat intelligence (CTI) is a difficult task. There is a need for an automated mechanism to extract features from unstructured reports and attribute cyber-threat actor (CTA). The aim of this research is to develop a mechanism to attribute or profile cyber threat actors (CTA) by extracting features from CTI reports. Moreover define a methodology to extract features from unstructured CTI reports by using natural language processing (NLP) techniques and then attributing cyber threat actor by using machine learning algorithms. Extracting features i.e., tactics, techniques, tools, malware, target organization/country and application by using novel embedding model known as” Attack2vec” which is trained on domain specific embeddings. Training model on domain specific embedding produces high results as compared to model train on general embeddings specially in the field of cyber security. Results of this novel model is compared with different methods. Machine learning algorithms such as decision tree, random forest, support vector machine is used for classification of CTA. This novel model produces high results as compared to other models with Accuracy of 96%, Precision of 96.4%, Recall of 95.58% and F1-measure of 95.75%.

NEGATIVE RESULTS ON PRECIPITOUS IDEALS ON

AbstractWe show that in many extender models, e.g., the minimal one with infinitely many Woodin cardinals or the minimal with a Woodin cardinal that is a limit of Woodin cardinals, there are no generic embeddings with critical point $\omega _1$ that resemble the stationary tower at the second Woodin cardinal. The meaning of “resemble” is made precise in the paper (see Definition 0.3).

Context Sensitive Verb Similarity Dataset for Legal Information Extraction

Existing literature demonstrates that verbs are pivotal in legal information extraction tasks due to their semantic and argumentative properties. However, granting computers the ability to interpret the meaning of a verb and its semantic properties in relation to a given context can be considered as a challenging task, mainly due to the polysemic and domain specific behaviours of verbs. Therefore, developing mechanisms to identify behaviors of verbs and evaluate how artificial models detect the domain specific and polysemic behaviours of verbs can be considered as tasks with significant importance. In this regard, a comprehensive dataset that can be used as an evaluation resource, as well as a training data set, can be considered as a major requirement. In this paper, we introduce LeCoVe, which is a verb similarity dataset intended towards facilitating the process of identifying verbs with similar meanings in a legal domain specific context. Using the dataset, we evaluated both domain specific and domain generic embedding models, which were developed using state-of-the-art word representation and language modelling techniques. As a part of the experiments carried out using the announced dataset, Sense2Vec and BERT models were trained using a corpus of legal opinion texts in order to capture domain specific behaviours. In addition to LeCoVe, we demonstrate that a neural network model, which was developed by combining semantic, syntactic, and contextual features that can be obtained from the outputs of embedding models, can perform comparatively well, even in a low resource scenario.

Embedding Principle: A Hierarchical Structure of Loss Landscape of Deep Neural Networks

We prove a general Embedding Principle of loss landscape of deep neural networks (NNs) that unravels a hierarchical structure of the loss landscape of NNs, i.e., loss landscape of an NN contains all critical points of all the narrower NNs. This result is obtained by constructing a class of critical embeddings which map any critical point of a narrower NN to a critical point of the target NN with the same output function. By discovering a wide class of general compatible critical embeddings, we provide a gross estimate of the dimension of critical submanifolds embedded from critical points of narrower NNs. We further prove an irreversiblility property of any critical embedding that the number of negative/zero/positive eigenvalues of the Hessian matrix of a critical point may increase but never decrease as an NN becomes wider through the embedding. Using a special realization of general compatible critical embedding, we prove a stringent necessary condition for being a "truly-bad" critical point that never becomes a strict-saddle point through any critical embedding. This result implies the commonplace of strict-saddle points in wide NNs, which may be an important reason underlying the easy optimization of wide NNs widely observed in practice.

Comparison of Word and Character Level Information for Medical Term Identification Using Convolutional Neural Networks and Transformers.

Complexity and domain-specificity make medical text hard to understand for patients and their next of kin. To simplify such text, this paper explored how word and character level information can be leveraged to identify medical terms when training data is limited. We created a dataset of medical and general terms using the Human Disease Ontology from BioPortal and Wikipedia pages. Our results from 10-fold cross validation indicated that convolutional neural networks (CNNs) and transformers perform competitively. The best F score of 93.9% was achieved by a CNN trained on both word and character level embeddings. Statistical significance tests demonstrated that general word embeddings provide rich word representations for medical term identification. Consequently, focusing on words is favorable for medical term identification if using deep learning architectures.

The monodromy conjecture for a space monomial curve with a plane semigroup

This article investigates the monodromy conjecture for a space monomial curve that appears as the special fiber of an equisingular family of curves with a plane branch as generic fiber. Roughly speaking, the monodromy conjecture states that every pole of the motivic, or related, Igusa zeta function induces an eigenvalue of monodromy. As the poles of the motivic zeta function associated with such a space monomial curve have been determined in earlier work, it remains to study the eigenvalues of monodromy. After reducing the problem to the curve seen as a Cartier divisor on a generic embedding surface, we construct an embedded Q-resolution of this pair and use an A’Campo formula in terms of this resolution to compute the zeta function of monodromy. Combining all results, we prove the monodromy conjecture for this class of monomial curves.

Development and evaluation of novel ophthalmology domain-specific neural word embeddings to predict visual prognosis

ObjectiveTo develop and evaluate novel word embeddings (WEs) specific to ophthalmology, using text corpora from published literature and electronic health records (EHR). Materials and MethodsWe trained ophthalmology-specific WEs using 121,740 PubMed abstracts and 89,282 EHR notes using word2vec continuous bag-of-words architecture. PubMed and EHR WEs were compared to general domain GloVe WEs and general biomedical domain BioWordVec embeddings using a novel ophthalmology-domain-specific 200-question analogy test and prediction of prognosis in 5547 low vision patients using EHR notes as inputs to a deep learning model. ResultsWe found that many words representing important ophthalmic concepts in the EHR were missing from the general domain GloVe vocabulary, but covered in the ophthalmology abstract corpus. On ophthalmology analogy testing, PubMed WEs scored 95.0 %, outperforming EHR (86.0 %) and GloVe (91.0 %) but less than BioWordVec (99.5 %). On predicting low vision prognosis, PubMed and EHR WEs resulted in similar AUROC (0.830; 0.826), outperforming GloVe (0.778) and BioWordVec (0.784). ConclusionWe found that using ophthalmology domain-specific WEs improved performance in ophthalmology-related clinical prediction compared to general WEs. Deep learning models using clinical notes as inputs can predict the prognosis of visually impaired patients. This work provides a framework to improve predictive models using domain-specific WEs.

Comparing general and specialized word embeddings for biomedical named entity recognition.

Increased interest in the use of word embeddings, such as word representation, for biomedical named entity recognition (BioNER) has highlighted the need for evaluations that aid in selecting the best word embedding to be used. One common criterion for selecting a word embedding is the type of source from which it is generated; that is, general (e.g., Wikipedia, Common Crawl), or specific (e.g., biomedical literature). Using specific word embeddings for the BioNER task has been strongly recommended, considering that they have provided better coverage and semantic relationships among medical entities. To the best of our knowledge, most studies have focused on improving BioNER task performance by, on the one hand, combining several features extracted from the text (for instance, linguistic, morphological, character embedding, and word embedding itself) and, on the other, testing several state-of-the-art named entity recognition algorithms. The latter, however, do not pay great attention to the influence of the word embeddings, and do not facilitate observing their real impact on the BioNER task. For this reason, the present study evaluates three well-known NER algorithms (CRF, BiLSTM, BiLSTM-CRF) with respect to two corpora (DrugBank and MedLine) using two classic word embeddings, GloVe Common Crawl (of the general type) and Pyysalo PM + PMC (specific), as unique features. Furthermore, three contextualized word embeddings (ELMo, Pooled Flair, and Transformer) are compared in their general and specific versions. The aim is to determine whether general embeddings can perform better than specialized ones on the BioNER task. To this end, four experiments were designed. In the first, we set out to identify the combination of classic word embedding, NER algorithm, and corpus that results in the best performance. The second evaluated the effect of the size of the corpus on performance. The third assessed the semantic cohesiveness of the classic word embeddings and their correlation with respect to several gold standards; while the fourth evaluates the performance of general and specific contextualized word embeddings on the BioNER task. Results show that the classic general word embedding GloVe Common Crawl performed better in the DrugBank corpus, despite having less word coverage and a lower internal semantic relationship than the classic specific word embedding, Pyysalo PM + PMC; while in the contextualized word embeddings the best results are presented in the specific ones. We conclude, therefore, when using classic word embeddings as features on the BioNER task, the general ones could be considered a good option. On the other hand, when using contextualized word embeddings, the specific ones are the best option.

Novel metric for hyperbolic phylogenetic tree embeddings.

Advances in experimental technologies, such as DNA sequencing, have opened up new avenues for the applications of phylogenetic methods to various fields beyond their traditional application in evolutionary investigations, extending to the fields of development, differentiation, cancer genomics, and immunogenomics. Thus, the importance of phylogenetic methods is increasingly being recognized, and the development of a novel phylogenetic approach can contribute to several areas of research. Recently, the use of hyperbolic geometry has attracted attention in artificial intelligence research. Hyperbolic space can better represent a hierarchical structure compared to Euclidean space, and can therefore be useful for describing and analyzing a phylogenetic tree. In this study, we developed a novel metric that considers the characteristics of a phylogenetic tree for representation in hyperbolic space. We compared the performance of the proposed hyperbolic embeddings, general hyperbolic embeddings, and Euclidean embeddings, and confirmed that our method could be used to more precisely reconstruct evolutionary distance. We also demonstrate that our approach is useful for predicting the nearest-neighbor node in a partial phylogenetic tree with missing nodes. Furthermore, we proposed a novel approach based on our metric to integrate multiple trees for analyzing tree nodes or imputing missing distances. This study highlights the utility of adopting a geometric approach for further advancing the applications of phylogenetic methods.