Set Of Embeddings Research Articles

Fine-grained semantic oriented embedding set alignment for text-based person search

Fine-grained semantic oriented embedding set alignment for text-based person search

Semantic Interaction Meta-Learning Based on Patch Matching Metric.

Metric-based meta-learning methods have demonstrated remarkable success in the domain of few-shot image classification. However, their performance is significantly contingent upon the choice of metric and the feature representation for the support classes. Current approaches, which predominantly rely on holistic image features, may inadvertently disregard critical details necessary for novel tasks, a phenomenon known as "supervision collapse". Moreover, relying solely on visual features to characterize support classes can prove to be insufficient, particularly in scenarios involving limited sample sizes. In this paper, we introduce an innovative framework named Patch Matching Metric-based Semantic Interaction Meta-Learning (PatSiML), designed to overcome these challenges. To counteract supervision collapse, we have developed a patch matching metric strategy based on the Transformer architecture to transform input images into a set of distinct patch embeddings. This approach dynamically creates task-specific embeddings, facilitated by a graph convolutional network, to formulate precise matching metrics between the support classes and the query image patches. To enhance the integration of semantic knowledge, we have also integrated a label-assisted channel semantic interaction strategy. This strategy merges word embeddings with patch-level visual features across the channel dimension, utilizing a sophisticated language model to combine semantic understanding with visual information. Our empirical findings across four diverse datasets reveal that the PatSiML method achieves a classification accuracy improvement of 0.65% to 21.15% over existing methodologies, underscoring its robustness and efficacy.

GCReID: Generalized continual person re-identification via meta learning and knowledge accumulation

Person re-identification (ReID) has made good progress in stationary domains. The ReID model must be retrained to adapt to new scenarios (domains) as they emerge unexpectedly, which leads to catastrophic forgetting. Continual learning trains the model in the order of domain emergence to alleviate catastrophic forgetting. However, generalization ability of the model is still limited due to the distribution difference between training and testing domains. To address the above problem, we propose the generalized continual person re-Identification (GCReID) model to continuously train an anti-forgetting and generalizable model. We endeavor to increase the diversity of samples by prior to simulate unseen domains. Meta-train and meta-test are adopted to enhance generalization of the model. Universal knowledge extracted from all seen domains and the simulated domains is stored in a set of feature embeddings. The knowledge is continually updated and applied to guide meta-train and meta-test via a graph attention network. Extensive experiments on 12 benchmark datasets and comparisons with 6 representative models demonstrate the effectiveness of the proposed model GCReID in enhancing generalization performance on unseen domains and alleviating catastrophic forgetting of seen domains. The code will be available at https://github.com/DFLAG-NEU/GCReID if our work is accepted.

Particle identification with machine learning from incomplete data in the ALICE experiment

The ALICE experiment at the LHC measures properties of the strongly interacting matter formed in ultrarelativistic heavy-ion collisions. Such studies require accurate particle identification (PID). ALICE provides PID information via several detectors for particles with momentum from about 100 MeV/c up to 20 GeV/c. Traditionally, particles are selected with rectangular cuts. A much better performance can be achieved with machine learning (ML) methods. Our solution uses multiple neural networks (NN) serving as binary classifiers. Moreover, we extended our particle classifier with Feature Set Embedding and attention in order to train on data with incomplete samples. We also present the integration of the ML project with the ALICE analysis software, and we discuss domain adaptation, the ML technique needed to transfer the knowledge between simulated and real experimental data.

Advancing Front Surface Mapping

AbstractWe present Advancing Front Mapping (AFM), a novel algorithm for the computation of injective maps to simple planar domains. AFM is inspired by the advancing front meshing paradigm, which is here revisited to operate on two embeddings at once, becoming a tool for compatible mesh generation. AFM extends the capabilities of existing robust approaches, supporting a broader set of embeddings (star‐shaped polygons) with a direct approach, without resorting to intermediate constructions. Our method only relies on two topological operators (split and flip) and on the computation of segment intersections, thus permitting to compute a valid embedding without solving any numerical problem. AFM is therefore easy to implement, debug and deploy. This article is mainly focused on the presentation of the compatible advancing front idea and on the demonstration that the algorithm provably converges to an injective map. We also complement our theoretical analysis with an extensive practical validation, executing more than one billion advancing front moves on 36K mapping tasks.

Tensorially absorbing inclusions of C*-algebras

Abstract When $\mathcal {D}$ is strongly self-absorbing, we say an inclusion $B \subseteq A$ of C*-algebras is $\mathcal {D}$ -stable if it is isomorphic to the inclusion $B \otimes \mathcal {D} \subseteq A \otimes \mathcal {D}$ . We give ultrapower characterizations and show that if a unital inclusion is $\mathcal {D}$ -stable, then $\mathcal {D}$ -stability can be exhibited for countably many intermediate C*-algebras concurrently. We show that such unital embeddings between unital $\mathcal {D}$ -stable C*-algebras are point-norm dense in the set of all unital embeddings, and that every unital embedding between $\mathcal {D}$ -stable C*-algebras is approximately unitarily equivalent to a $\mathcal {D}$ -stable embedding. Examples are provided.

Audiovisual Moments in Time: A large-scale annotated dataset of audiovisual actions.

We present Audiovisual Moments in Time (AVMIT), a large-scale dataset of audiovisual action events. In an extensive annotation task 11 participants labelled a subset of 3-second audiovisual videos from the Moments in Time dataset (MIT). For each trial, participants assessed whether the labelled audiovisual action event was present and whether it was the most prominent feature of the video. The dataset includes the annotation of 57,177 audiovisual videos, each independently evaluated by 3 of 11 trained participants. From this initial collection, we created a curated test set of 16 distinct action classes, with 60 videos each (960 videos). We also offer 2 sets of pre-computed audiovisual feature embeddings, using VGGish/YamNet for audio data and VGG16/EfficientNetB0 for visual data, thereby lowering the barrier to entry for audiovisual DNN research. We explored the advantages of AVMIT annotations and feature embeddings to improve performance on audiovisual event recognition. A series of 6 Recurrent Neural Networks (RNNs) were trained on either AVMIT-filtered audiovisual events or modality-agnostic events from MIT, and then tested on our audiovisual test set. In all RNNs, top 1 accuracy was increased by 2.71-5.94% by training exclusively on audiovisual events, even outweighing a three-fold increase in training data. Additionally, we introduce the Supervised Audiovisual Correspondence (SAVC) task whereby a classifier must discern whether audio and visual streams correspond to the same action label. We trained 6 RNNs on the SAVC task, with or without AVMIT-filtering, to explore whether AVMIT is helpful for cross-modal learning. In all RNNs, accuracy improved by 2.09-19.16% with AVMIT-filtered data. We anticipate that the newly annotated AVMIT dataset will serve as a valuable resource for research and comparative experiments involving computational models and human participants, specifically when addressing research questions where audiovisual correspondence is of critical importance.

BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions

Vision Language Models (VLMs), which extend Large Language Models (LLM) by incorporating visual understanding capability, have demonstrated significant advancements in addressing open-ended visual question-answering (VQA) tasks. However, these models cannot accurately interpret images infused with text, a common occurrence in real-world scenarios. Standard procedures for extracting information from images often involve learning a fixed set of query embeddings. These embeddings are designed to encapsulate image contexts and are later used as soft prompt inputs in LLMs. Yet, this process is limited to the token count, potentially curtailing the recognition of scenes with text-rich context. To improve upon them, the present study introduces BLIVA: an augmented version of InstructBLIP with Visual Assistant. BLIVA incorporates the query embeddings from InstructBLIP and also directly projects encoded patch embeddings into the LLM, a technique inspired by LLaVA. This approach assists the model to capture intricate details potentially missed during the query decoding process. Empirical evidence demonstrates that our model, BLIVA, significantly enhances performance in processing text-rich VQA benchmarks (up to 17.76% in OCR-VQA benchmark) and in undertaking general (not particularly text-rich) VQA benchmarks (up to 7.9% in Visual Spatial Reasoning benchmark), and achieved 17.72% overall improvement in a comprehensive multimodal LLM benchmark (MME), comparing to our baseline InstructBLIP. BLIVA demonstrates significant capability in decoding real-world images, irrespective of text presence. To demonstrate the broad industry applications enabled by BLIVA, we evaluate the model using a new dataset comprising YouTube thumbnails paired with question-answer sets across 11 diverse categories. For researchers interested in further exploration, our code and models are freely accessible at https://github.com/mlpc-ucsd/BLIVA.

Unsupervised framework for evaluating and explaining structural node embeddings of graphs

Abstract An embedding is a mapping from a set of nodes of a network into a real vector space. Embeddings can have various aims like capturing the underlying graph topology and structure, node-to-node relationship, or other relevant information about the graph, its subgraphs or nodes themselves. A practical challenge with using embeddings is that there are many available variants to choose from. Selecting a small set of most promising embeddings from the long list of possible options for a given task is challenging and often requires domain expertise. Embeddings can be categorized into two main types: classical embeddings and structural embeddings. Classical embeddings focus on learning both local and global proximity of nodes, while structural embeddings learn information specifically about the local structure of nodes’ neighbourhood. For classical node embeddings, there exists a framework which helps data scientists to identify (in an unsupervised way) a few embeddings that are worth further investigation. Unfortunately, no such framework exists for structural embeddings. In this article, we propose a framework for unsupervised ranking of structural graph embeddings. The proposed framework, apart from assigning an aggregate quality score for a structural embedding, additionally gives a data scientist insights into properties of this embedding. It produces information which predefined node features the embedding learns, how well it learns them, and which dimensions in the embedded space represent the predefined node features. Using this information, the user gets a level of explainability to an otherwise complex black-box embedding algorithm.

Development and multicentre validation of the FLEX score: personalised preoperative surgical risk prediction using attention-based ICD-10 and Current Procedural Terminology set embeddings

BackgroundPreoperative knowledge of surgical risks can improve perioperative care and patient outcomes. However, assessments requiring clinician examination of patients or manual chart review can be too burdensome for routine use. MethodsWe conducted a multicentre retrospective study of 243 479 adult noncardiac surgical patients at four hospitals within the Mass General Brigham (MGB) system in the USA. We developed a machine learning method using routinely collected coding and patient characteristics data from the electronic health record which predicts 30-day mortality, 30-day readmission, discharge to long-term care, and hospital length of stay. ResultsOur method, the Flexible Surgical Set Embedding (FLEX) score, achieved state-of-the-art performance to identify comorbidities that significantly contribute to the risk of each adverse outcome. The contributions of comorbidities are weighted based on patient-specific context, yielding personalised risk predictions. Understanding the significant drivers of risk of adverse outcomes for each patient can inform clinicians of potential targets for intervention. ConclusionsFLEX utilises information from a wider range of medical diagnostic and procedural codes than previously possible and can adapt to different coding practices to accurately predict adverse postoperative outcomes.

Improved Diversity-Promoting Collaborative Metric Learning for Recommendation.

Collaborative Metric Learning (CML) has recently emerged as a popular method in recommendation systems (RS), closing the gap between metric learning and collaborative filtering. Following the convention of RS, existing practices exploit unique user representation in their model design. This paper focuses on a challenging scenario where a user has multiple categories of interests. Under this setting, the unique user representation might induce preference bias, especially when the item category distribution is imbalanced. To address this issue, we propose a novel method called Diversity-Promoting Collaborative Metric Learning (DPCML), with the hope of considering the commonly ignored minority interest of the user. The key idea behind DPCML is to introduce a set of multiple representations for each user in the system where users' preference toward an item is aggregated by taking the minimum item-user distance among their embedding set. Specifically, we instantiate two effective assignment strategies to explore a proper quantity of vectors for each user. Meanwhile, a Diversity Control Regularization Scheme (DCRS) is developed to accommodate the multi-vector representation strategy better. Theoretically, we show that DPCML could induce a smaller generalization error than traditional CML. Furthermore, we notice that CML-based approaches usually require negative sampling to reduce the heavy computational burden caused by the pairwise objective therein. In this paper, we reveal the fundamental limitation of the widely adopted hard-aware sampling from the One-Way Partial AUC (OPAUC) perspective and then develop an effective sampling alternative for the CML-based paradigm. Finally, comprehensive experiments over a range of benchmark datasets speak to the efficacy of DPCML.

A Language and Its Dimensions: Intrinsic Dimensions of Language Fractal Structures

The present paper introduces a novel object of study, a language fractal structure; we hypothesize that a set of embeddings of all n‐grams of a natural language constitutes a representative sample of this fractal set. (We use the term Hailonakea to refer to the sum total of all language fractal structures, over all n). The paper estimates intrinsic (genuine) dimensions of language fractal structures for the Russian and English languages. To this end, we employ methods based on (1) topological data analysis and (2) a minimum spanning tree of a data graph for a cloud of points considered (Steele theorem). For both languages, for all n, the intrinsic dimensions appear to be noninteger values (typical for fractal sets), close to 9 for both of the Russian and English language.

Audio Representation Learning by Distilling Video as Privileged Information

Deep audio representation learning using multi-modal audio-visual data often leads to a better performance compared to uni-modal approaches. However, in real-world scenarios both modalities are not always available at the time of inference, leading to performance degradation by models trained for multi-modal inference. In this work, we propose a novel approach for deep audio representation learning using audio-visual data when the video modality is absent at inference. For this purpose, we adopt teacher-student knowledge distillation under the framework of learning using privileged information (LUPI). While the previous methods proposed for LUPI use soft-labels generated by the teacher, in our proposed method we use <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">embeddings</i> learned by the teacher to train the student network. We integrate our method in two different settings: sequential data where the features are divided into multiple segments throughout time, and non-sequential data where the entire features are treated as one whole segment. In the non-sequential setting both the teacher and student networks are comprised of an encoder component and a task header. We use the embeddings produced by the encoder component of the teacher to train the encoder of the student, while the task header of the student is trained using ground-truth labels. In the sequential setting, the networks have an additional aggregation component that is placed between the encoder and task header. We use two sets of embeddings produced by the encoder and aggregation component of the teacher to train the student. Similar to the non-sequential setting, the task header of the student network is trained using ground-truth labels. We test our framework on two different audio-visual tasks, namely speaker recognition and speech emotion recognition. Through these experiments we show that by treating the video modality as privileged information for the main goal of audio representation learning, our method results in considerable improvements over sole audio-based recognition as well as prior works that use LUPI.

Development and prospective validation of postoperative pain prediction from preoperative EHR data using attention-based set embeddings

Preoperative knowledge of expected postoperative pain can help guide perioperative pain management and focus interventions on patients with the greatest risk of acute pain. However, current methods for predicting postoperative pain require patient and clinician input or laborious manual chart review and often do not achieve sufficient performance. We use routinely collected electronic health record data from a multicenter dataset of 234,274 adult non-cardiac surgical patients to develop a machine learning method which predicts maximum pain scores on the day of surgery and four subsequent days and validate this method in a prospective cohort. Our method, POPS, is fully automated and relies only on data available prior to surgery, allowing application in all patients scheduled for or considering surgery. Here we report that POPS achieves state-of-the-art performance and outperforms clinician predictions on all postoperative days when predicting maximum pain on the 0–10 NRS in prospective validation, though with degraded calibration. POPS is interpretable, identifying comorbidities that significantly contribute to postoperative pain based on patient-specific context, which can assist clinicians in mitigating cases of acute pain.

Heterogeneous Evolution Network Embedding with Temporal Extension for Intelligent Tutoring Systems

Graph embedding (GE) aims to acquire low-dimensional node representations while maintaining the graph’s structural and semantic attributes. Intelligent tutoring systems (ITS) signify a noteworthy achievement in the fusion of AI and education. Utilizing GE to model ITS can elevate their performance in predictive and annotation tasks. Current GE techniques, whether applied to heterogeneous or dynamic graphs, struggle to efficiently model ITS data. The GEs within ITS should retain their semidynamic, independent, and smooth characteristics. This article introduces a heterogeneous evolution network (HEN) for illustrating entities and relations within an ITS. Additionally, we introduce a temporal extension graph neural network (TEGNN) to model both evolving and static nodes within the HEN. In the TEGNN framework, dynamic nodes are initially improved over time through temporal extension (TE), providing an accurate depiction of each learner’s implicit state at each time step. Subsequently, we propose a stochastic temporal pooling (STP) strategy to estimate the embedding sets of all evolving nodes. This effectively enhances model efficiency and usability. Following this, a heterogeneous aggregation network is devised to proficiently extract heterogeneous features from the HEN. This network employs both node-level and relation-level attention mechanisms to craft aggregated node features. To emphasize the superiority of TEGNN, we perform experiments on several real ITS datasets and show that our method significantly outperforms the state-of-the-art approaches. The experiments validate that TE serves as an efficient framework for modeling temporal information in GE, and STP not only accelerates the training process but also enhances the resultant accuracy.

Simulating doctors’ thinking logic for chest X-ray report generation via Transformer-based Semantic Query learning

Medical report generation can be treated as a process of doctors’ observing, understanding, and describing images from different perspectives. Following this process, this paper innovatively proposes a Transformer-based Semantic Query learning paradigm (TranSQ). Briefly, this paradigm is to learn an intention embedding set and make a semantic query to the visual features, generate intent-compliant sentence candidates, and form a coherent report. We apply a bipartite matching mechanism during training to realize the dynamic correspondence between the intention embeddings and the sentences to induct medical concepts into the observation intentions. Experimental results on two major radiology reporting datasets (i.e., IU X-ray and MIMIC-CXR) demonstrate that our model outperforms state-of-the-art models regarding generation effectiveness and clinical efficacy. In addition, comprehensive ablation experiments fully validate the TranSQ model’s innovation and interpretation. The code is available at https://github.com/zjukongming/TranSQ.

Multiple sparse graphs condensation

The high complexity of graph neural networks (GNNs) on large-scale networks hinders their industrial application. Graph condensation (GCond) was recently proposed to condense the original large-scale graph into a small-scale one to address the problem. The goal is to make GNNs trained on the condensed graph perform similarly to those trained on the original graph. GCond achieves satisfactory performance on some datasets. However, GCond uses a single fully connected graph to model the condensed graph, which limits the diversity of embeddings obtained, especially when there are few synthetic nodes. We propose Multiple Sparse Graphs Condensation (MSGC), which condenses the original large-scale graph into multiple small-scale sparse graphs. MSGC takes standard neighborhood patterns as the essential substructures and can construct various connection schemes. Correspondingly, GNNs can obtain numerous sets of embeddings, which significantly enriches the diversity of embeddings. Experiments show that, compared with GCond and other baselines, MSGC has significant advantages at the same condensed graph scale. MSGC can retain nearly 100% performance on Flickr and Citeseer datasets while reducing their graph scale by over 99.0%.

Optimal sufficient requirements on the embedded Ising problem in polynomial time

One of the central applications for quantum annealers is to find the solutions of Ising problems. Suitable Ising problems, however, need to be formulated such that they, on the one hand, respect the specific restrictions of the hardware and, on the other hand, represent the original problems which shall actually be solved. We evaluate sufficient requirements on such an embedded Ising problem analytically and transform them into a linear optimization problem. With an objective function aiming to minimize the maximal absolute problem parameter, the precision issues of the annealers are addressed. Due to the redundancy of several constraints, we can show that the formally exponentially large optimization problem can be reduced and finally solved in polynomial time for the standard embedding setting where the embedded vertices induce trees. This allows to formulate provably equivalent embedded Ising problems in a practical setup.

GTR-GA: Harnessing the power of graph-based neural networks and genetic algorithms for text augmentation

Text augmentation is a popular technique in natural language processing (NLP) that has been shown to improve the performance of various downstream tasks. The goal of text augmentation is to generate additional training data from existing data, thereby increasing the amount of data available for training machine learning models. In this paper, we propose a novel approach to text augmentation that combines the power of graph-based neural networks and genetic algorithms. Our proposed scheme, called GTR-GA, is designed to generate diverse and high-quality augmented text data by exploring the high-dimensional feature space of text data using graph-based neural networks and genetic algorithms. Our approach utilizes a graph attention network (GAT) based model, called HetGAPN, to learn node representations in a heterogeneous graph representing text data. We introduce the concept of node aggregation and bidirectional attention in HetGAPN to better capture the relationships between different features of the input text data. We then use a genetic algorithm to generate a diverse set of candidate embeddings that explore different parts of the input space. Our approach uses perplexity as the objective function to evaluate the quality of the augmented text data generated by each candidate embedding. The most fit candidate embeddings are selected as parents, and crossover and mutation operators are used to generate a new population of candidate embeddings. Our experimental results show that GTR-GA is capable of generating high-quality augmented text data that improves the performance of various downstream NLP tasks, such as sentiment analysis and text classification. The proposed scheme can be applied to a wide range of NLP tasks and can help overcome the data scarcity problem that is often encountered in NLP.

Misinformation and Public Health Messaging in the Early Stages of the Mpox Outbreak: Mapping the Twitter Narrative With Deep Learning.

Shortly after the worst of the COVID-19 pandemic, an outbreak of mpox introduced another critical public health emergency. Like the COVID-19 pandemic, the mpox outbreak was characterized by a rising prevalence of public health misinformation on social media, through which many US adults receive and engage with news. Digital misinformation continues to challenge the efforts of public health officials in providing accurate and timely information to the public. We examine the evolving topic distributions of social media narratives during the mpox outbreak to map the tension between rapidly diffusing misinformation and public health communication. This study aims to observe topical themes occurring in a large-scale collection of tweets about mpox using deep learning. We leveraged a data set comprised of all mpox-related tweets that were posted between May 7, 2022, and July 23, 2022. We then applied Sentence Bidirectional Encoder Representations From Transformers (S-BERT) to the content of each tweet to generate a representation of its content in high-dimensional vector space, where semantically similar tweets will be located closely together. We projected the set of tweet embeddings to a 2D map by applying principal component analysis and Uniform Manifold Approximation Projection (UMAP). Finally, we group these data points into 7 topical clusters using k-means clustering and analyze each cluster to determine its dominant topics. We analyze the prevalence of each cluster over time to evaluate longitudinal thematic changes. Our deep-learning pipeline revealed 7 distinct clusters of content: (1) cynicism, (2) exasperation, (3) COVID-19, (4) men who have sex with men, (5) case reports, (6) vaccination, and (7) World Health Organization (WHO). Clusters that largely communicated erroneous or irrelevant information began earlier and grew faster, reaching a wider audience than later communications by official instances and health officials. Within a few weeks of the first reported mpox cases, an avalanche of mostly false, misleading, irrelevant, or damaging information started to circulate on social media. Official institutions, including the WHO, acted promptly, providing case reports and accurate information within weeks, but were overshadowed by rapidly spreading social media chatter. Our results point to the need for real-time monitoring of social media content to optimize responses to public health emergencies.