High-dimensional Vector Space Research Articles

Improving information theory of context-aware phrase embeddings in HR domain

The object of the study is context-aware phrase representations. The growing need to automate candidate recruitment and job recommendation processes has paved the way for the utilization of text embeddings. These embeddings involve translating the semantic essence of text into a continuous, high-dimensional vector space. By learning context-aware rich and meaningful representations of phrases within the human resource domain, the efficacy of similarity searches and matching procedures is enhanced, which contributes to a more streamlined and effective recruitment process. However, existing approaches do not take into account the context when modeling phrases. This necessitates the improvement of information technology analysis in this area. In this paper, it is proposed to mark the beginning and end of phrases in the text using special tokens. This made it possible to reduce the requirements for computing power by calculating all phrase representations present in the text simultaneously. The effectiveness of the improvement was tested on a new dataset to compare and evaluate the models in the task of modeling phrases in the field of human resources management. The proposed approach to modeling phrase representations with regard to context in the field of human resources management leads to an improvement in computational efficiency by up to 50 % and an increase in accuracy by up to 10 %. The architecture of the machine learning model for creating context-aware phrase representations is developed, which is characterized by the presence of blocks for taking into account phrase boundaries. Experiments and comparisons with existing approaches have confirmed the effectiveness of the proposed solution. In practice, the proposed information analysis technology can be used to automate the process of identifying and normalizing candidates' skills in online recruiting

Hyperdimensional computing: a framework for stochastic computation and symbolic AI

Hyperdimensional Computing (HDC), also known as Vector Symbolic Architectures (VSA), is a neuro-inspired computing framework that exploits high-dimensional random vector spaces. HDC uses extremely parallelizable arithmetic to provide computational solutions that balance accuracy, efficiency and robustness. The majority of current HDC research focuses on the learning capabilities of these high-dimensional spaces. However, a tangential research direction investigates the properties of these high-dimensional spaces more generally as a probabilistic model for computation. In this manuscript, we provide an approachable, yet thorough, survey of the components of HDC. To highlight the dual use of HDC, we provide an in-depth analysis of two vastly different applications. The first uses HDC in a learning setting to classify graphs. Graphs are among the most important forms of information representation, and graph learning in IoT and sensor networks introduces challenges because of the limited compute capabilities. Compared to the state-of-the-art Graph Neural Networks, our proposed method achieves comparable accuracy, while training and inference times are on average 14.6× and 2.0× faster, respectively. Secondly, we analyse a dynamic hash table that uses a novel hypervector type called circular-hypervectors to map requests to a dynamic set of resources. The proposed hyperdimensional hashing method has the efficiency to be deployed in large systems. Moreover, our approach remains unaffected by a realistic level of memory errors which causes significant mismatches for existing methods.

The classification of Boolean degree 1 functions in high-dimensional finite vector spaces

We classify the Boolean degree 1 1 functions of k k -spaces in a vector space of dimension n n (also known as Cameron-Liebler classes) over the field with q q elements for n ≥ n 0 ( k , q ) n \geq n_0(k, q) . This also implies that two-intersecting sets with respect to k k -spaces do not exist for n ≥ n 0 ( k , q ) n \geq n_0(k, q) . Our main ingredient is the Ramsey theory for geometric lattices.

A Lightweight Model Enhancing Facial Expression Recognition with Spatial Bias and Cosine-Harmony Loss

This paper proposes a facial expression recognition network called the Lightweight Facial Network with Spatial Bias (LFNSB). The LFNSB model effectively balances model complexity and recognition accuracy. It has two key components: a lightweight feature extraction network (LFN) and a Spatial Bias (SB) module for aggregating global information. The LFN introduces combined channel operations and depth-wise convolution techniques, effectively reducing the number of parameters while enhancing feature representation capability. The Spatial Bias module enables the model to focus on local facial features while capturing the dependencies between different facial regions. Additionally, a new loss function called Cosine-Harmony Loss is designed. This function optimizes the relative positions of feature vectors in high-dimensional space, resulting in better feature separation and clustering. Experimental results on the AffectNet and RAF-DB datasets demonstrate that the LFNSB model achieves competitive recognition accuracy, with 63.12% accuracy on AffectNet-8, 66.57% accuracy on AffectNet-7, and 91.07% accuracy on RAF-DB, while significantly reducing the model complexity.

Document Similarity Using Term Frequency-Inverse Document Frequency Representation and Cosine Similarity

Document similarity is a fundamental task in natural language processing and information retrieval, with applications ranging from plagiarism detection to recommendation systems. In this study, we leverage the term frequency-inverse document frequency (TF-IDF) to represent documents in a high-dimensional vector space, capturing their unique content while mitigating the influence of common terms. Subsequently, we employ the cosine similarity metric to measure the similarity between pairs of documents, which assesses the angle between their respective TF-IDF vectors. To evaluate the effectiveness of our approach, we conducted experiments on the Document Similarity Triplets Dataset, a benchmark dataset specifically designed for assessing document similarity techniques. Our experimental results demonstrate a significant performance with an accuracy score of 93.6% using bigram-only representation. However, we observed instances where false predictions occurred due to paired documents having similar terms but differing semantics, revealing a weakness in the TF-IDF approach. To address this limitation, future research could focus on augmenting document representations with semantic features. Incorporating semantic information, such as word embeddings or contextual embeddings, could enhance the model's ability to capture nuanced semantic relationships between documents, thereby improving accuracy in scenarios where term overlap does not adequately signify similarity.

Random walk theory and application

This project presents an overview of Random Walk Theory and its applications, as discussed in the provided project work. Random Walk Theory posits that changes in elements like stock prices follow a distribution independent of past movements, making future predictions challenging. Originating from the work of French mathematician Louis Bachelier and later popularized by economist Burton Markiel, the theory finds extensive applications beyond finance, spanning fields such as psychology, economics, and physics. The project delves into various types of random walks, including symmetric random walks, and explores their implications in different spaces, from graphs to higher-dimensional vector spaces. It provides definitions, examples, and graphical representations to elucidate random walk concepts, highlighting their relevance in practical scenarios like particle movement and stock price fluctuations. Key concepts such as the reflection principle and the main lemma are discussed to provide a comprehensive understanding of random walks and their properties. Through examples and lemmas, the project elucidates the mathematical foundations of random walks, offering insights into their behavior and applications across diverse disciplines. In summary, this project contributes to a deeper comprehension of Random Walk Theory, serving as a fundamental framework for understanding stochastic processes and their real-world implications.

Emotion detection from handwriting and drawing samples using an attention-based transformer model.

Emotion detection (ED) involves the identification and understanding of an individual's emotional state through various cues such as facial expressions, voice tones, physiological changes, and behavioral patterns. In this context, behavioral analysis is employed to observe actions and behaviors for emotional interpretation. This work specifically employs behavioral metrics like drawing and handwriting to determine a person's emotional state, recognizing these actions as physical functions integrating motor and cognitive processes. The study proposes an attention-based transformer model as an innovative approach to identify emotions from handwriting and drawing samples, thereby advancing the capabilities of ED into the domains of fine motor skills and artistic expression. The initial data obtained provides a set of points that correspond to the handwriting or drawing strokes. Each stroke point is subsequently delivered to the attention-based transformer model, which embeds it into a high-dimensional vector space. The model builds a prediction about the emotional state of the person who generated the sample by integrating the most important components and patterns in the input sequence using self-attentional processes. The proposed approach possesses a distinct advantage in its enhanced capacity to capture long-range correlations compared to conventional recurrent neural networks (RNN). This characteristic makes it particularly well-suited for the precise identification of emotions from samples of handwriting and drawings, signifying a notable advancement in the field of emotion detection. The proposed method produced cutting-edge outcomes of 92.64% on the benchmark dataset known as EMOTHAW (Emotion Recognition via Handwriting and Drawing).

Exploiting Data Geometry in Machine Learning

A key challenge in Machine Learning (ML) is the identification of geometric structure in high-dimensional data. Most algorithms assume that data lives in a high-dimensional vector space; however, many applications involve non-Euclidean data, such as graphs, strings and matrices, or data whose structure is determined by symmetries in the underlying system. Here, we discuss methods for identifying geometric structure in data and how leveraging data geometry can give rise to efficient ML algorithms with provable guarantees.

Sentiment Analysis of Short Texts Using SVMs and VSMs-Based Multiclass Semantic Classification

ABSTRACT In our approach, a hybrid machine learning model is proposed which uses Enhanced Vector Space Model (EVSM) along with Hybrid Support Vector Machine (HSVM) classifier. Initially the social media-based information is retrieved using Enhanced Vector Space Model (EVSM). EVSMs are employed in order to characterize the text content by mapping them into high-dimensional vector spaces, capturing the relationships between words and their contextual meanings. Rigorous feature selection methods are employed to designate texts for review, and a multiclass semantic classification algorithm, specifically the HSVM classifier, is utilized for categorization. Decision tree algorithm is used along with SVM to refine the selection process. To enhance sentiment analysis accuracy, sentiment dictionaries are not only presented but also extended through the expansion of Stanford’s GloVE tool. To enhance precision, the proposed work introduces weight-enhancing methods for processing renowned text weights. Sentiments are classified into positive, negative, and neutral categories. Notably, the achieved results demonstrate improved accuracy, attributed to the incorporation of an emotional sentiment enhancement factor for determining weights and leveraging sentiment dictionaries for word availability. The accuracy is obtained to be 92.78% with 91.33% positive sentiment rate and 97.32% negative sentiment rate.

SeRF: Segment Graph for Range-Filtering Approximate Nearest Neighbor Search

Effective vector representation models, e.g., word2vec and node2vec, embed real-world objects such as images and documents in high dimensional vector space. In the meanwhile, the objects are often associated with attributes such as timestamps and prices. Many scenarios need to jointly query the vector representations of the objects together with their attributes. These queries can be formalized as range-filtering approximate nearest neighbor search (ANNS) queries. Specifically, given a collection of data vectors, each associated with an attribute value whose domain has a total order. The range-filtering ANNS consists of a query range and a query vector. It finds the approximate nearest neighbors of the query vector among all the data vectors whose attribute values fall in the query range. Existing approaches suffer from a rapidly degrading query performance when the query range width shifts. The query performance can be optimized by a solution that builds an ANNS index for every possible query range; however, the index time and index size become prohibitive -- the number of query ranges is quadratic to the number n of data vectors. To overcome these challenges, for the query range contains all attribute values smaller than a user-provided threshold, we design a structure called the segment graph whose index time and size are the same as a single ANNS index, yet can losslessly compress the n ANNS indexes, reducing the indexing cost by a factor of Ω(n). To handle general range queries, we propose a 2D segment graph with average-case index size O(n log n) to compress n segment graphs, breaking the quadratic barrier. Extensive experiments conducted on real-world datasets show that our proposed structures outperformed existing methods significantly; our index also exhibits superior scalability.

PREDICTING 10 YEAR MORTALITY WITH MACHINE LEARNING: EVIDENCE FROM NATIONAL SOCIAL LIFE, HEALTH, AND AGING PROJECT

Abstract The current academic literature extensively used linear logistic models with social, behavioral, and psychological status to predict mortality. However, few address the interdependency of predictors and the imbalance of targets that adversely bias the results. Using the National Social Life Health and Aging Project (NSHAP), we developed two machine learning models predicting the 10-year mortality of older adults in the US. We first used tree-based algorithms of Decision Tree (DT) that account for the interdependency of the social features and decide the splitting nodes and thresholds using entropy gain conditional on the previous splitting predictor to discern disposition status. Second, we used the Fuzzy Support Vector Machine (FSVM) that regards every sample as a node in high-dimensional vector space and splits the nodes with an optimum plane by finding the best linear combination of features to get an optimum prediction accuracy. Additionally, FSVM addresses the target imbalance problem by conducting a more delicate classification of samples with close predicted probabilities of being alive and deceased. Compared to the accuracy rates achieved by the Logistic Regression, our algorithms perform better on the entire population and the population near the class boundary. We also discussed the social and demographic characteristics of the cases whose disposition statuses were either wrongly predicted as deceased or alive by our algorithms. The findings serve important purposes for public health practitioners in accurately understanding the risk and protective factors of mortality in aging.

Identifying the Causes of Ship Collisions Accident Using Text Mining and Bayesian Networks

Under the backdrop of the robust growth of the global economy, the water transport industry is experiencing rapid development, resulting in an increase in ship collisions and a critical water traffic safety situation. This study uses text mining techniques to gather a corpus of data. The corpus includes human factors, ship factors, natural environmental factors, and management factors, which are used as target data to obtain a high-dimensional sparse original feature vector space set comprising eigenvalues and eigenvalue weight attributes. Chi-square statistics are utilised to reduce dimensionality, resulting in a final set of 33-dimensional text feature items that determine the causal factors of ship collision risk. Taking the four steps involved in the collision process as the primary focus, a Bayesian network structure for ship collision risk is constructed based on the “human-ship-environment-management” system. By incorporating existing ship collision accident/danger reports, conditional probability tables are computed for each node in the Bayesian network structure, enabling the modelling of ship collision risk. The model is validated through an example, revealing that, under relevant conditions, the probability of collision exceeds 90 %. This finding demonstrates the validity of the model and allows one to deduce the primary cause of ship collision accidents.

A novel scheme based on modified hierarchical time-shift multi-scale amplitude-aware permutation entropy for rolling bearing condition assessment and fault recognition

To avoid production interruptions and equipment damage caused by rolling bearing failure, this study presents a novel diagnosis scheme applicable to both condition monitoring and fault recognition. In detail, to fully exploit feature information, a modified Hierarchical Time-Shift Multi-scale Amplitude-Aware Permutation Entropy (MHTSMAAPE) is put forward to map the raw vibration signal into a high-dimensional vector space of features. This method is developed from the AAPE algorithm and integrates the time-shift procedure and modified hierarchical analysis, extracting more feature details while ensuring entropy stability. Afterwards, the averaged value of the feature vector is regarded as an indicator to assess the condition of rolling bearing, this indicator exceeds a specific value indicates the rolling bearing steps into faulty. Then, the Unsupervised Discriminative Feature Selection (UDFS) algorithm is first introduced to directly pick fault-sensitive features from the vector space, relieving the calculation burden of the fault recognition task. The fault identification algorithm is based on Harris Hawks Optimization-Joint Opposite Selection optimized for Support Vector Machine (HHO-JOS-SVM) to improve classification accuracy. Finally, the proposed scheme fused by MHTSMAAPE, UDFS, and HHO-JOS-SVM is validated in two different cases with four common evaluation metrics, respectively. The experimental results demonstrate its reliability and robustness in rolling bearing condition assessment and fault recognition.

Two-stage framework with improved U-Net based on self-supervised contrastive learning for pavement crack segmentation

After the deep learning method emerged, the automated detection technology of pavement crack images has significantly progressed. The dominant approach is supervised deep learning, which relies on large-scale labeled ground truth. However, the problems are mostly unlabeled original crack images, which are difficult to fully utilize by the supervised deep learning network model. As a representative method of self-supervised learning, contrast learning can learn feature representations from unlabeled data, thus improving the accuracy of downstream tasks. This paper proposes a two-stage framework with improved U-Net based on self-supervised contrastive learning for pavement crack image segmentation. The framework takes improved U-Net as the basic architecture to highlight the significant features of the target segment of fine cracks. U-Net is improved by integrating the residual structure and attention mechanism in the typical U-Net architecture. The framework includes two learning stages: pre-training and fine-tuning. In the pre-training stage, the potential feature representation is learned from the unlabeled crack image. Crack images and pavement background images are used in the training data so that the model learns the distinguishable mapping relationship between crack and its background in the high-dimensional vector space without supervision comparison. In the fine-tuning stage, the network loads the parameters after the pre-training and uses the labeled training data for the retraining. Experimental results show that the proposed two-stage framework significantly improves the performance of crack segmentation accuracy without increasing the number of existing training samples and their labeling.

Code semantic enrichment for deep code search

Code search aims to retrieve code snippets from a large-scale codebase, where the semantics of the searched code match developers’ query intent. Code is a low-level implementation of programming intents, but query is always expressed as clear and high-level semantics, which makes it difficult for DL-based approaches to learn the semantic relationship between them. Through a large-scale empirical analysis on more than 2.2 million pairs of Java code and description, we found that the semantics of code and query can be aligned by enriching code with the descriptions of other code in terms of similar implementation. Based on the finding, we propose a code semantic enrichment approach for deep code search, named SemEnr. Specifically, we first enrich semantics for all code snippets in the training and testing data. We estimated the syntactic similarity of each code snippet from the training data and retrieved the most similar one for each. Thereafter, the semantics of one code snippet is represented by its code tokens and the description of the retrieved most similar code. During the model training, we used the attention mechanism to embed pairs of enriched code and query into the shared high-dimensional vector space. To enhance the quality of our learned representations, we integrated a multi-perspective co-attention mechanism, employing Convolutional Neural Networks (CNNs) to capture local correlations between code and query. Finally, we evaluated the effectiveness of our approach by performing experiments on two extensively used Java datasets. Our experimental results reveal that SemEnr achieves an MRR of 0.698 and 0.631, outperforming the best baseline CAT (a state-of-the-art DL-based model) by 19.93% and 18.83%, respectively. In addition, we conducted a user study involving 50 real-world queries to assess SemEnr’s performance, and the findings suggest that SemEnr outperformed baseline models by returning more relevant code snippets.

Scalable Bayesian optimization with randomized prior networks

Several fundamental problems in science and engineering consist of global optimization tasks involving unknown high-dimensional (black-box) functions that map a set of controllable variables to the outcomes of an expensive experiment. Bayesian Optimization (BO) techniques are known to be effective in tackling global optimization problems using a relatively small number objective function evaluations, but their performance suffers when dealing with high-dimensional outputs. To overcome the major challenge of dimensionality, here we propose a deep learning framework for BO and sequential decision making based on bootstrapped ensembles of neural architectures with randomized priors. Using appropriate architecture choices, we show that the proposed framework can approximate functional relationships between design variables and quantities of interest, even in cases where the latter take values in high-dimensional vector spaces or even infinite-dimensional function spaces. In the context of BO, we augmented the proposed probabilistic surrogates with re-parameterized Monte Carlo approximations of multiple-point (parallel) acquisition functions, as well as methodological extensions for accommodating black-box constraints and multi-fidelity information sources. We test the proposed framework against state-of-the-art methods for BO and demonstrate superior performance across several challenging tasks with high-dimensional outputs, including a constrained multi-fidelity optimization task involving shape optimization of rotor blades in turbo-machinery.

ViSpa (Vision Spaces): A computer-vision-based representation system for individual images and concept prototypes, with large-scale evaluation.

Quantitative, data-driven models for mental representations have long enjoyed popularity and success in psychology (e.g., distributional semantic models in the language domain), but have largely been missing for the visual domain. To overcome this, we present ViSpa (Vision Spaces), high-dimensional vector spaces that include vision-based representation for naturalistic images as well as concept prototypes. These vectors are derived directly from visual stimuli through a deep convolutional neural network trained to classify images and allow us to compute vision-based similarity scores between any pair of images and/or concept prototypes. We successfully evaluate these similarities against human behavioral data in a series of large-scale studies, including off-line judgments-visual similarity judgments for the referents of word pairs (Study 1) and for image pairs (Study 2), and typicality judgments for images given a label (Study 3)-as well as online processing times and error rates in a discrimination (Study 4) and priming task (Study 5) with naturalistic image material. ViSpa similarities predict behavioral data across all tasks, which renders ViSpa a theoretically appealing model for vision-based representations and a valuable research tool for data analysis and the construction of experimental material: ViSpa allows for precise control over experimental material consisting of images and/or words denoting imageable concepts and introduces a specifically vision-based similarity for word pairs. To make ViSpa available to a wide audience, this article (a) includes (video) tutorials on how to use ViSpa in R and (b) presents a user-friendly web interface at http://vispa.fritzguenther.de. (PsycInfo Database Record (c) 2022 APA, all rights reserved).

High-Dimensional Approximate Nearest Neighbor Search: with Reliable and Efficient Distance Comparison Operations

Approximate K nearest neighbor (AKNN) search in the high-dimensional Euclidean vector space is a fundamental and challenging problem. We observe that in high-dimensional space, the time consumption of nearly all AKNN algorithms is dominated by that of the distance comparison operations (DCOs). For each operation, it scans full dimensions of an object and thus, runs in linear time wrt the dimensionality. To speed it up, we propose a randomized algorithm named ADSampling which runs in logarithmic time wrt the dimensionality for the majority of DCOs and succeeds with high probability. In addition, based on ADSampling we develop one generic and two algorithm-specific techniques as plugins to enhance existing AKNN algorithms. Both theoretical and empirical studies confirm that: (1) our techniques introduce nearly no accuracy loss and (2) they consistently improve the efficiency.

Misinformation and Public Health Messaging in the Early Stages of the Mpox Outbreak: Mapping the Twitter Narrative With Deep Learning.

Shortly after the worst of the COVID-19 pandemic, an outbreak of mpox introduced another critical public health emergency. Like the COVID-19 pandemic, the mpox outbreak was characterized by a rising prevalence of public health misinformation on social media, through which many US adults receive and engage with news. Digital misinformation continues to challenge the efforts of public health officials in providing accurate and timely information to the public. We examine the evolving topic distributions of social media narratives during the mpox outbreak to map the tension between rapidly diffusing misinformation and public health communication. This study aims to observe topical themes occurring in a large-scale collection of tweets about mpox using deep learning. We leveraged a data set comprised of all mpox-related tweets that were posted between May 7, 2022, and July 23, 2022. We then applied Sentence Bidirectional Encoder Representations From Transformers (S-BERT) to the content of each tweet to generate a representation of its content in high-dimensional vector space, where semantically similar tweets will be located closely together. We projected the set of tweet embeddings to a 2D map by applying principal component analysis and Uniform Manifold Approximation Projection (UMAP). Finally, we group these data points into 7 topical clusters using k-means clustering and analyze each cluster to determine its dominant topics. We analyze the prevalence of each cluster over time to evaluate longitudinal thematic changes. Our deep-learning pipeline revealed 7 distinct clusters of content: (1) cynicism, (2) exasperation, (3) COVID-19, (4) men who have sex with men, (5) case reports, (6) vaccination, and (7) World Health Organization (WHO). Clusters that largely communicated erroneous or irrelevant information began earlier and grew faster, reaching a wider audience than later communications by official instances and health officials. Within a few weeks of the first reported mpox cases, an avalanche of mostly false, misleading, irrelevant, or damaging information started to circulate on social media. Official institutions, including the WHO, acted promptly, providing case reports and accurate information within weeks, but were overshadowed by rapidly spreading social media chatter. Our results point to the need for real-time monitoring of social media content to optimize responses to public health emergencies.

A hard segmentation network guided by soft segmentation for tumor segmentation on PET/CT images

Cancer is considered one of the leading causes of death. We can detect cancers through the anatomical and functional imaging provided by PET/CT. However, many tumors in PET/CT are obvious in only one modality, and PET contains many non-lesional hypermetabolic regions, which increases the difficulty of segmentation. Furthermore, traditional two-stage segmentation improves segmentation efficiency by breaking down a segmentation task into two independent subtasks. The second stage loses most of the feature information obtained in the first stage. To address these problems, we propose a hard segmentation network guided by soft segmentation for tumor segmentation on PET/CT images. The proposed network has a soft segmentation branch and a hard segmentation branch. The output of the soft segmentation branch is a logits map composed of gradient values, which is corrected with the soft ground truth by the proposed similarity loss function so that the logits map and the soft ground truth are approximately consistent in the high-dimensional vector space. The output of the hard segmentation branch is the final prediction map. The two branches are connected by a soft segmentation-guided mechanism. This guidance mechanism can generate a soft segmentation-guided map with stable distribution according to the logits map obtained by the soft segmentation branch. We validated the proposed network on two datasets. The Dice of 0.7324 on the public soft tissue sarcoma dataset and 0.7693 on the private liver tumor dataset. By only using U-Net as the backbone network, our method achieves the best performance.