High-dimensional Vectors Research Articles

Modeling and Learning on High-Dimensional Matrix-Variate Sequences

We propose a new matrix factor model, named RaDFaM, which is strictly derived from the general rank decomposition and assumes a high-dimensional vector factor model structure for each basis vector. RaDFaM contributes a novel class of low-rank latent structures that trade off between signal intensity and dimension reduction from a tensor subspace perspective. Based on the intrinsic separable covariance structure of RaDFaM, for a collection of matrix-valued observations, we derive a new class of PCA variants for estimating loading matrices, and sequentially the latent factor matrices. The peak signal-to-noise ratio of RaDFaM is proved to be superior in the category of PCA-type estimators. We also establish an asymptotic theory including the consistency, convergence rates, and asymptotic distributions for components in the signal part. Numerically, we demonstrate the performance of RaDFaM in applications such as matrix reconstruction, supervised learning, and clustering, on uncorrelated and correlated data, respectively. Supplementary materials for this article are available online, including a standardized description of the materials available for reproducing the work.

Towards Robust Multi-Agent Reinforcement Learning

Stochastic gradient descent (SGD) is at the heart of large-scale distributed machine learning paradigms such as federated learning (FL). In these applications, the task of training high-dimensional weight vectors is distributed among several workers that exchange information over networks of limited bandwidth. While parallelization at such an immense scale helps to reduce the computational burden, it creates several other challenges: delays, asynchrony, and most importantly, a significant communication bottleneck. The popularity and success of SGD can be attributed in no small part to the fact that it is extremely robust to such deviations from ideal operating conditions. Inspired by these findings, we ask: Are common reinforcement learning (RL) algorithms also robust to similarly structured perturbations? Perhaps surprisingly, despite the recent surge of interest in multi-agent/federated RL, almost nothing is known about the above question. This paper collects some of our recent results in filling this void.

Linear Codes for Hyperdimensional Computing.

Hyperdimensional computing (HDC) is an emerging computational paradigm for representing compositional information as high-dimensional vectors and has a promising potential in applications ranging from machine learning to neuromorphic computing. One of the long-standing challenges in HDC is factoring a compositional representation to its constituent factors, also known as the recovery problem. In this article, we take a novel approach to solve the recovery problem and propose the use of random linear codes. These codes are subspaces over the Boolean field and are a well-studied topic in information theory with various applications in digital communication. We begin by showing that hyperdimensional encoding using random linear codes retains favorable properties of the prevalent (ordinary) random codes; hence, HD representations using the two methods have comparable information storage capabilities. We proceed to show that random linear codes offer a rich subcode structure that can be used to form key-value stores, which encapsulate the most used cases of HDC. Most important, we show that under the framework we develop, random linear codes admit simple recovery algorithms to factor (either bundled or bound) compositional representations. The former relies on constructing certain linear equationsystems over the Boolean field, the solution to which reduces the search space dramatically and strictly outperforms exhaustive search in many cases. The latter employs the subspace structure of these codes to achieve provably correct factorization. Both methods are strictly faster than the state-of-the-art resonator networks, often by an order of magnitude. We implemented our techniques in Python using a benchmark software library and demonstrated promising experimental results.

Transductive zero-shot learning with generative model-driven structure alignment

Zero-shot learning (ZSL) facilitates the transfer of knowledge from seen to unseen categories through high-dimensional vectors that capture both known and unknown class names. However it encounters challenges with domain shift arising from a lack of sufficient labeled data. Although transductive zero-shot learning (TZSL) addresses this bias by including samples from unseen classes, it still faces obstacles in enhancing TZSL performance. In this study, We introduce the Structure Alignment Variational Autoencoder Generative Adversarial Network (SA-VAEGAN), a novel approach that enhances the alignment between visual and auxiliary spaces. We delved into the underlying causes of domain shift and introduced a structural alignment (SA) strategy to tackle these challenges. The SA model thoroughly accounts for both inter-class and intra-class dynamics, designed to leverage the model’s comprehension of high-level semantic relations to disambiguate confusion among similar classes and mitigate intra-class confusion by penalizing atypical visual samples within classes. Assessed across four benchmark datasets, SA-VAEGAN has established a new performance standard, underscoring its efficiency in addressing the domain shift challenge within TZSL tasks, and achieving high accuracy.

From hearing to seeing: Linking auditory and visual place perceptions with soundscape-to-image generative artificial intelligence

People experience the world through multiple senses simultaneously, contributing to our sense of place. Prior quantitative geography studies have mostly emphasized human visual perceptions, neglecting human auditory perceptions at place due to the challenges in characterizing the acoustic environment vividly. Also, few studies have synthesized the two-dimensional (auditory and visual) perceptions in understanding human sense of place. To bridge these gaps, we propose a Soundscape-to-Image Diffusion model, a generative Artificial Intelligence (AI) model supported by Large Language Models (LLMs), aiming to visualize soundscapes through the generation of street view images. By creating audio-image pairs, acoustic environments are first represented as high-dimensional semantic audio vectors. Our proposed Soundscape-to-Image Diffusion model, which contains a Low-Resolution Diffusion Model and a Super-Resolution Diffusion Model, can then translate those semantic audio vectors into visual representations of place effectively. We evaluated our proposed model by using both machine-based and human-centered approaches. We proved that the generated street view images align with our common perceptions, and accurately create several key street elements of the original soundscapes. It also demonstrates that soundscapes provide sufficient visual information places. This study stands at the forefront of the intersection between generative AI and human geography, demonstrating how human multi-sensory experiences can be linked. We aim to enrich geospatial data science and AI studies with human experiences. It has the potential to inform multiple domains such as human geography, environmental psychology, and urban design and planning, as well as advancing our knowledge of human-environment relationships.

Random walk theory and application

This project presents an overview of Random Walk Theory and its applications, as discussed in the provided project work. Random Walk Theory posits that changes in elements like stock prices follow a distribution independent of past movements, making future predictions challenging. Originating from the work of French mathematician Louis Bachelier and later popularized by economist Burton Markiel, the theory finds extensive applications beyond finance, spanning fields such as psychology, economics, and physics. The project delves into various types of random walks, including symmetric random walks, and explores their implications in different spaces, from graphs to higher-dimensional vector spaces. It provides definitions, examples, and graphical representations to elucidate random walk concepts, highlighting their relevance in practical scenarios like particle movement and stock price fluctuations. Key concepts such as the reflection principle and the main lemma are discussed to provide a comprehensive understanding of random walks and their properties. Through examples and lemmas, the project elucidates the mathematical foundations of random walks, offering insights into their behavior and applications across diverse disciplines. In summary, this project contributes to a deeper comprehension of Random Walk Theory, serving as a fundamental framework for understanding stochastic processes and their real-world implications.

Design and Evaluation of English Vocabulary Learning Aids Based on Word Vector Modelling

English vocabulary learning aids based on word vector modeling involve creating tools that leverage advanced techniques to enhance vocabulary acquisition. Word vector modeling, often using methods like Word2Vec or GloVe, represents words as high-dimensional vectors capturing semantic relationships. These models can power vocabulary learning aids by offering context-based word suggestions, personalized word quizzes, or interactive visualizations of word associations.. The paper introduces the Hierarchical Spiking Vocabulary Deep Learning (HSV-DL) framework, a novel approach aimed at enhancing vocabulary learning and classification tasks. With word vector modeling and spiking neural networks, HSV-DL offers a sophisticated methodology for accurately categorizing vocabulary words into their respective semantic categories. The Hierarchical Spiking Vocabulary Deep Learning (HSV-DL) framework introduces novel methodologies for vocabulary learning and classification tasks, achieving outstanding performance metrics. Experimental results demonstrate high accuracy (95%), precision (96%), recall (94%), and F1-score (95%) in categorizing vocabulary words into their semantic categories. Moreover, HSV-DL exhibits robustness to noise and efficient resource utilization, showcasing its potential for real-world applications in natural language processing.

НЕЙРОННА МЕРЕЖА ТИПУ АВТОКОДУВАЛЬНИК ДЛЯ ВКЛАДЕННЯ ОДНОВИМІРНИХ ЧАСОВИХ РЯДІВ

The problem of time series embedding is a universal one. It is the main prerequisite when it comes to modeling of dynamical processes using systems of autonomous ordinary differential equations (ODEs) because they have hard requirements for the dimensionality of the problem. One-dimensional ODE can only exhibit 3 types of behavior while two-dimensional ODE can exhibit 9. This is why it is important to increase the dimensionality of the problem before starting the modeling to allow for wider range of possible behaviors in the final model. One way to increase the dimensionality is to delay-embed the time series data but this approach can be extended to allow the use of an autoencoder neural network that would associate a higher-dimensional vector to each point in the time series and will allow the modeling to be performed in higher dimension.

A boosted degradation representation learning for blind image super-resolution

The significant gap between the assumed and actual degradation model will undoubtedly lead to a serious performance decrease for deep learning based image super-resolution (SR) methods. To solve this shortcoming, this paper presents an enhanced blind image super-resolution (EBSR) network based on unsupervised degradation representation learning. The proposed EBSR mainly is made up of two branching subnetworks. The first subnetwork is a residual-based degradation encoder, which is responsible to learn a high-dimensional abstract degradation representation vector from the input low-resolution (LR) image patches using contrast learning. Another branch is the degradation-aware fusion SR (DAFSR) network that completes the image SR task with the aid of the degradation representation vector learned by the encoder. To obtain an improved SR performance under various degradation settings, the degradation-aware fusion block (DAFB), the core block of DAFSR, has been elaborated, which embeds the degradation model information learned by the encoder into every layer of the network. Additionally, a shallow feature extraction block (SFEB) is constructed for DAFSR to extract global information from the LR image. The proposed EBSR can flexibly adapt to various degradation models. Extensive experimental results on synthetic datasets and real images show that the proposed EBSR is able to achieve a leading reconstruction performance over other blind SR algorithms.

Statistical Inference For Noisy Matrix Completion Incorporating Auxiliary Information

This article investigates statistical inference for noisy matrix completion in a semi-supervised model when auxiliary covariates are available. The model consists of two parts. One part is a low-rank matrix induced by unobserved latent factors; the other part models the effects of the observed covariates through a coefficient matrix which is composed of high-dimensional column vectors. We model the observational pattern of the responses through a logistic regression of the covariates, and allow its probability to go to zero as the sample size increases. We apply an iterative least squares (LS) estimation approach in our considered context. The iterative LS methods in general enjoy a low computational cost, but deriving the statistical properties of the resulting estimators is a challenging task. We show that our method only needs a few iterations, and the resulting entry-wise estimators of the low-rank matrix and the coefficient matrix are guaranteed to have asymptotic normal distributions. As a result, individual inference can be conducted for each entry of the unknown matrices. We also propose a simultaneous testing procedure with multiplier bootstrap for the high-dimensional coefficient matrix. This simultaneous inferential tool can help us further investigate the effects of covariates for the prediction of missing entries. Supplementary materials for this article are available online, including a standardized description of the materials available for reproducing the work.

Learning scene-vectors for remote sensing image scene classification

Representing the scenes by learning the subtle variations in the spatial content of different classes is crucial for scene classification in remote sensing images. In this paper, we propose a scene attribute modeling to obtain a discriminative and compact representation for scene classification. First, we construct a scene attribute model (SAM) by training a Gaussian mixture model (GMM) using convolutional features to capture the scene attributes implicitly. Then, we perform a maximum a posteriori (MAP) adaptation to enhance the contribution of significant attributes in each scene resulting in a high-dimensional feature vector which contains redundant attributes from all the scenes. Hence, we use factor analysis to obtain a compact representation of the high-dimensional feature vector termed scene-vector, which retains only the significant attributes specific to a scene. The proposed approach is demonstrated on three benchmark datasets, namely, UC Merced Land Use, AID, and NWPU-RESISC45 datasets. We further show that, being a compact representation, our scene-vector outperforms state-of-the-art methods for scene classification in remote sensing images.

Personalized Dynamic Pricing Based on Improved Thompson Sampling

This study investigates personalized pricing with demand learning. We first encode consumer-personalized feature information into high-dimensional vectors, then establish the relationship between this feature vector and product demand using a logit model, and finally learn demand parameters through historical transaction data. To address the balance between learning and revenue, we introduce the Thompson Sampling algorithm. Considering the difficulty of Bayesian inference in Thompson Sampling owing to high-dimensional feature vectors, we improve the basic Thompson Sampling by approximating the likelihood function of the logit model with the Pólya-Gamma (PG) distribution and by proposing a Thompson Sampling algorithm based on the PG distribution. To validate the proposed algorithm’s effectiveness, we conduct experiments using both simulated data and real loan data provided by the Columbia University Revenue Management Center. The study results demonstrate that the Thompson Sampling algorithm based on the PG distribution proposed outperforms traditional Laplace approximation methods regarding convergence speed and regret value in both real and simulated data experiments. The real-time personalized pricing algorithm developed here not only enriches the theoretical research of personalized dynamic pricing, but also provides a theoretical basis and guidance for enterprises to implement personalized pricing.

Robust multiple focusing through scattering media via feedback wavefront shaping

The control of multiply-scattered light has greatly boosted the understanding of optics in complex photonic systems, leading to many applications including imaging, information processing, random lasing, to name just a few. Here, we demonstrate a real-time feedback approach to realize multiple focusing of multiply-scattered light, which is more robust to the inherent background field compared to the time reversal approach. Our method retrieves the optimized wavefront by aligning the high-dimensional vector of the measured pattern with that of the target pattern. Our theoretical simulations and experimental results show that the feedback approach is capable of constructing homogeneous diffraction-limited spots at predefined positions. Moreover, the peak-to-background ratio exhibits a sub-1/m decay with m being the number of focuses, indicating the excellent capability of the proposed approach to focus scattered light at multiple positions. This efficient approach provides a promising avenue toward complex control on scattered light in complex photonic media, and therefore would be useful in relevant applications such as optical imaging and optical manipulation.

Assessing Performance of Tests for Equality of High-Dimensional Mean Vectors with Unequal Block Diagonal Covariances

Assessing Performance of Tests for Equality of High-Dimensional Mean Vectors with Unequal Block Diagonal Covariances

The Risk Spillover Effects and Network Connectedness Between Real Estate and Other Sectors in China

Using a high-dimensional time-varying parameter vector autoregression framework, we propose a volatility spillover network to analyze risk contagion direction, path, and intensity under supply-demand relationships when important events break out in the real estate market. We characterize daily return volatility spillovers across firms from January 2010 to April 2022 and find a high correlation among these firms because the real linkage based on industrial chain structure strengthened. Meanwhile, the outbreak of essential events in the real estate market corresponds to several peak values of return volatility. The empirical results show that excessive correlation induces risk contagion across the network, and spillover effects are significant under extreme market conditions, manifesting as cross-industry contagion and intersectoral diffusion. The volatility spillover effects within real estate firms are higher than spillover effects across markets. In addition, the industrial chain structure better explains the mechanism of cross-industry risk contagion, firms located upstream of the real estate industry chain are risk receivers, and those located downstream are risk spillovers. Real estate firms mainly spread risks to financial institutions and firms that provide production goods. Our findings are of constructive significance to the related policies by regulatory authorities and provide multi-angle empirical evidence support for the risk supervision theory of “too connected to fail.”

Scalable gradients enable Hamiltonian Monte Carlo sampling for phylodynamic inference under episodic birth-death-sampling models.

Birth-death models play a key role in phylodynamic analysis for their interpretation in terms of key epidemiological parameters. In particular, models with piecewise-constant rates varying at different epochs in time, to which we refer as episodic birth-death-sampling (EBDS) models, are valuable for their reflection of changing transmission dynamics over time. A challenge, however, that persists with current time-varying model inference procedures is their lack of computational efficiency. This limitation hinders the full utilization of these models in large-scale phylodynamic analyses, especially when dealing with high-dimensional parameter vectors that exhibit strong correlations. We present here a linear-time algorithm to compute the gradient of the birth-death model sampling density with respect to all time-varying parameters, and we implement this algorithm within a gradient-based Hamiltonian Monte Carlo (HMC) sampler to alleviate the computational burden of conducting inference under a wide variety of structures of, as well as priors for, EBDS processes. We assess this approach using three different real world data examples, including the HIV epidemic in Odesa, Ukraine, seasonal influenza A/H3N2 virus dynamics in New York state, America, and Ebola outbreak in West Africa. HMC sampling exhibits a substantial efficiency boost, delivering a 10- to 200-fold increase in minimum effective sample size per unit-time, in comparison to a Metropolis-Hastings-based approach. Additionally, we show the robustness of our implementation in both allowing for flexible prior choices and in modeling the transmission dynamics of various pathogens by accurately capturing the changing trend of viral effective reproductive number.

Emotion detection from handwriting and drawing samples using an attention-based transformer model.

Emotion detection (ED) involves the identification and understanding of an individual's emotional state through various cues such as facial expressions, voice tones, physiological changes, and behavioral patterns. In this context, behavioral analysis is employed to observe actions and behaviors for emotional interpretation. This work specifically employs behavioral metrics like drawing and handwriting to determine a person's emotional state, recognizing these actions as physical functions integrating motor and cognitive processes. The study proposes an attention-based transformer model as an innovative approach to identify emotions from handwriting and drawing samples, thereby advancing the capabilities of ED into the domains of fine motor skills and artistic expression. The initial data obtained provides a set of points that correspond to the handwriting or drawing strokes. Each stroke point is subsequently delivered to the attention-based transformer model, which embeds it into a high-dimensional vector space. The model builds a prediction about the emotional state of the person who generated the sample by integrating the most important components and patterns in the input sequence using self-attentional processes. The proposed approach possesses a distinct advantage in its enhanced capacity to capture long-range correlations compared to conventional recurrent neural networks (RNN). This characteristic makes it particularly well-suited for the precise identification of emotions from samples of handwriting and drawings, signifying a notable advancement in the field of emotion detection. The proposed method produced cutting-edge outcomes of 92.64% on the benchmark dataset known as EMOTHAW (Emotion Recognition via Handwriting and Drawing).

Efficient data integration under prior probability shift.

Conventional supervised learning usually operates under the premise that data are collected from the same underlying population. However, challenges may arise when integrating new data from different populations, resulting in a phenomenon known as dataset shift. This paper focuses on prior probability shift, where the distribution of the outcome varies across datasets but the conditional distribution of features given the outcome remains the same. To tackle the challenges posed by such shift, we propose an estimation algorithm that can efficiently combine information from multiple sources. Unlike existing methods that are restricted to discrete outcomes, the proposed approach accommodates both discrete and continuous outcomes. It also handles high-dimensional covariate vectors through variable selection using an adaptive least absolute shrinkage and selection operator penalty, producing efficient estimates that possess the oracle property. Moreover, a novel semiparametric likelihood ratio test is proposed to check the validity of prior probability shift assumptions by embedding the null conditional density function into Neyman's smooth alternatives (Neyman, 1937) and testing study-specific parameters. We demonstrate the effectiveness of our proposed method through extensive simulations and a real data example. The proposed methods serve as a useful addition to the repertoire of tools for dealing dataset shifts.

Exploiting Data Geometry in Machine Learning

A key challenge in Machine Learning (ML) is the identification of geometric structure in high-dimensional data. Most algorithms assume that data lives in a high-dimensional vector space; however, many applications involve non-Euclidean data, such as graphs, strings and matrices, or data whose structure is determined by symmetries in the underlying system. Here, we discuss methods for identifying geometric structure in data and how leveraging data geometry can give rise to efficient ML algorithms with provable guarantees.

Multi-sample hypothesis testing of high-dimensional mean vectors under covariance heterogeneity

Multi-sample hypothesis testing of high-dimensional mean vectors under covariance heterogeneity