Low Intrinsic Dimensionality Research Articles

A novel classification of dyslipidaemia through the analysis of five million lipid profiles: an unsupervised machine learning approach

Abstract Background Dyslipidemia encompasses a wide range of lipoprotein disorders categorised through two classifications (Fredrickson-Levy [FL] or Sniderman).(1,2) However, both classifications are criticised for relying on incomplete knowledge of lipoprotein metabolism, especially with the emergence of novel treatment options and variations in individual treatment responses.(3) Clustering, an unsupervised machine learning (ML) algorithm that can process a wide range of variables, has the potential to unmask patient groups with distinct molecular profiles and unique therapeutic targets that can inform more effective prevention strategies for cardiovascular disease (CVD).(4) Aim We aimed to use unsupervised ML algorithms to discover intrinsic dyslipidaemia categories from lipoprotein measurements, recognise the necessary components of lipid panels for classification, and analyse the similarities between the newly formed clusters, FL and Sniderman classifications. Methods Lipid profiles of 5,080,248 patients were obtained from the ‘Very Large Database of Lipids’ database. This yielded up to 78 blood components per patient, including at least 31 lipoprotein variables. The analysis involved unsupervised K-means clustering with optimised values for K and the subset of variables, determined in an unsupervised manner using a suitable measure of complexity. We then interpreted our clusters using probabilistic decision trees to provide compact and interpretable representations. Finally, we compared the clusters with Sniderman and FL categories. Results In a completely unsupervised fashion, we identified 14 clusters that could be matched to Sniderman categories. The confusion matrix showed total agreement of 76% (see Figure 1, left panel), relative Cohen’s kappa of 0.78 (the relative version captures accuracy on categories containing smaller numbers of patient profiles) and an accuracy of 96% on the small Type III class. Similar results were observed when matching to FL types. We accurately represented our clusters using probabilistic decision trees of small depth (see Figure 2). We discovered that the data had low intrinsic dimension and a manifold-like structure in which the different clusters could be illustrated (see Figure 1, right panel). Specifically, only 3 variables were needed to obtain our classification: apolipoprotein b, total cholesterol and triglycerides. Conclusion We showed that completely unsupervised ML techniques can uncover dyslipidaemia categories in lipoprotein profiles from a large patient population. The categories largely align with existing classifications based on prior knowledge of lipoprotein metabolism. Furthermore, few lipoprotein variables were required for categorisation (low-dimension data), which could aid in determining which lipoproteins should be measured in a clinical setting. Further analysis of the differences between ML clusters and traditional classifications is needed, which may enhance CVD risk management.

Computed tomography of chemiluminescence using a data-driven sparse sensing framework

The aim of this work is to demonstrate the use of data-driven sparse sensing for the reconstruction of the three-dimensional chemiluminescence field of a flame from the two-dimensional line-of-sight integrated chemiluminescence signal. By leveraging the intrinsic low-dimensionality of physical phenomena, sparse sensing can be employed to reconstruct a signal from few samples. This makes it a good candidate to be employed in computed tomography of chemiluminescence, an imaging technique used to reconstruct the chemiluminescence field from chemiluminescence images. In the proposed methodology, the transforming basis used to project the system’s state in the low-dimensional manifold is computed using the proper orthogonal decomposition, which is a popular machine-learning technique for data compression. The methodology is demonstrated on a virtual experiment based on the data coming from the large eddy simulation of a jet flame in a vitiated coflow, where OH is employed as a surrogate of OH∗. This methodology is then compared to conventional techniques that are generally employed to solve tomographic problems. The results show that the proposed technique is able to reconstruct the chemiluminescence field using only one view, compared to the multiple views required by conventional methods. Moreover, our techniques is also capable of predicting the distribution of other features such as temperature and species concentrations.

Towards Metric DBSCAN: Exact, Approximate, and Streaming Algorithms

DBSCAN is a popular density-based clustering algorithm that has many different applications in practice. However, the running time of DBSCAN in high-dimensional space or general metric space (\em e.g., clustering a set of texts by using edit distance) can be as large as quadratic in the input size. Moreover, most of existing accelerating techniques for DBSCAN are only available for low-dimensional Euclidean space. In this paper, we study the DBSCAN problem under the assumption that the inliers (the core points and border points) have a low intrinsic dimension (which is a realistic assumption for many high-dimensional applications), where the outliers can locate anywhere in the space without any assumption. First, we propose a k-center clustering based algorithm that can reduce the time-consuming labeling and merging tasks of DBSCAN to be linear. Further, we propose a linear time approximate DBSCAN algorithm, where the key idea is building a novel small-size summary for the core points. Also, our algorithm can be efficiently implemented for streaming data and the required memory is independent of the input size. Finally, we conduct our experiments and compare our algorithms with several popular DBSCAN algorithms. The experimental results suggest that our proposed approach can significantly reduce the computational complexity in practice.

Neural Network Approximation for Pessimistic Offline Reinforcement Learning

Deep reinforcement learning (RL) has shown remarkable success in specific offline decision-making scenarios, yet its theoretical guarantees are still under development. Existing works on offline RL theory primarily emphasize a few trivial settings, such as linear MDP or general function approximation with strong assumptions and independent data, which lack guidance for practical use. The coupling of deep learning and Bellman residuals makes this problem challenging, in addition to the difficulty of data dependence. In this paper, we establish a non-asymptotic estimation error of pessimistic offline RL using general neural network approximation with C-mixing data regarding the structure of networks, the dimension of datasets, and the concentrability of data coverage, under mild assumptions. Our result shows that the estimation error consists of two parts: the first converges to zero at a desired rate on the sample size with partially controllable concentrability, and the second becomes negligible if the residual constraint is tight. This result demonstrates the explicit efficiency of deep adversarial offline RL frameworks. We utilize the empirical process tool for C-mixing sequences and the neural network approximation theory for the Holder class to achieve this. We also develop methods to bound the Bellman estimation error caused by function approximation with empirical Bellman constraint perturbations. Additionally, we present a result that lessens the curse of dimensionality using data with low intrinsic dimensionality and function classes with low complexity. Our estimation provides valuable insights into the development of deep offline RL and guidance for algorithm model design.

Simple Orthogonal Graph Representation Learning (Student Abstract)

Graph neural networks (GNNs) have attracted significant interest recently since they can effectively process and analyze graph-structured data commonly found in real-world applications. However, the predicament that GNNs are difficult to train becomes worse as the layers increase. The essence of this problem is that stacking layers will reduce the stability of forward propagation and gradient back-propagation. And as the increasing scale of models (measured by the number of parameters), how to efficiently and effectively adapt it to particular downstream tasks becomes an intriguing research issue. In this work, motivated by the effect of orthogonality constraints, we propose a simple orthogonal training framework to impose the orthogonality constraints on GNNs, which can help models find a solution vector in a specific low dimensional subspace and stabilize the signaling processes at both the forward and backward directions. Specifically, we propose a novel polar decomposition-based orthogonal initialization (PDOI-R) algorithm, which can identify the low intrinsic dimension within the Stiefel Manifold and stabilize the training process. Extensive experiments demonstrate the effectiveness of the proposed method in multiple downstream tasks, showcasing its generality. The simple method can help existing state-of-the-art models achieve better performance.

Self-Assembly of Delta-Formamidinium Lead Iodide Nanoparticles to Nanorods: Study of Memristor Properties and Resistive Switching Mechanism.

In the quest for advanced memristor technologies, this study introduces the synthesis of delta-formamidinium lead iodide (δ-FAPbI3) nanoparticles (NPs) and their self-assembly into nanorods (NRs). The formation of these NRs is facilitated by iodide vacancies, promoting the fusion of individual NPs at higher concentrations. Notably, these NRs exhibit robust stability under ambient conditions, a distinctive advantage attributed to the presence of capping ligands and a crystal lattice structured around face-sharing octahedra. When employed as the active layer in resistive random-access memory devices, these NRs demonstrate exceptional bipolar switching properties. A remarkable on/off ratio (105) is achieved, surpassing the performances of previously reported low-dimensional perovskite derivatives and α-FAPbI3 NP-based devices. This enhanced performance is attributed to the low off-state current owing to the reduced number of halide vacancies, intrinsic low dimensionality, and the parallel alignment of NRs on the FTO substrate. This study not only provides significant insights into the development of superior materials for memristor applications but also opens new avenues for exploring low-dimensional perovskite derivatives in advanced electronic devices.

Derivative-Informed Neural Operator: An efficient framework for high-dimensional parametric derivative learning

We propose derivative-informed neural operators (DINOs), a general family of neural networks to approximate operators as infinite-dimensional mappings from input function spaces to output function spaces or quantities of interest. After discretizations both inputs and outputs are high-dimensional. We aim to approximate not only the operators with improved accuracy but also their derivatives (Jacobians) with respect to the input function-valued parameter to empower derivative-based algorithms in many applications, e.g., Bayesian inverse problems, optimization under parameter uncertainty, and optimal experimental design. The major difficulties include the computational cost of generating derivative training data and the high dimensionality of the problem leading to large training cost. To address these challenges, we exploit the intrinsic low-dimensionality of the derivatives and develop algorithms for compressing derivative information and efficiently imposing it in neural operator training yielding derivative-informed neural operators. We demonstrate that these advances can significantly reduce the costs of both data generation and training for large classes of problems (e.g., nonlinear steady state parametric PDE maps), making the costs marginal or comparable to the costs without using derivatives, and in particular independent of the discretization dimension of the input and output functions. Moreover, we show that the proposed DINO achieves significantly higher accuracy than neural operators trained without derivative information, for both function approximation and derivative approximation (e.g., Gauss–Newton Hessian), especially when the training data are limited.

Symplectic model reduction of Hamiltonian systems using data-driven quadratic manifolds

This work presents two novel approaches for the symplectic model reduction of high-dimensional Hamiltonian systems using data-driven quadratic manifolds. Classical symplectic model reduction approaches employ linear symplectic subspaces for representing the high-dimensional system states in a reduced-dimensional coordinate system. While these approximations respect the symplectic nature of Hamiltonian systems, linear basis approximations can suffer from slowly decaying Kolmogorov N-width, especially in wave-type problems, which then requires a large basis size. We propose two different model reduction methods based on recently developed quadratic manifolds, each presenting its own advantages and limitations. The addition of quadratic terms to the state approximation, which sits at the heart of the proposed methodologies, enables us to better represent intrinsic low-dimensionality in the problem at hand. Both approaches are effective for issuing predictions in settings well outside the range of their training data while providing more accurate solutions than the linear symplectic reduced-order models.

Multi‐fidelity data fusion through parameter space reduction with applications to automotive engineering

AbstractMulti‐fidelity models are of great importance due to their capability of fusing information coming from different numerical simulations, surrogates, and sensors. We focus on the approximation of high‐dimensional scalar functions with low intrinsic dimensionality. By introducing a low dimensional bias we can fight the curse of dimensionality affecting these quantities of interest, especially for many‐query applications. We seek a gradient‐based reduction of the parameter space through linear active subspaces or a nonlinear transformation of the input space. Then we build a low‐fidelity response surface based on such reduction, thus enabling nonlinear autoregressive multi‐fidelity Gaussian process regression without the need of running new simulations with simplified physical models. This has a great potential in the data scarcity regime affecting many engineering applications. In this work we present a new multi‐fidelity approach that involves active subspaces and the nonlinear level‐set learning method, starting from the preliminary analysis previously conducted (Romor F, Tezzele M, Rozza G. Proceedings in Applied Mathematics & Mechanics. Wiley Online Library; 2021). The proposed framework is tested on two high‐dimensional benchmark functions, and on a more complex car aerodynamics problem. We show how a low intrinsic dimensionality bias can increase the accuracy of Gaussian process response surfaces.

Deep estimation for Q⁎ with minimax Bellman error minimization

In this paper we consider the estimation of optimal state-action value function Q⁎ with ReLU ResNet based on minimax Bellman error minimization. We construct the non-asymptotic error bounds for the minimax estimator and the estimated Q function induced by the estimated greedy policy. To bound the Bellman residual error, we guarantee the approximation errors based on deep approximation theory and the statistical ones by utilizing empirical processes taking into account the Markov decision process dependency. We provide a novel generalization bound with dependent data and an approximation bound in the Hölder class which are of independent interest. This bound depends on the sample size, the ambient dimension, the width and depth of the neural network, which can bring prior insights into tuning these hyper-parameters to achieve a desired convergence rate in practice. Furthermore, the bound circumvents the curse of dimensionality if the distribution of state-action pairs is assumed to be supported on a set of low intrinsic dimension.

A Data-dependent Approach for High-dimensional (Robust) Wasserstein Alignment

Many real-world problems can be formulated as the alignment between two geometric patterns. Previously, a great amount of research focus on the alignment of two-dimensional (2D) or 3D patterns in the field of computer vision. Recently, the alignment problem in high dimensions finds several novel applications in practice. However, the research is still rather limited in the algorithmic aspect. To the best of our knowledge, most existing approaches are just simple extensions of their counterparts for 2D and 3D cases and often suffer from the issues such as high computational complexities. In this article, we propose an effective framework to compress the high-dimensional geometric patterns. Any existing alignment method can be applied to the compressed geometric patterns and the time complexity can be significantly reduced. Our idea is inspired by the observation that high-dimensional data often has a low intrinsic dimension. Our framework is a “data-dependent” approach that has the complexity depending on the intrinsic dimension of the input data. Our experimental results reveal that running the alignment algorithm on compressed patterns can achieve similar qualities, comparing with the results on the original patterns, but the runtimes (including the times cost for compression) are substantially lower.

Correction to: Just Least Squares: Binary Compressive Sampling with Low Generative Intrinsic Dimension

Correction to: Just Least Squares: Binary Compressive Sampling with Low Generative Intrinsic Dimension

HIPPYlib-MUQ: A Bayesian Inference Software Framework for Integration of Data with Complex Predictive Models under Uncertainty

Bayesian inference provides a systematic framework for integration of data with mathematical models to quantify the uncertainty in the solution of the inverse problem. However, the solution of Bayesian inverse problems governed by complex forward models described bypartial differential equations (PDEs)remains prohibitive with black-boxMarkov chain Monte Carlo (MCMC)methods. We present hIPPYlib-MUQ, an extensible and scalable software framework that contains implementations of state-of-the art algorithms aimed to overcome the challenges of high-dimensional, PDE-constrained Bayesian inverse problems. These algorithms accelerate MCMC sampling by exploiting the geometry and intrinsic low-dimensionality of parameter space via derivative information and low rank approximation. The software integrates two complementary open-source software packages, hIPPYlib and MUQ. hIPPYlib solves PDE-constrained inverse problems using automatically-generated adjoint-based derivatives, but it lacks full Bayesian capabilities. MUQ provides a spectrum of powerful Bayesian inversion models and algorithms, but expects forward models to come equipped with gradients and Hessians to permit large-scale solution. By combining these two complementary libraries, we created a robust, scalable, and efficient software framework that realizes the benefits of each and allows us to tackle complex large-scale Bayesian inverse problems across a broad spectrum of scientific and engineering disciplines. To illustrate the capabilities of hIPPYlib-MUQ, we present a comparison of a number of MCMC methods available in the integrated software on several high-dimensional Bayesian inverse problems. These include problems characterized by both linear and nonlinear PDEs, various noise models, and different parameter dimensions. The results demonstrate that large (∼ 50×) speedups over conventional black box and gradient-based MCMC algorithms can be obtained by exploiting Hessian information (from the log-posterior), underscoring the power of the integrated hIPPYlib-MUQ framework.

Large-Scale Bayesian Optimal Experimental Design with Derivative-Informed Projected Neural Network

We address the solution of large-scale Bayesian optimal experimental design (OED) problems governed by partial differential equations (PDEs) with infinite-dimensional parameter fields. The OED problem seeks to find sensor locations that maximize the expected information gain (EIG) in the solution of the underlying Bayesian inverse problem. Computation of the EIG is usually prohibitive for PDE-based OED problems. To make the evaluation of the EIG tractable, we approximate the (PDE-based) parameter-to-observable map with a derivative-informed projected neural network (DIPNet) surrogate, which exploits the geometry, smoothness, and intrinsic low-dimensionality of the map using a small and dimension-independent number of PDE solves. The surrogate is then deployed within a greedy algorithm-based solution of the OED problem such that no further PDE solves are required. We analyze the EIG approximation error in terms of the generalization error of the DIPNet and show they are of the same order. Finally, the efficiency and accuracy of the method are demonstrated via numerical experiments on OED problems governed by inverse scattering and inverse reactive transport with up to 16,641 uncertain parameters and 100 experimental design variables, where we observe up to three orders of magnitude speedup relative to a reference double loop Monte Carlo method.

Just Least Squares: Binary Compressive Sampling with Low Generative Intrinsic Dimension

In this paper, we consider recovering $$n-$$ dimensional signals from m binary measurements corrupted by noises and sign flips under the assumption that the target signals have low generative intrinsic dimension, i.e., the target signals can be approximately generated via an L-Lipschitz generator $$G: \mathbb {R}^k\rightarrow \mathbb {R}^{n}, k\ll n$$ . Although the binary measurements model is highly nonlinear, we propose a least square decoder and prove that, up to a constant c, with high probability, the least square decoder achieves a sharp estimation error $$C\sqrt{\frac{k\log (Ln)}{m}}$$ as long as $$m\ge C( k\log (Ln))$$ . Extensive numerical simulations and comparisons with state-of-the-art methods demonstrated the least square decoder is robust to noise and sign flips, as indicated by our theory. By constructing a ReLU network with properly chosen depth and width, we verify the (approximately) deep generative prior, which is of independent interest.

High-dimensional automated radiation therapy treatment planning via Bayesian optimization.

Radiation therapy treatment planning can be viewed as an iterative hyperparameter tuning process to balance conflicting clinical goals. In this work, we investigated the performance of modern Bayesian optimization (BO) methods on automated treatment planning problems in high-dimensionalsettings. Twenty locally advanced rectal cancer patients treated with intensity-modulated radiation therapy (IMRT) were retrospectively selected as test cases. The adjustable planning parameters included both dose objectives and their corresponding weights. We implemented an automated treatment planning framework and tested the performance of two BO methods on the treatment planning task: one standard BO method (Gaussian Process with Expected Improvement [GPEI]) and one BO method dedicated to high-dimensional problems (Sparse Axis Aligned Subspace BO [SAAS-BO]). Another derivative-free method (Nelder-Mead simplex search) and the random tuning method were also included as baselines. The four automated methods' plan quality and planning efficiency were compared with the clinical plans regarding target coverage and organs at risk (OAR) sparing. The predictive models in both BO methods were compared to analyze the different search patterns of the two BOmethods. For the target structures, the SAAS-BO plans achieved comparable hot spot control ( ) and homogeneity ( ) with the clinical plans, significantly better than the GPEI and Nelder-Mead plans ( ). Both SAAS-BO and GPEI plans significantly outperformed the clinical plans in conformity and dose spillage ( ). Compared with the clinical plans, the treatment plans generated by the four automated methods all made reductions in evaluated dosimetric indices for the femoral head and the bladder. The Nelder-Mead plans achieved similar plan quality scores compared with the BO plans, but exhibited poorer control in the target hot spot and dose spillage. The analysis of the underlying predictive models has shown that both BO methods have identified similar sensitive planning parameters. This work implemented a BO-based hyperparameter tuning framework for automated treatment planning. Both tested BO methods were able to produce high-quality treatment plans and reduce the workload of treatment planners. The model analysis also confirmed the intrinsic low dimensionality of the tested treatment planningproblems.

Inverse problems on low-dimensional manifolds

We consider abstract inverse problems between infinite-dimensional Banach spaces. These inverse problems are typically nonlinear and ill-posed, making the inversion with limited and noisy measurements a delicate process. In this work, we assume that the unknown belongs to a finite-dimensional manifold: this assumption arises in many real-world scenarios where natural objects have a low intrinsic dimension and belong to a certain submanifold of a much larger ambient space. We prove uniqueness and Hölder and Lipschitz stability results in this general setting, also in the case when only a finite discretization of the measurements is available. Then, a Landweber-type reconstruction algorithm from a finite number of measurements is proposed, for which we prove global convergence, thanks to a new criterion for finding a suitable initial guess.These general results are then applied to several examples, including two classical nonlinear ill-posed inverse boundary value problems. The first is Calderón’s inverse conductivity problem, for which we prove a Lipschitz stability estimate from a finite number of measurements for piece-wise constant conductivities with discontinuities on an unknown triangle. A similar stability result is then obtained for Gel’fand-Calderón’s problem for the Schrödinger equation, in the case of piece-wise constant potentials with discontinuities on a finite number of non-intersecting balls.

Bound-constrained global optimization of functions with low effective dimensionality using multiple random embeddings

We consider the bound-constrained global optimization of functions with low effective dimensionality, that are constant along an (unknown) linear subspace and only vary over the effective (complement) subspace. We aim to implicitly explore the intrinsic low dimensionality of the constrained landscape using feasible random embeddings, in order to understand and improve the scalability of algorithms for the global optimization of these special-structure problems. A reduced subproblem formulation is investigated that solves the original problem over a random low-dimensional subspace subject to affine constraints, so as to preserve feasibility with respect to the given domain. Under reasonable assumptions, we show that the probability that the reduced problem is successful in solving the original, full-dimensional problem is positive. Furthermore, in the case when the objective’s effective subspace is aligned with the coordinate axes, we provide an asymptotic bound on this success probability that captures its polynomial dependence on the effective and, surprisingly, ambient dimensions. We then propose X-REGO, a generic algorithmic framework that uses multiple random embeddings, solving the above reduced problem repeatedly, approximately and possibly, adaptively. Using the success probability of the reduced subproblems, we prove that X-REGO converges globally, with probability one, and linearly in the number of embeddings, to an epsilon -neighbourhood of a constrained global minimizer. Our numerical experiments on special structure functions illustrate our theoretical findings and the improved scalability of X-REGO variants when coupled with state-of-the-art global—and even local—optimization solvers for the subproblems.

Dynamics of Drosophila endoderm specification

During early Drosophila embryogenesis, a network of gene regulatory interactions orchestrates terminal patterning, playing a critical role in the subsequent formation of the gut. We utilized CRISPR gene editing at endogenous loci to create live reporters of transcription and light-sheet microscopy to monitor the individual components of the posterior gut patterning network across 90 min prior to gastrulation. We developed a computational approach for fusing imaging datasets of the individual components into a common multivariable trajectory. Data fusion revealed low intrinsic dimensionality of posterior patterning and cell fate specification in wild-type embryos. The simple structure that we uncovered allowed us to construct a model of interactions within the posterior patterning regulatory network and make testable predictions about its dynamics at the protein level. The presented data fusion strategy is a step toward establishing a unified framework that would explore how stochastic spatiotemporal signals give rise to highly reproducible morphogenetic outcomes.

Functional principal subspace sampling for large scale functional data analysis

Functional data analysis (FDA) methods have computational and theoretical appeals for some high dimensional data, but lack the scalability to modern large sample datasets. To tackle the challenge, we develop randomized algorithms for two important FDA methods: functional principal component analysis (FPCA) and functional linear regression (FLR) with scalar response. The two methods are connected as they both rely on the accurate estimation of functional principal subspace. The proposed algorithms draw subsamples from the large dataset at hand and apply FPCA or FLR over the subsamples to reduce the computational cost. To effectively preserve subspace information in the subsamples, we propose a functional principal subspace sampling probability, which removes the eigenvalue scale effect inside the functional principal subspace and properly weights the residual. Based on the operator perturbation analysis, we show the proposed probability has precise control over the first order error of the subspace projection operator and can be interpreted as an importance sampling for functional subspace estimation. Moreover, concentration bounds for the proposed algorithms are established to reflect the low intrinsic dimension nature of functional data in an infinite dimensional space. The effectiveness of the proposed algorithms is demonstrated upon synthetic and real datasets.