High-dimensional Problems Research Articles

Transfer learning for high-dimensional linear regression via the elastic net

In this paper, the high-dimensional linear regression problem is explored via the Elastic Net under the transfer learning framework. Within this framework, potentially related source datasets are leveraged to enhance estimation or prediction beyond what can be achieved solely with the target data. When transferable sources are known, an oracle transfer learning algorithm is proposed based on the Elastic Net. Additionally, the ℓ1/ℓ2 estimation error bounds for the corresponding estimator are established. When the transferable sources are unknown, a novel procedure for detecting transferable sources via the Elastic Net is also proposed, with its selection consistency demonstrated under regular conditions. This method transforms the source detection problem into a variable selection problem in high-dimensional space and always gets results that are consistent with the true outcomes. The performance of these methods is further demonstrated through a variety of numerical examples. Finally, our approach is applied to analyze several real datasets for illustrative purposes.

CATOM : Causal Topology Map for Spatiotemporal Traffic Analysis with Granger Causality in Urban Areas.

The transportation network is an important element in an urban system that supports daily activities, enabling people to travel from one place to another. One of the key challenges is the network complexity, which is composed of many node pairs distributed over the area. This spatial characteristic results in the high dimensional network problem in understanding the 'cause' of problems such as traffic congestion. Recent studies have proposed visual analytics systems aimed at understanding these underlying causes. Despite these efforts, the analysis of such causes is limited to identified patterns. However, given the intricate distribution of roads and their mutual influence, new patterns continuously emerge across all roads within urban transportation. At this stage, a well-defined visual analytics system can be a good solution for transportation practitioners. In this paper, we propose a system, CATOM (Causal Topology Map), for the cause-effect analysis of traffic patterns based on Granger causality for extracting causal topology maps. CATOM discovers causal relationships between roads through the Granger causality test and quantifies these relationships through the causal density. During the design process, the system was developed to fully utilize spatial information with visualization techniques to overcome the previous problems in the literature. We also evaluate the usability of our approach by conducting a SUS(System Usability Scale) test and traffic cause analysis with the real-world data from two study sites in collaboration with domain experts.

Real-Time Policy Optimization for UAV Swarms Based on Evolution Strategies

Multi-agent decision-making faces many challenges such as non-stationarity and sparse rewards, while the complexity and randomness of the real environment further complicate policy development. This paper addresses the high-dimensional policy optimization problems of unmanned aerial vehicle (UAV) swarms. By modeling the problem scenario as a Markov decision process, a real-time policy optimization algorithm based on evolution strategy (ES) pre-training is proposed. This approach combines decision-time planning with background planning to evaluate and integrate different sets of policy parameters in a temporal context. In the experimental phase, the policy network is trained using both ES and REINFORCE algorithms on a constructed simulation platform. Comparative experiments demonstrate the effectiveness of using ES for policy pre-training. Finally, the proposed real-time policy optimization algorithm further improves the performance of the swarm by approximately 10% in simulations, offering a feasible solution for adversarial games between swarms and extending the research scope of evolutionary algorithms.

Enhancing IoT Security Using GA-HDLAD: A Hybrid Deep Learning Approach for Anomaly Detection

The adoption and use of the Internet of Things (IoT) have increased rapidly over recent years, and cyber threats in IoT devices have also become more common. Thus, the development of a system that can effectively identify malicious attacks and reduce security threats in IoT devices has become a topic of great importance. One of the most serious threats comes from botnets, which commonly attack IoT devices by interrupting the networks required for the devices to run. There are a number of methods that can be used to improve security by identifying unknown patterns in IoT networks, including deep learning and machine learning approaches. In this study, an algorithm named the genetic algorithm with hybrid deep learning-based anomaly detection (GA-HDLAD) is developed, with the aim of improving security by identifying botnets within the IoT environment. The GA-HDLAD technique addresses the problem of high dimensionality by using a genetic algorithm during feature selection. Hybrid deep learning is used to detect botnets; the approach is a combination of recurrent neural networks (RNNs), feature extraction techniques (FETs), and attention concepts. Botnet attacks commonly involve complex patterns that the hybrid deep learning (HDL) method can detect. Moreover, the use of FETs in the model ensures that features can be effectively extracted from spatial data, while temporal dependencies are captured by RNNs. Simulated annealing (SA) is utilized to select the hyperparameters necessary for the HDL approach. In this study, the GA-HDLAD system is experimentally assessed using a benchmark botnet dataset, and the findings reveal that the system provides superior results in comparison to existing detection methods.

Functional EWMA control chart for phase II profile monitoring

Due to rapid developments in technology, profile monitoring is becoming challenging, as the profile data are a series of observations on a specific quality characteristic that are taken across some continuum. In this paper, a functional exponentially weighted moving average (FEWMA) control chart is developed for phase-II monitoring of nonlinear profiles based on the functional data approach. In particular, the profiles are estimated using cubic B-splines to solve the problem of high dimensionality that hinders the implementation of classical multivariate control charts. An extensive simulation study has been conducted to validate the effectiveness of the proposed control scheme for detecting changes in the functional mean of the process to its magnitude and assuming scenarios of independence and dependence between curves. The properties of the run-length distribution of the proposed scheme are discussed. The FEWMA control chart is also applied to the real-time data of the vertical density profile of particleboard manufacturing.

A Local Macroscopic Conservative (LoMaC) Low Rank Tensor Method for the Vlasov Dynamics

In this paper, we propose a novel Local Macroscopic Conservative (LoMaC) low rank tensor method for simulating the Vlasov-Poisson (VP) system. The LoMaC property refers to the exact local conservation of macroscopic mass, momentum and energy at the discrete level. This is a follow-up work of our previous development of a conservative low rank tensor approach for Vlasov dynamics (arXiv:2201.10397). In that work, we applied a low rank tensor method with a conservative singular value decomposition to the high dimensional VP system to mitigate the curse of dimensionality, while maintaining the local conservation of mass and momentum. However, energy conservation is not guaranteed, which is a critical property to avoid unphysical plasma self-heating or cooling. The new ingredient in the LoMaC low rank tensor algorithm is that we simultaneously evolve the macroscopic conservation laws of mass, momentum and energy using a flux-difference form with kinetic flux vector splitting; then the LoMaC property is realized by projecting the low rank kinetic solution onto a subspace that shares the same macroscopic observables by a conservative orthogonal projection. The algorithm is extended to the high dimensional problems by hierarchical Tuck decomposition of solution tensors and a corresponding conservative projection algorithm. Extensive numerical tests on the VP system are showcased for the algorithm’s efficacy.

Bayesian grouping-Gibbs sampling estimation of high-dimensional linear model with non-sparsity

In high-dimensional linear regression models, common assumptions typically entail sparsity of regression coefficients β∈Rp. However, these assumptions may not hold when the majority, if not all, of regression coefficients are non-zeros. Statistical methods designed for sparse models may lead to substantial bias in model estimation. Therefore, this article proposes a novel Bayesian Grouping-Gibbs Sampling (BGGS) method, which departs from the common sparse assumptions in high-dimensional problems. The BGGS method leverages a grouping strategy that partitions β into distinct groups, facilitating rapid sampling in high-dimensional space. The grouping number (k) can be determined using the ‘Elbow plot’, which operates efficiently and is robust against the initial value. Theoretical analysis, under some regular conditions, guarantees model selection and parameter estimation consistency, and bound for the prediction error. Furthermore, three finite simulations are conducted to assess the competitive advantages of the proposed method in terms of parameter estimation and prediction accuracy. Finally, the BGGS method is applied to a financial dataset to explore its practical utility.

Multi-Strategy Bald Eagle Search Algorithm Embedded Orthogonal Learning for Wireless Sensor Network (WSN) Coverage Optimization

Coverage control is a fundamental and critical issue in plentiful wireless sensor network (WSN) applications. Aiming at the high-dimensional optimization problem of sensor node deployment and the complexity of the monitoring area, an orthogonal learning multi-strategy bald eagle search (OLMBES) algorithm is proposed to optimize the location deployment of sensor nodes. This paper incorporates three kinds of strategies into the bald eagle search (BES) algorithm, including Lévy flight, quasi-reflection-based learning, and quadratic interpolation, which enhances the global exploration ability of the algorithm and accelerates the convergence speed. Furthermore, orthogonal learning is integrated into BES to improve the algorithm’s robustness and premature convergence problem. By this way, population search information is fully utilized to generate a more superior position guidance vector, which helps the algorithm jump out of the local optimal solution. Simulation results on CEC2014 benchmark functions reveal that the optimization performance of the proposed approach is better than that of the existing method. On the WSN coverage optimization problem, the proposed method has greater network coverage ratio, node uniformity, and stronger optimization stability when compared to other state-of-the-art algorithms.

Development of Two Methods for Estimating High-Dimensional Data in the Case of Multicollinearity and Outliers

High-dimensional problems involve datasets or models characterized by a substantial number of variables or parameters prevalent across various domains such as statistics, machine learning, optimization, physics, and engineering. Challenges in these scenarios include computational complexity, data sparsity, over-fitting, and the curse of dimensionality. This study introduces two innovative techniques that combine the Random Forest machine learning approach with both the least absolute shrinkage and selection operator and the elastic net, which are statistical methodologies tailored to address high-dimensional challenges. We compared performance evaluations of these hybrid methods against traditional statistical approaches and standalone machine learning methods. The assessment is conducted using goodness-of-fit measures and involves both Monte Carlo simulation and a real-world application. The findings show that the strategies proposed in this study exhibit superior performance compared to conventional approaches when tackling high-dimensional challenges.

Use of Deep-Learning-Accelerated Gradient Approximation for Reservoir Geological Parameter Estimation

The estimation of space-varying geological parameters is often not computationally affordable for high-dimensional subsurface reservoir modeling systems. The adjoint method is generally regarded as an efficient approach for obtaining analytical gradient and, thus, proceeding with the gradient-based iteration algorithm; however, the infeasible memory requirement and computational demands strictly prohibit its generic implementation, especially for high-dimensional problems. The autoregressive neural network (aNN) model, as a nonlinear surrogate approximation, has gradually received increasing popularity due to significant reduction of computational cost, but one prominent limitation is that the generic application of aNN to large-scale reservoir models inevitably poses challenges in the training procedure, which remains unresolved. To address this issue, model-order reduction could be a promising strategy, which enables us to train the neural network in a very efficient manner. A very popular projection-based linear reduction method, i.e., propel orthogonal decomposition (POD), is adopted to achieve dimensionality reduction. This paper presents an architecture of a projection-based autoregressive neural network that efficiently derives an easy-to-use adjoint model by the use of an auto-differentiation module inside the popular deep learning frameworks. This hybrid neural network proxy, referred to as POD-aNN, is capable of speeding up derivation of reduced-order adjoint models. The performance of POD-aNN is validated through a synthetic 2D subsurface transport model. The use of POD-aNN significantly reduces the computation cost while the accuracy remains. In addition, our proposed POD-aNN can easily obtain multiple posterior realizations for uncertainty evaluation. The developed POD-aNN emulator is a data-driven approach for reduced-order modeling of nonlinear dynamic systems and, thus, should be a very efficient modeling tool to address many engineering applications related to intensive simulation-based optimization.

Global Stochastic Comprehensive Sensitivity Analysis Based on Robustness Grouping and Improved Pelican Algorithm-Optimized Radial Basis Function Neural Network

Abstract Modeling analysis is one of the important means to analyze practical engineering, and as technology continues to evolve, various models are getting closer and closer to reality, while at the same time, there are more and more parameters in the models. It is important to analyse the impact of these parameters on the project to assist engineers in making plans or decisions. Sensitivity analysis (SA) can describe the effect of changes in these parameters on the model. However, complex models often have dozens or even hundreds of parameters, and most current SA methods struggle to deal reliably and effectively with these high-dimensional problems. In addition, it is difficult to obtain the sensitivity of continuous points in the parameter space with traditional SA methods. Therefore, this paper proposes a method that combines adaptive grouping and an Improved Pelican Optimization Algorithm for an optimal radial basis function (IPOA-RBF) agent model to solve these problems. Firstly, a clustering grouping method considering grouping robustness is established to obtain objective and stable parameter grouping results in high-dimensional sensitivity analysis. Secondly, a proxy model based on radial basis function neural network and an improved Pelican optimization algorithm are proposed to capture the logic of the proxy model to obtain the parameter sensitivity of continuous points in the parameter space. Finally, the superiority and applicability of this method is verified using an arch dam simulation model.

U-AEFA: Online and offline learning-based unified artificial electric field algorithm for real parameter optimization

Optimization problems in real-world scenarios require algorithms that effectively balance exploration and exploitation to avoid local optima and achieve global solutions. To address this, we propose a unified artificial electric field algorithm (U-AEFA) that integrates both offline learning (inspired by high-performing metaheuristic algorithms) and online learning (through historical data generated during evolution). U-AEFA introduces a unique three-layer population structure to enhance search efficiency, consisting of the best agent in the first layer, the top-performing agents in the second layer, and the remaining agents in the third layer. Key features of U-AEFA include (i) an effective Coulomb’s constant for improved exploration, (ii) a non-uniform mutation operator to mitigate premature convergence, and (iii) two acceleration coefficients for enhanced performance. These three factors constitute offline learning and have been implemented to improve different design elements of the algorithm. As part of online learning, it employs a difference vector reuse (DVR) strategy to evolve the first-layer agents. The algorithm is evaluated using the CEC 2017 test suite across multiple dimensions (10, 30, 50, 100), where it consistently outperforms seven state-of-the-art algorithms, demonstrating superior accuracy and convergence speed. Moreover, U-AEFA’s robustness is validated on 12 high-dimensional feature selection problems, further highlighting its effectiveness in solving complex optimization tasks. The source code of U-AEFA is available at

Solving higher dimensional expensive black box global optimization problems using sparse directional search on surrogates

AbstractWe present a new algorithm Directional Optimization Search with Surrogate (DOSS), for optimizing problems with box constraints and a computationally expensive black-box objective function. DOSS is a radial basis function (RBF) based method that mainly focuses on higher dimensional and computationally expensive objective functions that can be multimodal. DOSS introduces three new techniques not previously used in earlier RBF algorithms, including using a combination of the coordinates knowledge level, fewer initial sampling points, and the surrogate’s gradient information. Numerical results on a test set including 14 test problems with 36, 48, and 60 dimensions show that DOSS outperforms two recently published algorithms RBFOpt and TuRBO, and earlier RBF algorithms such as DYCORS. TuRBO is a Gaussian process based optimization algorithm, which outperformed earlier state-of-the-art methods. DOSS algorithm also has a good performance on real-world optimization problems, including robot pushing and rover trajectory planning problems. Almost sure convergence for the DOSS algorithm is also proven in this paper. An implementation of DOSS is available at https://doi.org/10.5281/zenodo.13731558.

Modeling extreme events: Univariate and multivariate data-driven approaches

AbstractThis article summarizes the contribution of team genEVA to the EVA (2023) Conference Data Challenge. The challenge comprises four individual tasks, with two focused on univariate extremes and two related to multivariate extremes. In the first univariate assignment, we estimate a conditional extremal quantile using a quantile regression approach with neural networks. For the second, we develop a fine-tuning procedure for improved extremal quantile estimation with a given conservative loss function. In the first multivariate sub-challenge, we approximate the data-generating process with a copula model. In the remaining task, we use clustering to separate a high-dimensional problem into approximately independent components. Overall, competitive results were achieved for all challenges, and our approaches for the univariate tasks yielded the most accurate quantile estimates in the competition.

Multi-input Fourier neural network and its sparrow search optimization

In engineering applications, the back-propagation (BP) neural network often encounters many limitations due to its slow convergence and high noise sensitivity, and meanwhile the reported Fourier neural networks have no ability to extract the features of multi-attribute input data. Hereby, This work proposes a gradient descent-based multi-input Fourier neural network after integrating the multi-layer perceptron with an overlapping Fourier neural network. Thereafter, related to the difficulty of deciding the global optimal parameter settings, an improved sparrow search algorithm is developed to optimize the parameter settings and solve high dimensional function optimization problems, after the Cat chaotic map and the mechanisms of population-size adjustment and parameter adaptiveness are designed to promote the sparrow search algorithm's ability to balance global exploration and local exploitation. The theoretical analysis shows that the improved algorithm's computational complexity is decided by its population size and the optimization problem's dimension. Numerically comparative experiments have validated that not only the acquired Fourier neural network can effectively extract the features of multi-attribute data with strong generalization ability, but also the improved algorithm has significant advantages in coping with high dimensional function optimization problems.

A New Hybrid Improved Arithmetic Optimization Algorithm for Solving Global and Engineering Optimization Problems

The Arithmetic Optimization Algorithm (AOA) is a novel metaheuristic inspired by mathematical arithmetic operators. Due to its simple structure and flexible parameter adjustment, the AOA has been applied to solve various engineering problems. However, the AOA still faces challenges such as poor exploitation ability and a tendency to fall into local optima, especially in complex, high-dimensional problems. In this paper, we propose a Hybrid Improved Arithmetic Optimization Algorithm (HIAOA) to address the issues of susceptibility to local optima in AOAs. First, grey wolf optimization is incorporated into the AOAs, where the group hunting behavior of GWO allows multiple individuals to perform local searches at the same time, enabling the solution to be more finely tuned and avoiding over-concentration in a particular region, which can improve the exploitation capability of the AOA. Second, at the end of each AOA run, the follower mechanism and the Cauchy mutation operation of the Sparrow Search Algorithm are selected with the same probability and perturbed to enhance the ability of the AOA to escape from the local optimum. The overall performance of the improved algorithm is assessed by selecting 23 benchmark functions and using the Wilcoxon rank-sum test. The results of the HIAOA are compared with other intelligent optimization algorithms. Furthermore, the HIAOA can also solve three engineering design problems successfully, demonstrating its competitiveness. According to the experimental results, the HIAOA has better test results than the comparator.

A projection-based time-segmented reduced order model for fluid-structure interactions

In this paper, a type of novel projection-based, time-segmented reduced order model (ROM) is proposed for dynamic fluid-structure interaction (FSI) problems based upon the arbitrary Lagrangian–Eulerian (ALE)-finite element method (FEM) in a monolithic frame, where spatially, each variable is separated from others in terms of their attribution (fluid/structure), category (velocity/pressure) and component (two/three dimension) while temporally, the proper orthogonal decomposition (POD) bases are constructed in some deliberately partitioned time segments tailored through extensive numerical trials. By the combination of spatial and temporal decompositions, the developed ROM approach enables prolonged simulations under prescribed accuracy thresholds. Numerical experiments are carried out to compare numerical performances of the proposed ROM with corresponding full-order model (FOM) by solving a two-dimensional FSI benchmark problem that involves a vibrating elastic beam in the fluid, where the performance of offline ROM on perturbed physical parameters in the online phase is investigated as well. Extensive numerical results demonstrate that the proposed ROM has a comparable accuracy to while much higher efficiency than the FOM. The developed ROM approach is dimension-independent and can be seamlessly extended to solve high dimensional FSI problems.

Autonomous Scanning Tunneling Microscopy Imaging via Deep Learning.

Scanning tunneling microscopy (STM) is a powerful technique that provides the ability to manipulate and characterize individual atoms and molecules with atomic-level precision. However, the processes of scanning samples, operating the probe, and analyzing data are typically labor-intensive and subjective. Deep learning (DL) techniques have shown immense potential in automating complex tasks and solving high-dimensional problems. In this study, we developed an autonomous STM framework powered by DL to enable autonomous operations of the STM without human interventions. Our framework employs a convolutional neural network (CNN) for real-time evaluation of STM image quality, a U-net model for identifying bare surfaces, and a deep Q-learning network (DQN) agent for autonomous probe conditioning. Additionally, we integrated an object recognition model for the automated recognition of different adsorbates. This autonomous framework enables the acquisition of space-averaging information using STM techniques without compromising the high-resolution molecular imaging. We achieved measuring an area of approximately 1.9 μm2 within 48 h of continuous measurement and automatedly generated the statistics on the molecular species present within the mesoscopic area. We demonstrate the high robustness of the framework by conducting measurements at the liquid nitrogen temperature (∼78 K). We envision that the integration of DL techniques and high-resolution microscopy will not only extend the functionality and capability of scanning probe microscopes but also accelerate the understanding and discovery of new materials.

The intelligent operation and maintenance method and experimental research of CNC machine tool bearings

As the core components of high-precision rotary machine tools, machine tool bearings are prone to failure or damage, and their running state directly affects the safety and reliability of the entire equipment. Therefore, the paper proposes an intelligent operation and health management method for machine tool bearings based on Bayes-VMD-KPCA, Bayes-CNN-SVM, and health status evaluation index. Aiming at the complex working environment and high nonlinear degree of fault features of machine tool bearings, the Bayes-VMD-KPCA algorithm is used to perform variational mode decomposition of the original signal of the bearing fault, which avoids the influence of background noise and solves the problem of high dimension of multi-domain fault feature set. Second, the CNN-SVM fault diagnosis model uses a convolutional neural network (CNN) to deeply mine the extracted multi-domain fault features, and a support vector machine (SVM) can be used as a fault classifier to complete the diagnosis of various fault types of machine tool bearings. At the same time, the Bayes algorithm is introduced to tune the hyperparameters in the CNN-SVM model to further improve the prediction performance of the CNC machine tool-bearing fault diagnosis model. The results show that the intelligent operation and maintenance and health management method of CNC machine tool bearings proposed in this paper can effectively complete the fault type diagnosis and health status assessment of bearings, and its accuracy index reaches 99.33%. Compared with the single SVM model, Bayes-CNN model, Bayes-SVM model, and CNN-SVM model, the health effect is evaluated. The accuracy is increased by 6.33%, 4.33%, 2.66%, and 1%, respectively. It can be seen that the proposed method can more effectively realize the monitoring and health management of bearing fault.

Stabilization of a matrix via a low-rank-adaptive ODE

Let A be a square matrix with a given structure (e.g. real matrix, sparsity pattern, Toeplitz structure, etc.) and assume that it is unstable, i.e. at least one of its eigenvalues lies in the complex right half-plane. The problem of stabilizing A consists in the computation of a matrix B, whose eigenvalues have all negative real part and such that the perturbation Δ=B-A\\documentclass[12pt]{minimal} \\usepackage{amsmath} \\usepackage{wasysym} \\usepackage{amsfonts} \\usepackage{amssymb} \\usepackage{amsbsy} \\usepackage{mathrsfs} \\usepackage{upgreek} \\setlength{\\oddsidemargin}{-69pt} \\begin{document}$$\\Delta =B-A$$\\end{document} has minimal norm. The structured stabilization further requires that the perturbation preserves the structural pattern of A. This non-convex problem is solved by a two-level procedure which involves the computation of the stationary points of a matrix ODE. It is possible to exploit the underlying low-rank features of the problem by using an adaptive-rank integrator that follows rigidly the rank of the solution. Some benefits derived from the low-rank setting are shown in several numerical examples. These computational advantages also allow to deal with high dimensional problems.