High-dimensional Problems Research Articles

A Novel Artificial Electric Field Algorithm for Solving Global Optimization and Real-World Engineering Problems.

The Artificial Electric Field Algorithm (AEFA) stands out as a physics-inspired metaheuristic, drawing inspiration from Coulomb's law and electrostatic force; however, while AEFA has demonstrated efficacy, it can face challenges such as convergence issues and suboptimal solutions, especially in high-dimensional problems. To overcome these challenges, this paper introduces a modified version of AEFA, named mAEFA, which leverages the capabilities of Lévy flights, simulated annealing, and the Adaptive s-best Mutation and Natural Survivor Method (NSM) mechanisms. While Lévy flights enhance exploration potential and simulated annealing improves search exploitation, the Adaptive s-best Mutation and Natural Survivor Method (NSM) mechanisms are employed to add more diversity. The integration of these mechanisms in AEFA aims to expand its search space, enhance exploration potential, avoid local optima, and achieve improved performance, robustness, and a more equitable equilibrium between local intensification and global diversification. In this study, a comprehensive assessment of mAEFA is carried out, employing a combination of quantitative and qualitative measures, on a diverse range of 29 intricate CEC'17 constraint benchmarks that exhibit different characteristics. The practical compatibility of the proposed mAEFA is evaluated on five engineering benchmark problems derived from the civil, mechanical, and industrial engineering domains. Results from the mAEFA algorithm are compared with those from seven recently introduced metaheuristic algorithms using widely adopted statistical metrics. The mAEFA algorithm outperforms the LCA algorithm in all 29 CEC'17 test functions with 100% superiority and shows better results than SAO, GOA, CHIO, PSO, GSA, and AEFA in 96.6%, 96.6%, 93.1%, 86.2%, 82.8%, and 58.6% of test cases, respectively. In three out of five engineering design problems, mAEFA outperforms all the compared algorithms, securing second place in the remaining two problems. Results across all optimization problems highlight the effectiveness and robustness of mAEFA compared to baseline metaheuristics. The suggested enhancements in AEFA have proven effective, establishing competitiveness in diverse optimization problems.

Wong–Zakai approximation of a stochastic partial differential equation with multiplicative noise

In this article, we derive the convergence rate for the Wong–Zakai equation of some approximation of stochastic evolution equations with multiplicative noise. To be more precise, the diffusion coefficient in front of the noise is the multiplication operator, and, is therefore not bounded, a situation not treated in the literature. Since our motivation comes from problems in numerical ling, we consider a finite, high-dimensional problem approximating a stochastic evolution equation on a random time grid. By imposing suitable stability conditions on the drift term and the time grid, we achieve a convergence rate in the mean square of order min { 1 − δ , 2 − 2 γ } , for some 0 < δ < 1 and 0 < γ < 1 / 2 .

PolyARBerNN: A Neural Network Guided Solver and Optimizer for Bounded Polynomial Inequalities

Constraints solvers play a significant role in the analysis, synthesis, and formal verification of complex cyber-physical systems. In this article, we study the problem of designing a scalable constraints solver for an important class of constraints named polynomial constraint inequalities (also known as nonlinear real arithmetic theory). In this article, we introduce a solver named PolyARBerNN that uses convex polynomials as abstractions for highly nonlinears polynomials. Such abstractions were previously shown to be powerful to prune the search space and restrict the usage of sound and complete solvers to small search spaces. Compared with the previous efforts on using convex abstractions, PolyARBerNN provides three main contributions namely (i) a neural network guided abstraction refinement procedure that helps selecting the right abstraction out of a set of pre-defined abstractions, (ii) a Bernstein polynomial-based search space pruning mechanism that can be used to compute tight estimates of the polynomial maximum and minimum values which can be used as an additional abstraction of the polynomials, and (iii) an optimizer that transforms polynomial objective functions into polynomial constraints (on the gradient of the objective function) whose solutions are guaranteed to be close to the global optima. These enhancements together allowed the PolyARBerNN solver to solve complex instances and scales more favorably compared to the state-of-the-art nonlinear real arithmetic solvers while maintaining the soundness and completeness of the resulting solver. In particular, our test benches show that PolyARBerNN achieved 100X speedup compared with Z3 8.9, Yices 2.6, and PVS (a solver that uses Bernstein expansion to solve multivariate polynomial constraints) on a variety of standard test benches. Finally, we implemented an optimizer called PolyAROpt that uses PolyARBerNN to solve constrained polynomial optimization problems. Numerical results show that PolyAROpt is able to solve high-dimensional and high order polynomial optimization problems with higher speed compared to the built-in optimizer in the Z3 8.9 solver.

A LAPACK Implementation of the Dynamic Mode Decomposition

The Dynamic Mode Decomposition (DMD) is a method for computational analysis of nonlinear dynamical systems in data driven scenarios. Based on high fidelity numerical simulations or experimental data, the DMD can be used to reveal the latent structures in the dynamics or as a forecasting or a model order reduction tool. The theoretical underpinning of the DMD is the Koopman operator on a Hilbert space of observables of the dynamics under study. This paper describes a numerically robust and versatile variant of the DMD and its implementation using the state-of-the-art dense numerical linear algebra software package LAPACK . The features of the proposed software solution include residual bounds for the computed eigenpairs of the DMD matrix, eigenvectors refinements and computation of the eigenvectors of the Exact DMD, compressed DMD for efficient analysis of high dimensional problems that can be easily adapted for fast updates in a streaming DMD. Numerical analysis is the bedrock of numerical robustness and reliability of the software, that is tested following the highest standards and practices of LAPACK . Important numerical topics are discussed in detail and illustrated using numerous numerical examples.

Dynamic Exploration–Exploitation Pareto Approach for high-dimensional expensive black-box optimization

Surrogate Optimization (SO) plays a vital role in optimizing performance parameters for computationally expensive simulations. However, SO encounters significant challenges in high-dimensional spaces due to the curse of dimensionality, hampering effective point sampling around global optima. In this paper, we introduce “Dynamic Exploration–Exploitation Pareto Approach (DEEPA),” a novel SO method that combines Pareto sampling with a dynamic discretization schema to optimize high-dimensional black-box functions. Unlike traditional SO methods that heavily rely on specific surrogate models, Pareto sampling offers a more adaptable approach. Improvement-based acquisition functions, frequently employed in black-box optimization, are sensitive to model accuracy and tend to prioritize exploitation, potentially missing valuable regions of interest in complex landscapes. Furthermore, they can encounter challenges when dealing with high-dimensional problems due to the curse of dimensionality. DEEPA leverages dynamic coordinate importance to generate samples effectively, providing a solution for addressing high-dimensionality and complex functions. We employ feature selection strategies to assign importance probabilities to perturb each coordinate, demonstrating the impact of importance-based perturbation on convergence to a near-optimal region. We showcase DEEPA’s versatility in fixed-batch evaluation environments using complex global optimization test problems with various topological properties. We compare DEEPA’s performance with state-of-the-art black-box optimization algorithms, and our experimental results demonstrate DEEPA’s superior performance, particularly in complex problems with multiple local minima.

Computing multi-eigenpairs of high-dimensional eigenvalue problems using tensor neural networks

In this paper, we propose a type of tensor-neural-network-based machine learning method to compute multi-eigenpairs of high dimensional eigenvalue problems without Monte-Carlo procedure. Solving multi-eigenvalues and their corresponding eigenfunctions is one of the basic tasks in mathematical and computational physics. With the help of tensor neural network, the high dimensional integrations included in the loss functions of the machine learning process can be computed with high accuracy. The high accuracy of high dimensional integrations can improve the accuracy of the machine learning method for computing multi-eigenpairs of high dimensional eigenvalue problems. Here, we introduce the tensor neural network and design the machine learning method for computing multi-eigenpairs of the high dimensional eigenvalue problems. The proposed numerical method is validated with plenty of numerical examples.

Metaheuristic Algorithms for 6G wireless communications: Recent advances and applications

The widespread distribution of intelligent applications and devices, coupled with the vast array of mobile data, technologies, and architectures within Sixth Generation (6G) networks, highlights the importance of optimization strategies. Concurrently, state-of-the-art Metaheuristic Algorithms (MHAs) have emerged as a promising optimization approach for wireless networks, bridging game theory and convex optimization domains. Falling within the domain of Artificial Intelligence (AI), MHAs draw their inspiration from Evolutionary Algorithms (EAs), which are rooted in the principle of natural selection, the principles of Swarm Intelligence (SI), and Trajectory-based Algorithms (TAs). In recent times, MHAs have been implemented in the 6G network. These simple solutions with limited capabilities have proven effective in solving complex, challenging and high-dimensional problems. Nevertheless, the literature has yet to study the MHAs comprehensively, especially within 6G networks. The main objective of this study is to investigate how to merge the 6G and MHAs domains. We begin by giving a comprehensive overview of MHAs, from fundamental to popular optimization methods. Then, we examine the utilization of MHAs to address the challenges in 6G, such as network security, resource allocation, spectrum management, edge computing, wireless caching, and others. Finally, we identify the current literature’s limitations and suggest avenues for further research directions.

Local physical gradient-based coupling information interpolate method and application on double-wall turbine blade multidisciplinary analysis

A novel generalization interpolate modification method based on local gradient is presented. The main idea was to remove the sample points located in the gradient direction away from the unknown points compared to the sample points located in the perpendicular direction. This method first uses the Taylor expansion to construct a set of equations and solves the local physical gradient. Subsequently, a scale vector is derived to modify the interpolate space. Three classical interpolation methods were modified: Inverse Distance Weighted (IDW), Radial Basis Function (RBF), and Kriging method. The results of the test functions prove that the proposed method can effectively decrease the interpolate error regardless of low or high dimensional interpolate problem and irrespective of the sample point size. Surface and volume interpolate validations demonstrate that the local gradient generates a large interpolate error, and the modified method can essentially diminish this error with a maximum reduction of 9.94 %. Subsequently, this method is applied to implement a multidisciplinary analysis of a practical double-wall cooling blade with a large physical gradient. The results showed that the pressure and temperature interpolate errors could be decreased by 5.18 % and 14.5 %, respectively, with only an additional 115 s, at the fillet and film hole area where the local gradient was high. When the strength analysis was implemented, the maximum stress error reduction was 25.04 MPa at the stress concentration region.

Uncertainty quantification with high-dimensional correlated process variations for an in-situ thermal expansion coefficient test structure

The impacts of process variations on the π-shaped thermal expansion coefficient (TEC) in-situ test structure are analyzed. Correlations among high-dimensional design parameters have to be taken into account for this type of structure, while the widely studied polynomial chaos expansion (PCE) technique is incapable of solving such problems. This paper presents a novel method for high-dimensional uncertainty quantification problems with non-Gaussian correlated variants. As a pre-processing method, the Spearman correlation coefficient and generalized lambda distribution are merged to eliminate the correlation. Then a compressed-sensing based Legendre-sparse solver is applied benefiting from the uniformly distributed parameters derived from the pre-processing and thus the cost of rebuilding correlated basis functions is eliminated. This method has demonstrated superior performance over existing methods in the analysis of TEC in-situ test structures. This analysis shows that the measurement accuracy of the test structure demonstrates good performance and is insensitive to process errors, which is consistent with experimental data. The presented technique has the potential to be applied in the design of various microelectromechanical system (MEMS) devices.

Efficient least squares approximation and collocation methods using radial basis functions

We describe an efficient method for the approximation of functions using radial basis functions (RBFs), and extend this to a solver for boundary value problems on irregular domains. The method is based on RBFs with centers on a regular grid defined on a bounding box, with some of the centers outside the computational domain. The equation is discretized using collocation with oversampling, with collocation points inside the domain only, resulting in a rectangular linear system to be solved in a least squares sense. The goal of this paper is the efficient solution of that rectangular system. We show that the least squares problem splits into a regular part, which can be expedited with the FFT, and a low rank perturbation, which is treated separately with a direct solver. The rank of the perturbation is influenced by the irregular shape of the domain and by the weak enforcement of boundary conditions at points along the boundary. The solver extends the AZ algorithm which was previously proposed for function approximation involving frames and other overcomplete sets. The solver has near optimal log-linear complexity for univariate problems, and loses optimality for higher-dimensional problems but remains faster than a direct solver. A Matlab implementation is available at https://gitlab.kuleuven.be/numa/public/2023-RBF-AZalgorithm.

DF-ParPINN: parallel PINN based on velocity potential field division and single time slice focus

Modified Benney-Luke equation (mBL equation) is a three-dimensional temporal-spatial equation with complex structures, that is a high-dimensional partial differential equation (PDE), it is also a new equation of the physical ocean field, and its solution is important for studying the internal wave-wave interaction of inclined seafloor. For conventional PDE solvers such as the pseudo-spectral method, it is difficult to solve mBL equation with both accuracy and speed. Physics-informed neural network (PINN) incorporates physical prior knowledge in deep neural networks, which can solve PDE with relative accuracy and speed. However, PINN is only suitable for solving low-dimensional PDE with simple structures, and not suitable for solving high-dimensional PDE with complex structures. This is mainly because high-dimensional PDEs usually have complex structures and high-order derivatives and are likely to be high-dimensional non-convex functions, and the high-dimensional non-convex optimization problem is an NP-hard problem, resulting in the PINN easily falling into inaccurate local optimal solutions when solving high-dimensional PDEs. Therefore, we improve the PINN for the characteristics of mBL equation and propose “DF-ParPINN: parallel PINN based on velocity potential field division and single time slice focus” to solve mBL equation with large amounts of data. DF-ParPINN consists of three modules: temporal-spatial division module of overall velocity potential field, data rational selection module of multiple time slices, and parallel computation module of high-velocity fields and low-velocity fields. The experimental results show that the solution time of DF-ParPINN is no more than 0.5s, and its accuracy is much higher than that of PINN, PIRNN, cPINN, and DeepONet. Moreover, the relative error of DF-ParPINN after deep training 1000000 epochs can be reduced to less than 0.1. The validity of DF-ParPINN proves that the improved PINN also can solve high dimensional PDE with complex structures and large amounts of data quickly and accurately, which is of great significance to the deep learning of the physical ocean field.

D 3 PBO: D ynamic D omain D ecomposition-based P arallel B ayesian O ptimization for Large-scale Analog Circuit Sizing

Bayesian optimization (BO) is an efficient global optimization method for expensive black-box functions, but the expansion for high-dimensional problems and large sample budgets still remains a severe challenge. In order to extend BO for large-scale analog circuit synthesis, a novel computationally efficient parallel BO method, D 3 PBO, is proposed for high-dimensional problems in this work. We introduce the dynamic domain decomposition method based on maximum variance between clusters. The search space is decomposed into subdomains progressively to limit the maximal number of observations in each domain. The promising domain is explored by multi-trust region-based batch BO with the local Gaussian process (GP) model. As the domain decomposition progresses, the basin-shaped domain is identified using a GP-assisted quadratic regression method and exploited by the local search method BOBYQA to achieve a faster convergence rate. The time complexity of D 3 PBO is constant for each iteration. Experiments demonstrate that D 3 PBO obtains better results with significantly less runtime consumption compared to state-of-the-art methods. For the circuit optimization experiments, D 3 PBO achieves up to 10× runtime speedup compared to TuRBO with better solutions.

Manifold-guided multi-objective gradient algorithm combined with adjoint method for supersonic aircraft shape design

The aerodynamic design of supersonic aircraft necessitates the comprehensive consideration of multiple conflicting design objectives and multi-disciplinary characteristics. However, the efficiency of heuristic algorithms in solving multi-objective aerodynamic problems significantly decreases with the increase in the number of design variables and objectives. Although gradient-based adjoint optimization methods can eliminate the constraints of design variables on aerodynamic optimization and have high optimization efficiency, they are prone to getting stuck in local optima and are difficult to apply in multi-objective optimization. To overcome these shortcomings, a manifold-guided multi-objective gradient-based algorithm (MG-MOGBA) is proposed. This algorithm enhances the optimization efficiency of the algorithm by coupling the multi-objective gradient operator with the predicted Pareto front (PF) manifold structure to guide the population search direction and reduce the search space. A multi-objective aerodynamic optimization framework based on the discrete adjoint method is developed to automate the aerodynamic optimization process. Test functions and airfoil optimization results confirm that the proposed algorithm achieves superior results with fewer iterations and is more efficient than heuristic algorithms. The four-objective supersonic civil aircraft example demonstrates that the algorithm can effectively handle high-dimensional multi-objective aerodynamic optimization problems. A potential equilibrium solution increased the cruise factor by 17.5% and 12.8% under supersonic and transonic conditions respectively, reduced the maximum overpressure of the sonic boom by 25.1%, and decreased the root bending moment by 20.3%. Overall, MG-MOGBA can efficiently solve high-dimensional multi-objective aerodynamic optimization problems and has good scalability for the dimensions of design objectives and variables.

Equivalence of state equations from different methods in high-dimensional regression

State equations (SEs) were first introduced in the approximate message passing process to describe the mean square error in compressed sensing. Since then a set of state equations has appeared in studies of logistic regression, robust estimator, and other high-dimensional statistics problems. Recently, a convex Gaussian min-max theorem approach was proposed to study high-dimensional statistic problems accompanying with another set of different state equations. This article provides a uniform viewpoint on these methods and shows the equivalence of their reduction forms, which causes that the resulting SEs are essentially equivalent and can be converted into the same expression through parameter transformations. Combining these results, we show that these different state equations are derived from several equivalent reduction forms. We believe that this equivalence will shed light on discovering a deeper structure in high-dimensional statistics.

Study on the constraint handling method for high-dimensional optimization of cascade reservoirs

Multicomplex constraints often need to be considered in the optimal operation of cascade reservoirs with high-dimensional decision variables, making it difficult for traditional optimization methods and modern intelligent algorithms to solve such problems. Therefore, this study proposes a constraint handling method combining a penalty function nested DPSA-POA and an intelligent algorithm, to determine the optimal flood control operation of cascade reservoirs in the middle reaches of the Ganjiang River, a problem with decision variables of up to 2196 dimensions. The results indicate that the constraint handling method proposed in this paper can solve high-dimensional optimization problems in three modes: continuous nesting (Mode 1), optimization with intelligence after obtaining a feasible solution (Mode 2), and optimization with the DPSA-POA after obtaining a feasible solution (Mode 3). Of the three modes, Mode 2 has the highest accuracy, but its calculation time is approximately 10 h. Although the accuracy of Mode 3 is slightly worse, can only achieve 98%–99% of Mode 2, its calculation time is only approximately 1–3 h. The comprehensive performance of Mode 1 is poor, the convergence accuracy can only reach 97% of Mode 2, which corresponds to a calculation time of approximately 4–6 h. The existing superiority of feasibility (SF), stochastic ranking (SR), penalty function (PF), ε-constraint (EC) and adaptive ε-constraint (Adaptive EC) methods cannot converge to a feasible solution stably, and the accuracy of the results of these methods under the condition of obtaining a feasible solution is significantly lower than that of Mode 2 and Mode 3, reaching only 95%.

A parallel constrained Bayesian optimization algorithm for high-dimensional expensive problems and its application in optimization of VRB structures

A parallel constrained Bayesian optimization algorithm for high-dimensional expensive problems and its application in optimization of VRB structures

Coupling parameter and particle dynamics for adaptive sampling in Neural Galerkin schemes

Training nonlinear parametrizations such as deep neural networks to numerically approximate solutions of partial differential equations is often based on minimizing a loss that includes the residual, which is analytically available in limited settings only. At the same time, empirically estimating the training loss is challenging because residuals and related quantities can have high variance, especially for transport-dominated and high-dimensional problems that exhibit local features such as waves and coherent structures. Thus, estimators based on data samples from un-informed, uniform distributions are inefficient. This work introduces Neural Galerkin schemes that estimate the training loss with data from adaptive distributions, which are empirically represented via ensembles of particles. The ensembles are actively adapted by evolving the particles with dynamics coupled to the nonlinear parametrizations of the solution fields so that the ensembles remain informative for estimating the training loss. Numerical experiments indicate that few dynamic particles are sufficient for obtaining accurate empirical estimates of the training loss, even for problems with local features and with high-dimensional spatial domains.

Applying Proper Orthogonal Decomposition to Parabolic Equations: A Reduced Order Numerical Approach

n this paper we present a low-order numerical scheme developed using the Proper Orthogonal (POD) method to address non-homogeneous parabolic equations in both one and two dimensions. The proposed schemes leverage the POD technique to reduce the computational complexity associated with solving these equations while maintaining accuracy. By employing POD, the high-dimensional problem is approximated by a reduced set of models, allowing for a more efficient representation of the system dynamics. The application of this method to non-homogenous parabolic equations offers a promising approach for enhancing the computational efficiency of simulations in diverse fields, such as fluid dynamics, heat conduction, and reaction-diffusion processes. The presented numerical scheme demonstrates its efficacy in achieving accurate results with significantly reduced computational costs, making it a valuable tool for applications demanding efficient solutions to non-homogeneous parabolic equations in one and two dimensions.

Deep convolutional neural networks with genetic algorithm-based synthetic minority over-sampling technique for improved imbalanced data classification

Imbalanced data classification presents a challenge in machine learning, inducing biased model learning. Moreover, data dimensionality poses another challenge as it highly impacts classifier performance. This paper proposes a new deep-learning method that combines feature selection with oversampling to address these challenges. The proposed approach, GA-SMOTE-DCNN, integrates a genetic algorithm (GA) for feature selection, SMOTE for oversampling, and a deep 1D-convolutional neural network (DCNN) for classification. This study reveals that pre-splitting the data into training and testing sets before applying SMOTE results in higher accuracy, showing an improvement in accuracy ranging between 1.94% and 3.98% compared to post-SMOTE splitting for each dataset. This method achieved accuracy rates of 86.81% for the Balance Scale dataset, 86.15% for the Oil Spill dataset, 89.21% for the Yeast dataset, 91.32% for the Mammography dataset, 88.23% for the Australian credit dataset, and 89.53% for the German Credit dataset when compared with benchmark methods, underscoring its significance in tackling high-dimensional and imbalanced data classification problems. This method demonstrates scalability in effectively addressing challenges associated with high-dimensional and imbalanced data classification across various domains.

Applying Convolutional Neural Networks to data on unstructured meshes with space-filling curves

This paper presents the first classical Convolutional Neural Network (CNN) that can be applied directly to data from unstructured finite element meshes or control volume grids. CNNs have been hugely influential in the areas of image classification and image compression, both of which typically deal with data on structured grids. Unstructured meshes are frequently used to solve partial differential equations and are particularly suitable for problems that require the mesh to conform to complex geometries or for problems that require variable mesh resolution. Central to our approach are space-filling curves, which traverse the nodes or cells of a mesh tracing out a path that is as short as possible (in terms of numbers of edges) and that visits each node or cell exactly once. The space-filling curves (SFCs) are used to find an ordering of the nodes or cells that can transform multi-dimensional solutions on unstructured meshes into a one-dimensional (1D) representation, to which 1D convolutional layers can then be applied. Although developed in two dimensions, the approach is applicable to higher dimensional problems. To demonstrate the approach, the network we choose is a convolutional autoencoder (CAE), although other types of CNN could be used. The approach is tested by applying CAEs to data sets that have been reordered with a space-filling curve. Sparse layers are used at the input and output of the autoencoder, and the use of multiple SFCs is explored. We compare the accuracy of the SFC-based CAE with that of a classical CAE applied to two idealised problems on structured meshes, and then apply the approach to solutions of flow past a cylinder obtained using the finite-element method and an unstructured mesh.