Low-dimensional Problems Research Articles

A primal dual active set with continuation algorithm for high-dimensional nonconvex SICA-penalized regression

ABSTRACTThe smooth integration of counting and absolute deviation (SICA) penalty has been demonstrated theoretically and practically to be effective in nonconvex penalization for variable selection and parameter estimation. However, solving the nonconvex optimization problem associated with SICA penalty in high-dimensional setting remains to be enriched, mainly due to the singularity at the origin and the nonconvexity of the SICA penalty function. In this paper, we develop a fast primal dual active set (PDAS) with continuation algorithm for solving the nonconvex SICA-penalized least squares in high dimensions. Upon introducing the dual variable, the PDAS algorithm iteratively identify and update the active set in the optimization using both the primal and dual information, and then solve a low-dimensional least square problem on the active set. When combined with a continuation strategy and a high-dimensional Bayesian information criterion (BIC) selector on the tuning parameters, the proposed algorithm is very efficient and accurate. Extensive simulation studies and analysis of a high-dimensional microarray gene expression data are presented to illustrate the performance of the proposed method.

On Runtime and Classification Performance of the Discretize-Optimize (DISCO) Classification Approach

Using machine learning in high-speed networks for tasks such as flow classification typically requires either very resource efficient classification approaches, large amounts of computational resources, or specialized hardware. Here we provide a sketch of the discretize-optimize (DISCO) approach which can construct an extremely efficient classifier for low dimensional problems by combining feature selection, efficient discretization, novel bin placement, and lookup. As feature selection and discretization parameters are crucial, appropriate combinatorial optimization is an important aspect of the approach. A performance evaluation is performed for a YouTube classification task using a cellular traffic data set. The initial evaluation results show that the DISCO approach can move the Pareto boundary in the classification performance versus runtime trade-off by up to an order of magnitude compared to runtime optimized random forest and decision tree classifiers.

An epiperimetric inequality for the lower dimensional obstacle problem

In this paper we give a proof of an epiperimetric inequality in the setting of the lower dimensional obstacle problem. The inequality was introduced by Weiss [Invent. Math.138(1999) 23–50) for the classical obstacle problem and has striking consequences concerning the regularity of the free-boundary. Our proof follows the approach of Focardi and Spadaro [Adv. Differ. Equ.21(2015) 153–200] which uses an homogeneity approach and aΓ-convergence analysis.

Leaders and followers algorithm for constrained non-linear optimization

<p>Leaders and Followers algorithm was a novel metaheuristics proposed by Yasser Gonzalez-Fernandez and Stephen Chen. In solving unconstrained optimization, it performed better exploration than other well-known metaheuristics, e.g. Genetic Algorithm, Particle Swarm Optimization and Differential Evolution. Therefore, it performed well in multi-modal problems. In this paper, Leaders and Followers was modified for constrained non-linear optimization. Several well-known benchmark problems for constrained optimization were used to evaluate the proposed algorithm. The result of the evaluation showed that the proposed algorithm consistently and successfully found the optimal solution of low dimensional constrained optimization problems and high dimensional optimization with high number of linear inequality constraint only. Moreover, the proposed algorithm had difficulty in solving high dimensional optimization problem with non-linear constraints and any problem which has more than one equality constraint. In the comparison with other metaheuristics, Leaders and Followers had better performance in overall benchmark problems.</p>

Location-Invariant Physical Layer Identification Approach for WiFi Devices

Recently, Radio Frequency Fingerprinting (RFF) becomes a promising technique which augments existing multifactor authentication schemes at the device level to counter forgery and related threats. As RFF leverages the discriminable hardware imperfections reflected in Radio Frequency (RF) signals for device identification, it has a good property of scalability, accuracy, energy-efficiency and tamper resistance. However, its identification accuracy might be compromised when the locations of training and testing are different, which is a more realistic assumption in practical scenarios. To address this issue, we study the location-invariant RFF feature extraction and identification method for WiFi Network Interface Cards (NICs). Firstly, we present an RFF feature extraction approach named Differential Phase of Pilots (DPoP). To further address the low-dimensional feature space problem, we propose another novel RFF extraction approach named Amplitude of Quotient (AoQ). AoQ exploits the fact that the RFFs of two Long Training Sequences (LTSs) in WiFi frames exhibit semi-steady characteristics and two LTSs in the same frame have similar channel frequency responses. Next, we use Euclidean distance and Deep Neural Network (DNN) for AoQ authentication and identification, respectively. Experimental results verify the effectiveness of our proposed AoQ method among 55 WiFi NICs of 5 models. The identification accuracy is higher than 95% and the Equal Error Rate (EER) is around 4% when SNR is higher than 40 dB.

A Comparative Study of Meta-Heuristic Optimization Algorithms for 0 – 1 Knapsack Problem: Some Initial Results

In this paper, we present some initial results of several meta-heuristic optimization algorithms, namely, genetic algorithms, simulated annealing, branch and bound, dynamic programming, greedy search algorithm, and a hybrid genetic algorithm-simulated annealing for solving the 0-1 knapsack problems. Each algorithm is designed in such a way that it penalizes infeasible solutions and optimizes the feasible solution. The experiments are carried out using both low-dimensional and high-dimensional knapsack problems. The numerical results of the hybrid algorithm are compared with the results achieved by the individual algorithms. The results revealed the superior performances of the branch and bound dynamic programming, and hybrid genetic algorithm with simulated annealing methods over all the compared algorithms. This performance was established by taking into account both the algorithm computational time and the solution quality. In addition, the obtained results also indicated that the hybrid algorithm can be applied as an alternative to solve small- and large-sized 0-1 knapsack problems.

Improvement and Application of Chicken Swarm Optimization for Constrained Optimization

Aiming at the problem of slow convergence speed and ease of falling into local optimum when solving high dimensional problems, this paper proposes an improved chicken swarm optimization algorithm. The improved chicken swarm optimization includes four aspects, namely, cock position update mode, hen position update mode, chick position update mode, and population update strategy, so it is abbreviated as ICSO-RHC. On the basis of algorithm improvement, the influence of the number of retained elite individuals and control parameters on the convergence speed of the algorithm is discussed. The calculation results of the test function show that when the number of elite individuals in the population is 1, and the control parameters is a random number uniformly distributed between [0, 1], the algorithm has a faster convergence speed. In addition, in order to verify the performance of ICSO-RHC, 30 test functions and CEC 2005 benchmark functions were selected. The calculation results of these test functions show that the success rate of ICSO-RHC is significantly higher than other algorithms, both for low-dimensional and high-dimensional optimization problems. The average iteration number and average running time are significantly lower than other algorithms. Finally, ICSO-RHC and other improved algorithms in the literature are used to optimize the parameters of four practical engineering problems. The optimization results show that the statistical results obtained by ICSO-RHC are significantly better than other algorithms. The calculation results of the test functions and the actual engineering problems show that the performance of ICSO-RHC proposed in this paper is significantly better than other algorithms.

Open Source Robotic Simulators Platforms for Teaching Deep Reinforcement Learning Algorithms

One of the primary goals of the artificial intelligence field is to produce fully autonomous agents that interact with theirenvironments to learn optimal behaviors, improving over time through trial and error. A mathematical principled framework for experience-driven autonomous learning is reinforcement learning, but they are inherently limited to low-dimensional problems,but the deep learning boom has provided new tools to overcome these problems. For deep reinforcement learning teaching, we do not have an appropriate platform for making optimal labs. In the article, after studying the theoretical foundations and the requirements of the main platforms, we selected two open source platforms, according to their characteristics: robotic simulators platforms for teaching and benchmarking deep reinforcement learning algorithms. The first platform was Gym and V-REP and the second one, KNIME Deeplearning4J Integration supports and Teaching-Box.

Graph-based sparse linear discriminant analysis for high-dimensional classification

Linear discriminant analysis (LDA) is a well-known classification technique that enjoyed great success in practical applications. Despite its effectiveness for traditional low-dimensional problems, extensions of LDA are necessary in order to classify high-dimensional data. Many variants of LDA have been proposed in the literature. However, most of these methods do not fully incorporate the structure information among predictors when such information is available. In this paper, we introduce a new high-dimensional LDA technique, namely graph-based sparse LDA (GSLDA), that utilizes the graph structure among the features. In particular, we use the regularized regression formulation for penalized LDA techniques, and propose to impose a structure-based sparse penalty on the discriminant vector β. The graph structure can be either given or estimated from the training data. Moreover, we explore the relationship between the within-class feature structure and the overall feature structure. Based on this relationship, we further propose a variant of our proposed GSLDA to utilize effectively unlabeled data, which can be abundant in the semi-supervised learning setting. With the new regularization, we can obtain a sparse estimate of β and more accurate and interpretable classifiers than many existing methods. Both the selection consistency of β estimation and the convergence rate of the classifier are established, and the resulting classifier has an asymptotic Bayes error rate. Finally, we demonstrate the competitive performance of the proposed GSLDA on both simulated and real data studies.

Bat algorithm with principal component analysis

The bat algorithm (BA) is a novel evolutionary optimization algorithm, most studies of which have been performed with low-dimensional problems. To test and improve the global search ability of BA with large-scale problems, two new variants using principal component analysis (PCA_BA and PCA_LBA) are designed in this paper. A correlation threshold and generation threshold are determined using the golden section method to enhance the effectiveness of this new strategy. To test performance, CEC’2008 large-scale benchmark functions are utilized and compared with other algorithms; simulation results indicate the validity of this modification.

A computational method for large‐scale differential symmetric Stein equation

We propose a numerical method for solving large‐scale differential symmetric Stein equations having low‐rank right constant term. Our approach is based on projection the given problem onto a Krylov subspace then solving the low dimensional matrix problem by using an integration method, and the original problem solution is built by using obtained low‐rank approximate solution. Using the extended block Arnoldi process and backward differentiation formula (BDF), we give statements of the approximate solution and corresponding residual. Some numerical results are given to show the efficiency of the proposed method.

Weak antithetic MLMC estimation of SDEs with the Milstein scheme for low-dimensional Wiener processes

In this paper, we implement a weak Milstein Scheme to simulate low-dimensional stochastic differential equations (SDEs). We prove that combining the antithetic multilevel Monte-Carlo (MLMC) estimator introduced by Giles and Szpruch with the MLMC approach for weak SDE approximation methods by Belomestny and Nagapetyan, we can achieve a quadratic computational complexity in the inverse of the Root Mean Square Error (RMSE) when estimating expected values of smooth functionals of SDE solutions, without simulating Lévy areas and without requiring any strong convergence of the underlying SDE approximation method. By using appropriate discrete variables this approach allows us to calculate the expectation on the coarsest level of resolution by enumeration, which, for low-dimensional problems, results in a reduced computational effort compared to standard MLMC sampling. These theoretical results are also confirmed by a numerical experiment.

A Hybrid Optimization Method of Beetle Antennae Search Algorithm and Particle Swarm Optimization

Beetle Antennae Search (BAS) is an intelligent algorithm that mimics the beetle preying process. In BAS, there is a beetle which has a left and a right antennae sensing the intensity of the odor emitted by the food. The beetle learns the different sense of two antennas to determine the flight and finally finds the food. The BAS algorithm has been proved to have good optimizing speed and precision when it is applied for low-dimensional optimization problems. However, when solving high-dimensional problems, the algorithm is easily trapped into local optimum. In order to improve the optimization ability of BAS, we have combined particle swarm optimization (PSO) algorithm and proposed a new hybrid BAS and PSO algorithm (BAS-PSO). In BAS-PSO, firstly, the standard PSO is used for particle velocity and position updating. Particles learn ontology information and group optimal information for evolution. Then each particle in the swarm is taken as a beetle independently applying a local search with BAS algorithm. To verify the performance of BAS-PSO, four benchmark functions have been selected in our simulation experiments. The results show that the performance of BAS-PSO is better than BAS and PSO.

A comprehensive study of phase based optimization algorithm on global optimization problems and its applications

Inspired by the completely different motional features of individuals in three different phases of nature, i.e. gas phase, liquid phase and solid phase, this paper presents a phase based evolutionary model. Based on the proposed model, a specific implementation termed Phase Based Optimization (PBO) was systematically given. Meanwhile, the search behavior analysis and the evolution process of population are provided to further understand the search mechanisms of PBO. To evaluate the performance of PBO, numerical experiments are carried out on twenty-three benchmark test functions consisting of different types of unimodal and multimodal functions. The obtained results demonstrate the better performance of PBO compared with eight state-of-the-art nature-inspired optimization algorithms. Besides, the effects of population size on PBO and the performance comparison of PBO under different problem dimensions are systematically investigated, respectively. Finally, PBO is applied to two application problems which are parameter estimation for frequency modulated sound waves synthesis and large scale transmission pricing problem, and the promising results indicate the applicability of PBO in both low and high dimensional real-world optimization problems.

Switching and information exchange in compressed estimation of coupled high dimensional processes

Compressed Estimation approaches, such as the Generalised Compressed Kalman Filter (GCKF), reduce the computational cost and complexity of high-dimensional and high-frequency data assimilation problems, usually without sacrificing optimality. Configured using adequate cores, such as the Unscented Kalman Filter (UKF), the GCKF could also treat certain high-dimensional non-linear cases. However, the application of a compressed estimation process is limited to a class of problems which inherently allow the estimation process to be divided, at certain intervals of time, into a set of lower-dimensional problems. This limitation prohibits applying the compressing techniques for estimating coupled high-dimensional processes. However, those limitations can be overcome by applying proper techniques. In this paper, the concepts of subsystem switching and information exchange architecture, namely ‘Exploiting Local Statistical Dependency’ (ELSD), have been derived and explored, allowing compressed estimators to mimic optimal full-Gaussian estimators. The performances of the methods have been verified through applications in solving usual types of Stochastic Partial Differential Equations (SPDEs). The computational advantages of using the proposed techniques have also been highlighted with a recommendation for its usage over the full filter when dealing with high-dimensional and high-frequency data assimilation.

Ensemble of surrogates assisted particle swarm optimization of medium scale expensive problems

Surrogate-assisted evolutionary algorithms have been commonly used in extremely expensive optimization problems. However, many existing algorithms are only effectively used in low-dimensional optimization problems and are easily trapped into a local optimum for medium scaled complex problems. An ensemble of surrogates assisted particle swarm optimization (EAPSO) algorithm is proposed in this paper to address this challenge. EAPSO algorithm uses multiple trial positions for each particle in the swarm and selects the promising positions by using the superiority and uncertainty of the ensemble simultaneously. Besides, a novel variable weight coefficient based on evolutionary state is proposed to balance exploration and exploitation. For faster convergence and avoiding wrong global attraction of models, optima of two surrogates (polynomial regression model and radial basis function model) are evaluated in the convergence state of particles. The strategies ensure large exploration space for the swarm and control the time to converge. Forty-two benchmark functions widely adopted in literatures are used to evaluate the proposed approach. Experimental results demonstrate the superiority of EAPSO algorithm compared with three popular algorithms.

An Optimal Subgradient Algorithm with Subspace Search for Costly Convex Optimization Problems

This paper presents an acceleration of the optimal subgradient algorithm OSGA (Neumaier in Math Program 158(1–2):1–21, 2016) for solving structured convex optimization problems, where the objective function involves costly affine and cheap nonlinear terms. We combine OSGA with a multidimensional subspace search technique, which leads to a low-dimensional auxiliary problem that can be solved efficiently. Numerical results concerning some applications are reported. A software package implementing the new method is available.

A Weighted POD Method for Elliptic PDEs with Random Inputs

In this work we propose and analyze a weighted proper orthogonal decomposition method to solve elliptic partial differential equations depending on random input data, for stochastic problems that can be transformed into parametric systems. The algorithm is introduced alongside the weighted greedy method. Our proposed method aims to minimize the error in a $$L^2$$ norm and, in contrast to the weighted greedy approach, it does not require the availability of an error bound. Moreover, we consider sparse discretization of the input space in the construction of the reduced model; for high-dimensional problems, provided the sampling is done accordingly to the parameters distribution, this enables a sensible reduction of computational costs, while keeping a very good accuracy with respect to high fidelity solutions. We provide many numerical tests to assess the performance of the proposed method compared to an equivalent reduced order model without weighting, as well as to the weighted greedy approach, in both low and high dimensional problems.

A scalarization-based dominance evolutionary algorithm for many-objective optimization

Classical Pareto-dominance based multi-objective evolutionary algorithms underperform when applied to optimization problems with more than three objectives. A class of multi-objective evolutionary algorithms introduced in the literature, utilizing pre-determined reference points acting as target vectors to maintain diversity in the objective space, has shown promising results. Inspired by this approach, we propose a scalarization-based dominance evolutionary algorithm (SDEA) that utilizes a reference point-based method and combine it with a novel sorting strategy that employs fitness values determined via scalarization. SDEA reduces computation complexity by eliminating the need for a Pareto-dominance approach to obtain non-dominated solutions. By means of a set of common benchmark optimization problems with 3- to 15-objectives, we compare the performance of SDEA with state-of-the-art many-objective evolutionary algorithms. The results indicate that SDEA outperforms existing algorithms in undertaking complex optimization problems with a high number of objectives, and has comparable outcomes over low-dimensional objective space benchmark problems.

Comparative studies of error metrics in variable fidelity model uncertainty quantification

ABSTRACTVariable-fidelity (VF) surrogate models which integrate different fidelities of date are wildly used in simulation-based modelling due to a good balance of modelling expense and modelling accuracy. However, VF surrogate models built by limited sample points inevitably have large prediction uncertainty. Using inaccurate VF models in the design and optimisation process may lead to distort predictions or optimal solutions that locate in unfeasible region. Besides, if inappropriate error metrics are utilised in the uncertainty quantifying of a surrogate model, misleading or erroneous evaluation results will be obtained, which may lead to the wrong usage of it in design process. In this paper, the performance of four error metrics (bootstrap error, leave-one-out (LOO) error, mean square error (MSE) and predictive estimation of model fidelity (PEMF error) is systematically compared in uncertainty quantification of VF surrogate model. A set of numerical examples with different features and a long cylinder pressure vessel design problem are utilised to test the performance of the error metrics. The error metrics are evaluated from different aspects, including the number of sample points, sampling methods, and dimension of the test problems etc. Results show that in low dimensional problems, MSE shows excellent error prediction capability not only in efficiency but also in effectiveness while LOO error performs the best in high dimensional problems. Based on the comparison results, a useful guideline for selecting the most appropriate error metric for the problems with different characteristics is provided.