Large-scale Optimization Research Articles

A space-reduction based three-phase approach for large-scale optimization

With the rapid development of science and technology, large-scale global optimization (LSGO) has received more and more attention. However, large-scale problem is very challenging due to its high dimensionality, nonlinearity, and countless local optimal solutions. In this paper, a space-reduction based three-phase approach (SRTP) is proposed to solve this problem. In the first phase, a space-reduction based line search method is designed to roughly locate good solutions as well as to explore important information such as promising search regions and characteristics of each variable (dimension). In the second phase, a multi-grouping strategy is proposed for fully non-separable large-scale problems. This new decomposition method provides dozens of grouping results for non-separable large-scale problems that not only reduces the dimensionality but also maintains the correlation of variables to the most extent. Then in the third phase, a space-reduction based group search method is designed to optimize the sub-groups. By decomposing the large-scale problem as well as reducing the huge search space to focus on promising areas, this group search method can gain better efficiency. Experiments are conducted on 35 widely used benchmark functions and the proposed SRTP is compared with state-of-the-art LSGO algorithms. Experimental results show that SRTP is effective and efficient.

A binary chaotic horse herd optimization algorithm for feature selection

One of the most challenging and common problems in machine learning is the Feature Selection (FS) process, which reduces the dataset size by finding optimal subsets of features. The Horse Herd Optimization Algorithm (HOA) is a new metaheuristic algorithm created by modeling the herd behavior of horses and developed for large scale optimization problems. This paper proposes the binary version of the HOA as a wrapper FS method to solve the FS problem. The proposed algorithm is a binary chaotic horse herd optimization algorithm for feature selection (BCHOAFS). The proposed BCHOAFS is applied to select the optimal feature combination that maximizes classification accuracy while minimizing the number of selected features. Classifier algorithms from machine learning algorithms were used to test the accuracy of the reduced subsets. The proposed method was named binary horse herd optimization for feature selection (BHOAFS) before adding chaotic maps; the k-nearest neighbor (k-NN) and Support Vector Machine (SVM) were tested as separate classifiers. It has been seen that k-NN classification accuracy gives better results than SVM. The BHOAFS-kNN method using the k-NN classification was combined with five chaotic maps and named as BCHOAFS-Logistics, BCHOAFS-Piecewise, BCHOAFS-Singer, BCHOAFS-Sinusoidal, BCHOAFS-Tent. The BCHOAFS versions were run on datasets consisting of 18 different sizes and quality datasets (i.e., low scale, medium scale, and large scale) taken from the UCI repository and compared with state-of-the-art algorithms in previous studies. The results prove that the proposed version, especially with the BCHOAFS-Piecewise and the BCHOAFS-Singer chaotic map outperforms or competes with well-known methods. The proof of the proposed approach’s statistical significance has been validated using the Friedman Signed Rank test and post hoc Wilcoxon test. The novelty of BCHOAFS is that HOA, which is an optimization algorithm specially designed for large scale data, is the first binary chaotic-based algorithm developed for feature selection problems. It also proposes a new local search strategy called Similarity Measurement Function (SMF). As a result, versions of the proposed algorithm BCHOAFS can be used for the FS problem.

A parallel particle swarm optimization algorithm based on GPU/CUDA

Parallel computing is the main way to improve the computational efficiency of metaheuristic algorithms for solving high-dimensional, nonlinear optimization problems. Previous studies have typically only implemented local parallelism for the particle swarm optimization (PSO) algorithm. In this study, we proposed a new parallel particle swarm optimization algorithm (GPU-PSO) based on the Graphics Processing Units (GPU) and Compute Unified Device Architecture (CUDA), which uses a combination of coarse-grained parallelism and fine-grained parallelism to achieve global parallelism. In addition, we designed a data structure based on CUDA features and utilized a merged memory access mode to further improve data-parallel processing and data access efficiency. Experimental results show that the algorithm effectively reduces the solution time of PSO for solving high-dimensional, large-scale optimization problems. The speedup ratio increases with the dimensionality of the objective function, where the speedup ratio is up to 2000 times for the high-dimensional Ackley function.

Airline capacity distribution under financial budget and resource consideration

Capacity distribution is a challenging issue for an airline under financial budget and resource consideration. It is a large-scale optimization problem covering both long-term planning and short-term operating arrangements. This study investigates on the airline capacity distribution problem with financial budget and resource consideration. It contains subproblems of financial budget arrangement, fleet introduction, and fleet assignment. Among them, financial budget is arranged in multiple decision periods, fleet introduction is decided under fixed time points, while fleet assignment is decided under all available time points. To tackle this problem, an integer programming model is formulated for descriptions. Then, an integrated algorithm of modified Variable Neighborhood Search (VNS) and Branch-and-bound (B&B) strategy is developed to find solutions. In detail, a greedy heuristic approach is utilized to generate an initial solution for fleet introduction, the modified B&B strategy is utilized to generate the optimal solution for fleet assignment and the modified VNS is applied to update current solution for a new one with better quality. In addition, budget limit checks are added for financial budget arrangements. Finally, the hybrid algorithm is tested on efficiency and stability. It is also compared to other algorithms which replace the modified VNS by basic VNS, differential evolution and genetic algorithm. Computational results show that performance of our approach is powerful in terms of objective value, convergence speed and stability.

Fuzzy Adaptive Parameter in the Dai–Liao Optimization Method Based on Neutrosophy

The impact of neutrosophy has increased rapidly in many areas of science and technology in recent years. Furthermore, numerous applications of the neutrosophic theory have become more usual. We aim to use neutrosophy to enhance Dai–Liao conjugate gradient (CG) iterative method. In particular, we suggest and explore a new neutrosophic logic system intended to compute the essential parameter t required in Dai–Liao CG iterations. Theoretical examination and numerical experiments signify the effectiveness of the introduced method for controlling t. By incorporation of the neutrosophy in the Dai–Liao conjugate gradient principle, we established novel Dai–Liao CG iterations for solving large-scale unconstrained optimization problems. Global convergence is proved under standard assumptions and with the use of the inexact line search. Finally, computational evidence shows the computational effectiveness of the proposed fuzzy neutrosophic Dai–Liao CG method.

A cultural evolution with a modified selection function and adaptive [formula omitted]-cognition procedure for numerical optimization

In recent years, several population-based evolutionary and swarm algorithms have been developed and used in the literature. This work introduces an improved Cultural Algorithm with a modified selection function and a dynamic α-cognition procedure to handle a variety of challenging numerical optimization problems. The modified selection function is used to support a balanced evolutionary search. A process that starts with a clearer exploration early in the search process and gradually begins to focus on exploitation towards the end of the search process. This work uses the elites of each knowledge source that are at a certain distance from each other. The dynamic α-cognition procedure assists in providing effective learning of individuals through preserving the diversity of the population during the evolution process. In this procedure, each individual is able to learn from the top α% individuals controlled by its knowledge source in the belief space, where the proportion of the affecting subpopulation (α) is adaptively modified during the evolution. The performance of the proposed work has been evaluated on the CEC’2010 and CEC’2013 benchmark suites developed for the special sessions on large-scale global optimization problems. An appropriate comparative study with the best results in the literature is presented. The results confirm how the merits of the improved Cultural Algorithm can achieve superior performance over other cutting-edge algorithms for these data sets.

Guest Editorial Special Issue on Large-Scale Evolutionary Multiobjective Optimization and Its Practical Applications

Complex optimization problems with hundreds or even thousands of decision variables and dozens of conflicting objectives are not uncommon in the real world. In the past five years, increased research efforts have been dedicated to large-scale multiobjective optimization problems (LSMOPs) by using a variety of search strategies, including variable grouping, variable analysis, problem transformation, dimensionality reduction, and novel recombination operators. Despite the success of these efforts in solving some general LSMOPs, there still remains a big gap between the LSMOPs that have been addressed and those encountered in real life, such as sparse, highly constrained, dynamic, and expensive LMOPs, as well as very large-scale and many-objective optimization problems that are widely seen and of paramount importance for solving scientific and engineering problems. Due to the significant practical importance of large-scale multiobjective optimization, there is a high demand for computationally efficient and effective evolutionary algorithms for solving LSMOPs.

A Fuzzy Decision Variables Framework for Large-Scale Multiobjective Optimization

In large-scale multiobjective optimization, too many decision variables hinder the convergence search of evolutionary algorithms. Reducing the search range of the decision space will significantly alleviate this puzzle. With this in mind, this paper proposes a fuzzy decision variables framework for large-scale multiobjective optimization. The framework divides the entire evolutionary process into two main stages: fuzzy evolution and precise evolution. In fuzzy evolution, we blur the decision variables of the original solution to reduce the search range of the evolutionary algorithm in the decision space so that the evolutionary population can quickly converge. The degree of fuzzification gradually decreases with the evolutionary process. Once the population approximately converges, the framework will turn to precise evolution. In precise evolution, the actual decision variables of the solution are directly optimized to increase the diversity of the population so as to be closer to the true Pareto optimal front. Finally, this paper embeds some representative algorithms into the proposed framework and verifies the framework’s effectiveness through comparative experiments on various large-scale multiobjective problems with 500 to 5000 decision variables. Experimental results show that in large-scale multiobjective optimization, the framework proposed in this paper can significantly improve the performance and computational efficiency of multiobjective optimization algorithms.

Evolutionary Large-Scale Multiobjective Optimization: Benchmarks and Algorithms

Evolutionary large-scale multiobjective optimization (ELMO) has received increasing attention in recent years. This study has compared various existing optimizers for ELMO on different benchmarks, revealing that both benchmarks and algorithms for ELMO still need significant improvement. Thus, a new test suite and a new optimizer framework are proposed to further promote the research of ELMO. More realistic features are considered in the new benchmarks, such as mixed formulation of objective functions, mixed linkages in variables, and imbalanced contributions of variables to the objectives, which are challenging to the existing optimizers. To better tackle these benchmarks, a variable group-based learning strategy is embedded into the new optimizer framework for ELMO, which significantly improves the quality of reproduction in large-scale search space. The experimental results validate that the designed benchmarks can comprehensively evaluate the performance of existing optimizers for ELMO and the proposed optimizer shows distinct advantages in tackling these benchmarks.

A class of spectral three-term descent Hestenes-Stiefel conjugate gradient algorithms for large-scale unconstrained optimization and image restoration problems

Conjugate gradient methods are much effective and widely used for large-scale unconstrained optimization problems by their simple computation, low memory requirement and strong global convergence property. Spectral gradient methods are also effective for large-scale problems. In this paper, a class of new descent spectral three-term conjugate gradient algorithms are proposed which automatically have the sufficient descent property and satisfy the Dai-Liao conjugate condition. Under the Wolfe line search technique and some standard conditions, the proposed methods are globally convergent for strongly convex functions and general nonlinear functions with the help of the modified secant equations. In numerical part, 732 problems with dimensions varying from 1500 to 150000 and three image restoration problems with three noise levels are considered. Numerical results indicate that the proposed algorithms are more efficient, reliable and robust than the other methods for the testing problems.

Dual Differential Grouping: A More General Decomposition Method for Large-Scale Optimization.

Cooperative coevolution (CC) algorithms based on variable decomposition methods are efficient in solving large-scale optimization problems (LSOPs). However, many decomposition methods, such as the differential grouping (DG) method and its variants, are based on the theorem of function additively separable, which may not work well on problems that are not additively separable and will result in a bottleneck for CC to solve various LSOPs. This deficiency motivates us to study how the decomposition method can decompose more kinds of separable functions, such as the multiplicatively separable function, to improve the general problem-solving ability of CC on LSOPs. With this concern, this article makes the first attempt to decompose multiplicatively separable functions and proposes a novel method called dual DG (DDG) for better LSOP decomposition and optimization. The novelty and advantage of DDG are that it can be suitable for not only additively separable functions but also multiplicatively separable functions, which can considerably expand the application scope of CC. In this article, we will first define the multiplicatively separable function, and then mathematically show its relationship to the additively separable function and how they can be transformed into each other. Based on this, the DDG can use two kinds of differences to detect the separable structure of both additively and multiplicatively separable functions. In addition, the time complexity of DDG is analyzed and a DDG-based CC algorithm framework is developed for solving LSOPs. To verify the superiority of DDG, experiments and comparisons with some state-of-the-art and champion algorithms are conducted not only on 30 LSOPs based on the test suite of the IEEE CEC large-scale global optimization competition, but also on a case study of the parameter optimization for a neural network-based application.

Modified globally convergent Polak-Ribière-Polyak conjugate gradient methods with self-correcting property for large-scale unconstrained optimization

Modified globally convergent Polak-Ribière-Polyak conjugate gradient methods with self-correcting property for large-scale unconstrained optimization

Large-Scale Bandwidth and Power Optimization for Multi-Modal Edge Intelligence Autonomous Driving

Edge intelligence autonomous driving (EIAD) offers computing resources in autonomous vehicles for training deep neural networks. However, wireless channels between the edge server and the autonomous vehicles are time-varying due to the high-mobility of vehicles. Moreover, the required number of training samples for different data modalities, e.g., images, point-clouds, is diverse. Consequently, when collecting these datasets from vehicles to the edge server, the associated bandwidth and power allocation across all data frames is a large-scale multi-modal optimization problem. This article proposes a highly computationally efficient algorithm that directly maximizes the quality of training (QoT). The key ingredients include a data-driven model for quantifying the priority of data modality and two first-order methods termed accelerated gradient projection and dual decomposition for low-complexity resource allocation. Finally, high-fidelity simulations in Car Learning to Act (CARLA) show that the proposed algorithm reduces the perception error by 3% and the computation time by 98%.

Hybrid Evolutionary Algorithm for Solving the Large-Scale Global Optimization Problems

Hybrid Evolutionary Algorithm for Solving the Large-Scale Global Optimization Problems

A new hybrid conjugate gradient algorithm for unconstrained optimization

It is well known that conjugate gradient methods are useful for solving large-scale unconstrained nonlinear optimization problems. In this paper, we consider combining the best features of two conjugate gradient methods. In particular, we give a new conjugate gradient method, based on the hybridization of the useful DY (Dai-Yuan), and HZ (Hager-Zhang) methods. The hybrid parameters are chosen such that the proposed method satisfies the conjugacy and sufficient descent conditions. It is shown that the new method maintains the global convergence property of the above two methods. The numerical results are described for a set of standard test problems. It is shown that the performance of the proposed method is better than that of the DY and HZ methods in most cases.

An Efficient Adaptive Differential Grouping Algorithm for Large-Scale Black-Box Optimization

Decomposition plays a significant role in cooperative coevolution, which shows great potential in large-scale black-box optimization (LSBO). However, current learning-based decomposition algorithms require many fitness evaluations (FEs) to detect variable interdependencies and encounter the difficulty of threshold setting. To address these issues, this study proposes an efficient adaptive differential grouping (EADG) algorithm. Instead of homogeneously tackling different types of LSBO instances, EADG first identifies the instance type by detecting the interdependencies of a few pairs of variable subsets. Only if the instance is partially separable dose EADG further engage with it by converting its decomposition process into a search process in a binary tree. This facilitates the systematic reutilization of evaluated solutions so that half the interdependencies can be directly deduced without extra FEs. To promote the decomposition accuracy, EADG specially designs a normalized interdependency indicator that can adaptively generate a decomposition threshold according to its ordinal distribution. Theoretical analysis and experimental results show that EADG outperforms current popular decomposition algorithms. Further tests indicate that it can help cooperative coevolution achieve highly competitive optimization performance.

Solution of Large-Scale Many-Objective Optimization Problems Based on Dimension Reduction and Solving Knowledge-Guided Evolutionary Algorithm

There are lots of many-objective optimization problems (MaOPs) in real-world applications, which often have many decision variables. Although a variety of methods have been proposed to solve MaOPs, with the increasing number of decision variables or objective functions, the performance of these algorithms deteriorates appreciably. In view of this, the paper proposes a method to solve large-scale MaOPs (LSMaOPs) based on dimension reduction and a solving knowledge guided evolutionary algorithm. First, a dimension reduction method of objective functions is proposed. By clustering and aggregating the objective functions based on their correlation, the dimension of the original LSMaOP is effectively reduced. In addition, the correlations between the reduced objective functions are relatively low, so they can better represent different preferences. Then, we propose a solving knowledge guided evolutionary algorithm to solve the transformed LSMaOP. In order to get a better set of initial solutions, a population initialization method by mirror partitioning the decision space is given, in which we dynamically modify the sampling probability according to the performance of solutions contained in each sub-domain. At the same time, the algorithm will continuously supplement new excellent individuals using the solving knowledge obtained in the evolution of the population. To examine the performance of the proposed method, we carried out a number of comparative experiments. The experimental results demonstrated that the proposed algorithm can effectively tackle LSMaOPs.

Strengthening Gradient Descent by Sequential Motion Optimization for Deep Neural Networks

In this paper, we explore the advantages of heuristic mechanisms and devise a new optimization framework named Sequential Motion Optimization (SMO) to strengthen gradientbased methods. The key idea of SMO is inspired from a movement mechanism in a recent metaheuristic method called Balancing Composite Motion Optimization (BCMO). Specifically, SMO establishes a sequential motion chain of two gradientguided individuals including a leader and a follower to enhance the effectiveness of parameter updates in each iteration. A surrogate gradient model with low computation cost is theoretically established to estimate the gradient of the follower by that of the leader through chain rule during training process. Experimental results in terms of training quality on both fully-connected multilayer perceptrons (MLPs) and convolutional neural networks (CNNs) with respect to three popular benchmark datasets including MNIST, Fashion-MNIST and CIFAR-10 demonstrate the superior performance of the proposed framework in comparison with the vanilla stochastic gradient descent (SGD) implemented via back-propagation (BP) algorithm. Although this study only introduces the vanilla gradient descent (GD) as a main gradientguided factor in SMO for deep neural networks (DNNs) training application, it is greatly potential to combine with other gradientbased variants to improve its effectiveness and solve other largescale optimization problems in practice.

A Large-Scale Identification Approach for Thermal Parameters of Multichips IGBT Modules Based on LLSO-SQP Algorithm

This paper proposes a large-scale optimization approach for identifying thermal parameters of multi-chip IGBT modules. State-space equation, in which the coefficient matrix comprises the thermal resistance and capacitance, is provided to represent the compact three-dimensional thermal network model. Then, the level-based learning swarm optimization (LLSO) algorithm is firstly utilized to identify large-scale thermal parameters. Additionally, to solve the inefficient convergence problem, the optimization results obtained from the LLSO is provided as the initial value of the sequential quadratic programming (SQP) algorithm to find the global optimal solution. Hence, the proposed LLSO-SQP algorithm can identify the large-scale thermal parameters efficiently and accurately. The training dataset for the algorithm is derived from the transient temperature response of a finite element model (FEM) of the IGBT module under power step excitation. Since only one-time simulation is in-demand, this approach needs less computational effort than others. The identified thermal network model is utilized to estimate junction temperature profiles taking a two-level inverter as a case study. In comparison to that of the experiment and FEM, the results validate the feasibility and accuracy of the junction temperature estimation method based on the compact three-dimensional thermal network model.

An extended fuzzy decision variables framework for solving large-scale multiobjective optimization problems

In large-scale multiobjective optimization, the huge search space poses a great challenge to the convergence search of existing evolutionary algorithms. A fuzzy decision variables (FDV) framework for large-scale multiobjective optimization involves a complex parameter tuning process, and the convergence efficiency is moderate. Therefore, we propose an extended fuzzy decision variables (EFDV) framework based on linear search and the non-dominated rate (the proportion of non-dominated individuals in the whole population) to solve large-scale multiobjective optimization problems. First, guiding solutions are sampled from the central region to provide more possible search directions in large-scale decision space. In addition, fuzzy evolution or precise evolution is determined according to the non-dominated rate. When the non-dominated rate is low, the convergence of the whole population is relatively poor. Then a higher degree of fuzzy search is carried out, and the individuals with poor convergence are reversed to increase the population's diversity and enhance the effectiveness of searches. Finally, by comparing experiments on several large-scale multiobjective test suites with 500–5000 decision variables, the efficiency of the EFDV is confirmed. According to experimental results, the EFDV can significantly enhance the performance and computing effectiveness of multiobjective optimizers when used for large-scale multiobjective optimization.