Large-scale Optimization Algorithms Research Articles

High-efficient sample point transform algorithm for large-scale complex optimization

Decomposition algorithms and surrogate model methods are frequently employed to address large-scale, intricate optimization challenges. However, the iterative resolution phase inherent to decomposition algorithms can potentially alter the background vector, leading to the repetitive evaluation of samples across disparate iteration cycles. This phenomenon significantly diminishes the computational efficiency of optimization. Accordingly, a novel approach, designated the Sample Point Transformation Algorithm (SPTA), is put forth in this paper as a means of enhancing efficiency through a process of mathematical deduction. The mathematical deduction reveals that the difference between sample points in each iteration loop is a simple function related to the inter-group dependent variables. Consequently, the SPTA method achieves the comprehensive transformation of the sample set by establishing a surrogate model of the difference between the sample sets of two cycles with a limited number of sample points, as opposed to conducting a substantial number of repeated samplings. This SPTA is employed to substitute the most time-consuming step of direct calculation in the classical optimization process. To validate the calculation efficiency, a series of numerical examples were conducted, demonstrating an improvement of approximately 75 % while maintaining optimal accuracy. This illustrates the advantage of the SPTA in addressing large-scale and complex optimization problems.

A population hierarchical-based evolutionary algorithm for large-scale many-objective optimization

In large-scale many-objective optimization problems (LMaOPs), the performance of algorithms faces significant challenges as the number of objective functions and decision variables increases. The main challenges in addressing this type of problem are as follows: the large number of decision variables creates an enormous decision space that needs to be explored, leading to slow convergence; and the high-dimensional objective space presents difficulties in selecting dominant individuals within the population. To address this issue, this paper introduces an evolutionary algorithm based on population hierarchy to address LMaOPs. The algorithm employs different strategies for offspring generation at various population levels. Initially, the population is categorized into three levels by fitness value: poorly performing solutions with higher fitness (Ph), better solutions with lower fitness (Pl), and excellent individuals stored in the archive set (Pa). Subsequently, a hierarchical knowledge integration strategy (HKI) guides the evolution of individuals at different levels. Individuals in Pl generate offspring by integrating differential knowledge from Pa and Ph, while individuals in Ph generate offspring by learning prior knowledge from Pa. Finally, using a cluster-based environment selection strategy balances population diversity and convergence. Extensive experiments on LMaOPs with up to 10 objectives and 5000 decision variables validate the algorithm’s effectiveness, demonstrating superior performance.

A New Hybrid Descent Algorithm for Large-Scale Nonconvex Optimization and Application to Some Image Restoration Problems

Conjugate gradient methods are widely used and attractive for large-scale unconstrained smooth optimization problems, with simple computation, low memory requirements, and interesting theoretical information on the features of curvature. Based on the strongly convergent property of the Dai–Yuan method and attractive numerical performance of the Hestenes–Stiefel method, a new hybrid descent conjugate gradient method is proposed in this paper. The proposed method satisfies the sufficient descent property independent of the accuracy of the line search strategies. Under the standard conditions, the trust region property and the global convergence are established, respectively. Numerical results of 61 problems with 9 large-scale dimensions and 46 ill-conditioned matrix problems reveal that the proposed method is more effective, robust, and reliable than the other methods. Additionally, the hybrid method also demonstrates reliable results for some image restoration problems.

A dual-sampling based evolutionary algorithm for large-scale multi-objective optimization

The vast search space in large-scale multi-objective optimization represents a significant challenge for evolutionary algorithms to converge towards the Pareto Front. As an effective search strategy, direction-guided sampling technique could improve the search efficiency by exploring along the approximated directions to approach the Pareto set. However, the approximated directions may fail to interact with the true Pareto set and result in inefficient search. To address this issue, a dual-sampling method is proposed in this paper. In addition to the samples along the directions approximated by direction-guided sampling, fuzzy Gaussian sampling is applied to adjust the search direction and generate more accurate and evenly distributed solutions. Moreover, a convergence-and-diversity-based mating selection is introduced to balance the exploration and exploitation. The experiments on 72 test benchmarks with bi- and tri-objectives and 500 to 5000 decision variables show the superiority of the proposed algorithm compare with the state-of-the-art algorithms.

A multi-population multi-stage adaptive weighted large-scale multi-objective optimization algorithm framework

Weighted optimization framework (WOF) achieves variable dimensionality reduction by grouping variables and optimizing weights, playing an important role in large-scale multi-objective optimization problems. However, because of possible problems such as duplicate weight vectors in the selection process and loss of population diversity, the algorithm is susceptible to local optimization. Therefore, this paper develops an algorithm framework called multi-population multi-stage adaptive weighted optimization (MPSOF) to improve the performance of WOF in two aspects. First, the method of using multi-population is employed to address the issue of insufficient algorithmic diversity, while simultaneously reducing the likelihood of converging towards local optima. Secondly, a processing stage is incorporated into MPSOF, where a certain number of individuals are adaptively selected for updating based on the weight information and evolutionary status of different subpopulations, targeting different types of weights. This approach alleviates the impact of repetitive weights on the diversity of newly generated individuals, avoids the drawback of easily converging to local optima when using a single type of weight for updating, and effectively balances the diversity and convergence of subpopulations. Experiments of three types designed on several typical function sets demonstrate that MPSOF exceeds the comparison algorithms in the three metrics for Inverse Generation Distance, Hypervolume and Spacing.

An improved problem transformation algorithm for large-scale multi-objective optimization

Solving Large-Scale Multi-Objective Optimization Problems (LSMOPs) is the major challenge in evolution computation. Due to the large number of decision variables involved, it is not easy for existing multi-objective evolutionary algorithms to search the entire decision variable space with limited computation resources. To address the above problem, a large-scale multi-objective evolutionary algorithm based on problem transformation, called LSMOEA/PT, is presented in this paper. LSMOEA/PT uses an improved problem transformation strategy to transform LSMOPs into small-scale multi-objective problems to accelerate convergence speed and reduce computational resources. Specifically, The transformation strategy selects reference solutions to initialize a weight population by applying the opposite individual method. After optimizing the weight population, a dual line strategy is implemented to combine the weight vectors with the decision variables from the original population, thereby generating a new population. The new population is combined with the original population for environment selection. In LSMOEA/PT, a double learning swarm optimizer is introduced to speed up the convergence of the population. A uniform distribution strategy is conducted to update transformed solutions after the problem transformation process, increasing the diversity of the population. Experimental results show that LSMOEA/PT is competitive with eleven state-of-the-art algorithms on large-scale multi-objective optimization test problems with up to 2000 decision variables.

An agent-assisted heterogeneous learning swarm optimizer for large-scale optimization

In large-scale optimization problems, the PSO algorithm based on population information interaction performs well. However, the algorithm typically only uses population information for updating, neglecting potentially useful information during evolution. Furthermore, the pressure of computing resources and complex evolution processes pose challenges to search and balancing strategy. In this paper, an agent-assisted heterogeneous learning swarm optimizer is proposed. First, a global and local search collaborative particle learning structure is proposed, called a heterogeneous learning structure. The population distribution in the decision and objective spaces are used to select global and local search examples, respectively. Secondly, an agent-assisted evolutionary search is proposed, which evaluates the individual’s state in decision space and objective space and controls the global and local search structures to adapt to the evolutionary requirements of individuals in different evolutionary states. Finally, hierarchical resource allocation and level-based global search example selection mechanisms are proposed on the local and global search structures to maintain the balance between diversity and convergence. To evaluate the performance of the proposed algorithm, comprehensive and extensive experiments were conducted on a commonly used large-scale benchmark suite. Experiments show that compared with ten state-of-the-art large-scale optimization algorithms, the proposed algorithm demonstrates superior performance in most cases.

A two-stage direction-guided evolutionary algorithm for large-scale multiobjective optimization

Large-scale multiobjective optimization problems (LSMOPs) have exponential growth in the search space as the decision variables increase, and the vast search space poses a challenge to the performance of multiobjective evolutionary algorithms (MOEAs). Many current large-scale MOEAs need to consume a large amount of computational resources to get good performance. This paper proposes a two-stage direction-guided evolutionary algorithm for large-scale multiobjective optimization (LMOEA-S2D) to balance the performance and computational resource overhead. The algorithm exploits the Pareto-optimality property of domination and the diversity-preserving property of decomposition to optimize the performance in the two stages, respectively, and designs a corresponding direction-guided mechanism to improve search efficiency. LMOEA-S2D designs global direction search and local direction search in the domination-based stage for efficient exploitation to accelerate population convergence. To promote greater population diversity, a hybrid direction search was devised to aid diversity exploration in the decomposition-based stage, and this facilitates even distribution of candidate solutions across the Pareto optimal frontier. LMOEA-S2D is compared with five state-of-the-art large-scale MOEAs on some large-scale multiobjective test suites with 100 to 5,000 decision variables. The experimental results show that LMOEA-S2D significantly outperformed all compared algorithms under limited computational resources.

A similarity-detection-based evolutionary algorithm for large-scale multimodal multi-objective optimization

In recent years, there has been a surge in the development of evolutionary algorithms tailored for multimodal multi-objective optimization problems (MMOPs). These algorithms aim to find multiple equivalent Pareto optimal solution sets (PSs). However, little work has been done on MMOPs with large-scale decision variables, especially when the Pareto optimal solutions are sparse. These problems pose significant challenges due to the dimension curse, the unknown sparsity, and the unknown number of equivalent PSs. In this paper, we propose an evolutionary algorithm based on similarity detection called SD-MMEA to solve large-scale MMOPs with sparse Pareto-optimal solutions. Specifically, it employs a multi-population independent evolution to explore multiple PSs and distinguishes different PSs by double detection of the similarity between subpopulations. Simultaneously, develop online scoring mechanisms for decision variables to guide the subpopulations to explore in different directions. In addition, during the latter stage of evolution, the decision variables of individuals are further optimized by a double-layer grouping process. The proposed algorithm is compared with six state-of-the-art algorithms. Experimental results show that SD-MMEA has significant advantages in solving large-scale MMOPs with sparse solutions.

Improving Performance Insensitivity of Large-Scale Multiobjective Optimization via Monte Carlo Tree Search.

The large-scale multiobjective optimization problem (LSMOP) is characterized by simultaneously optimizing multiple conflicting objectives and involving hundreds of decision variables. Many real-world applications in engineering can be modeled as LSMOPs; simultaneously, engineering applications require insensitivity in performance. This requirement typically means that the algorithm should not only produce good results in terms of performance for every run but also the performance of multiple runs should not fluctuate too much. However, existing large-scale multiobjective optimization algorithms often focus on improving algorithm performance, but pay little attention to improving the insensitivity characteristic of algorithms. This directly leads to substantial limitations when solving practical problems. In this work, we propose an evolutionary algorithm called large-scale multiobjective optimization algorithm via Monte Carlo tree search, which is based on the Monte Carlo tree search and aims to improve the performance and insensitivity of solving LSMOPs. The proposed method samples decision variables to construct new nodes on the Monte Carlo tree for optimization and evaluation, and it selects nodes with good evaluations for further searches in order to reduce the performance sensitivity caused by large-scale decision variables. We propose two metrics to measure the sensitivity of the algorithm and compare the proposed algorithm with several state-of-the-art designs on different benchmark functions and metrics. The experimental results confirm the effectiveness and performance insensitivity of the proposed design for solving LSMOPs.

A multi-granularity clustering based evolutionary algorithm for large-scale sparse multi-objective optimization

Sparse multi-objective optimization problems (SMOPs) frequently exist in a variety of disciplines such as machine learning, economy, and signal processing. Evolutionary algorithms have demonstrated their proficiency in optimizing complex problems in recent years, although their performance often deteriorates significantly on large-scale SMOPs. In an effort to accelerate the convergence, this paper suggests a multi-granularity variable clustering method for evolutionary algorithms. This method estimates the sparse distribution of decision variables at each generation and partitions them into a varying number of layers, each with a distinct probability of being zero. These clustering outcomes inspire the development of a crossover operator and a mutation operator, which prove adept at efficiently generating sparse solutions. Experimental evaluations on both benchmark and real-world SMOPs confirm that an evolutionary algorithm incorporating the new crossover operator and mutation operator converges more rapidly than its state-of-the-art counterparts.

Large-scale evolutionary optimization: A review and comparative study

Large-scale global optimization (LSGO) problems have widely appeared in various real-world applications. However, their inherent complexity, coupled with the curse of dimensionality, makes them challenging to solve. Continuous efforts have been devoted to designing computational intelligence-based approaches to solve them. This paper offers a comprehensive review of the latest developments in the field, focusing on the advances in both single-objective and multi-objective large-scale evolutionary optimization algorithms over the past five years. We systematically categorize these algorithms, discuss their distinct features, and highlight benchmark test suites essential for performance evaluation. After that, comparative studies are conducted using numerical solutions to evaluate the performance of state-of-the-art LSGO for both single-objective and multi-objective problems. Finally, we discuss the real-world applications of LSGO, some challenges, and possible future research directions.

Activation Function-Assisted Objective Space Mapping to Enhance Evolutionary Algorithms for Large-Scale Many-Objective Optimization

Activation Function-Assisted Objective Space Mapping to Enhance Evolutionary Algorithms for Large-Scale Many-Objective Optimization

A Two-Population Algorithm for Large-Scale Multi-objective Optimization Based on Fitness-Aware Operator and Adaptive Environmental Selection

A Two-Population Algorithm for Large-Scale Multi-objective Optimization Based on Fitness-Aware Operator and Adaptive Environmental Selection

Large-Scale Multi-Objective Optimization Algorithm Based on Weighted Overlapping Grouping of Decision Variables

Large-Scale Multi-Objective Optimization Algorithm Based on Weighted Overlapping Grouping of Decision Variables

A problem transformation-based and decomposition-based evolutionary algorithm for large-scale multiobjective optimization

For large-scale multi-objective optimization problems, the search space becomes exceptionally vast, resulting in increased complexity in the search process. The search space usually contains several local optimal individuals, and the difficulty of finding the global optimal individuals increases greatly. A problem transformation-based and decomposition-based large-scale multi-objective evolutionary algorithm is proposed to solve the problem of reducing the dimensionality of the search space and the optimization of populations. Reducing the number of optimization problems in the decision space through problem transformation has lowered the complexity of multi-objective optimization problems and enhanced computational efficiency. In the decision space, employing two direction vectors adaptively guides the generation of promising individuals, thereby preventing the population from falling into local optima. Due to the dimensionality reduction of the decision space and the optimization of the individuals in the objective space, a set of optimal solutions are effectively obtained and uniformly distributed on the approximate Pareto optimal front. Experimental results show that the algorithm is highly competitive with five large-scale multi-objective evolutionary algorithms for large-scale multi-objective optimization test problems with up to 2000 decision variables.

Parallel-in-time multiple shooting for optimal control problems governed by the Navier–Stokes equations

In the past decades, multiple shooting methods have proven to be a promising direction to speed up the optimization process, especially in the context of ODE-based optimization. Very recently, Fang et al. (Journal of Computational Physics, vol. 452, 110926, 2022) proposed a multiple shooting algorithm for large-scale PDE-constrained optimization. The current paper continues this line of work and explores the potential of multiple shooting methods for optimal control problems governed by the three-dimensional Navier–Stokes equations. Similar to Fang et al., the augmented Lagrangian (AL) method is used to solve the resulting equality-constrained optimization problem, and we employ the classical limited-memory BFGS method for the unconstrained subproblems inside the AL loop. In the current work, we exploit the multiple shooting paradigm in full by processing the shooting windows parallel-in-time, allowing for significant parallel speed-ups compared to single shooting. The proposed method is validated on a velocity tracking case, using up to 100 windows. Our analysis shows that the multiple shooting algorithm allows for considerable algorithmic and parallel speed-ups. While algorithmic speed-up depends on the exact tracking case and initialization of the shooting windows, the multiple shooting algorithm always outperforms single shooting in terms of computational time (if the number of windows is sufficiently high) due to the parallel-in-time implementation. For a given amount of resources, we also show that the proposed parallel-in-time strategy can outperform spatial parallelization alone, especially when the spatial parallelization is saturated.

Approximating sparse Hessian matrices using large-scale linear least squares

Large-scale optimization algorithms frequently require sparse Hessian matrices that are not readily available. Existing methods for approximating large sparse Hessian matrices have limitations. To try and overcome these, we propose a novel approach that reformulates the problem as the solution of a large linear least squares problem. The least squares problem is sparse but can include a number of rows that contain significantly more entries than other rows and are regarded as dense. We exploit recent work on solving such problems using either the normal equations or an augmented system to derive a robust approach for computing approximate sparse Hessian matrices. Example sparse Hessians from the CUTEst test problem collection for optimization illustrate the effectiveness and robustness of the new method.

Coil sketching for computationally efficient MR iterative reconstruction.

Parallel imaging and compressed sensing reconstructions of large MRI datasets often have a prohibitive computational cost that bottlenecks clinical deployment, especially for three-dimensional (3D) non-Cartesian acquisitions. One common approach is to reduce the number of coil channels actively used during reconstruction as in coil compression. While effective for Cartesian imaging, coil compression inherently loses signal energy, producing shading artifacts that compromise image quality for 3D non-Cartesian imaging. We propose coil sketching, a general and versatile method for computationally-efficient iterative MR image reconstruction. We based our method on randomized sketching algorithms, a type of large-scale optimization algorithms well established in the fields of machine learning and big data analysis. We adapt the sketching theory to the MRI reconstruction problem via a structured sketching matrix that, similar to coil compression, considers high-energy virtual coils obtained from principal component analysis. But, unlike coil compression, it also considers random linear combinations of the remaining low-energy coils, effectively leveraging information from all coils. First, we performed ablation experiments to validate the sketching matrix design on both Cartesian and non-Cartesian datasets. The resulting design yielded both improved computatioanal efficiency and preserved signal-to-noise ratio (SNR) as measured by the inverse g-factor. Then, we verified the efficacy of our approach on high-dimensional non-Cartesian 3D cones datasets, where coil sketching yielded up to three-fold faster reconstructions with equivalent image quality. Coil sketching is a general and versatile reconstruction framework for computationally fast and memory-efficient reconstruction.

Stochastic Localization Methods for Convex Discrete Optimization via Simulation

Solving Convex Discrete Optimization via Simulation via Stochastic Localization Algorithms Many decision-making problems in operations research and management science require the optimization of large-scale complex stochastic systems. For a number of applications, the objective function exhibits convexity in the discrete decision variables or the problem can be transformed into a convex one. In “Stochastic Localization Methods for Convex Discrete Optimization via Simulation,” Zhang, Zheng, and Lavaei propose provably efficient simulation-optimization algorithms for general large-scale convex discrete optimization via simulation problems. By utilizing the convex structure and the idea of localization and cutting-plane methods, the developed stochastic localization algorithms demonstrate a polynomial dependence on the dimension and scale of the decision space. In addition, the simulation cost is upper bounded by a value that is independent of the objective function. The stochastic localization methods also exhibit a superior numerical performance compared with existing algorithms.