Mixed-variable Problem Research Articles

Single-objective and multi-objective mixed-variable grey wolf optimizer for joint feature selection and classifier parameter tuning

Feature selection plays an essential role in data preprocessing, which can extract valuable information from extensive data, thereby enhancing the performance of machine learning classification. However, existing feature selection methods primarily focus on selecting feature subsets without considering the impact of classifier parameters on the optimal subset. Different from these works, this paper considers jointly optimizing the feature subset and classifier parameters to minimize the number of features and achieve a low classification error rate. Since feature selection is an optimization problem with binary solution space while classifier parameters involve both continuous and discrete variables, our formulated problem becomes a complex multi-objective mixed-variable problem. To address this challenge, we consider a single-objective optimization method and a multi-objective optimization approach. Specifically, in the single-objective optimization method, we adopt the linear weight method to convert our multiple objectives into a fitness function and then propose a mixed-variable grey wolf optimizer (MGWO) to optimize the function. The proposed MGWO introduces Chaos-Faure initialization, Log convergence factor adjustment, and optimal solution adaptive update operators to enhance its adaptability and balance the global and local search of the algorithm. Subsequently, an improved multi-objective grey wolf optimizer (IMOGWO) is introduced to directly address the problem. The proposed IMOGWO introduces improved initialization, local search, and binary variable mutation operators to balance its exploration and exploitation abilities, making it more suitable for our mixed-variable problem. Extensive simulation results show that our MGWO and IMOGWO outperform recent and classic baselines. Moreover, we also find that jointly optimizing classifier parameters can significantly improve classification accuracy.

Mixed-Variable Global Sensitivity Analysis for Knowledge Discovery and Efficient Combinatorial Materials Design

Abstract Global Sensitivity Analysis (GSA) is the study of the influence of any given input on the outputs of a model. In the context of engineering design, GSA has been widely used to understand both individual and collective contributions of design variables on the design objectives. So far, global sensitivity studies have often been limited to design spaces with only quantitative (numerical) design variables. However, many engineering systems also contain, if not only, qualitative (categorical) design variables in addition to quantitative design variables. In this paper, we integrate Latent Variable Gaussian Process (LVGP) with Sobol’ analysis to develop the first metamodel-based mixed-variable GSA method. Through numerical case studies, we validate and demonstrate the effectiveness of our proposed method for mixed-variable problems. Furthermore, while the proposed GSA method is general enough to benefit various engineering design applications, we integrate it with multi-objective Bayesian optimization (BO) to create a sensitivity-aware design framework in accelerating the Pareto front design exploration for metal-organic framework (MOF) materials with many-level combinatorial design spaces. Although MOFs are constructed only from qualitative variables that are notoriously difficult to design, our method can utilize sensitivity analysis to navigate the optimization in the many-level large combinatorial design space, greatly expediting the exploration of novel MOF candidates.

An efficient mixed-variable generation operator for integrated energy system configuration optimization

Energy consumption has skyrocketed as society has progressed, resulting in energy crises and environmental issues. New energy technologies are being developed to address these challenges, and technologies such as the cogeneration of electricity, heating, and cooling are being used to improve energy utilization. The configuration of an energy system has a critical impact on the economics of the system, its ability to supply energy, etc. This study establishes a multiobjective mixed-variable configuration optimization model for a comprehensive combined cooling, heating, and power energy system. The model considers equipment types (categorical variables), equipment quantities (integer variables), and critical parameters (real variables) as optimization variables, where economic performance, the proportion of renewable energy, supply shortage, and dependence on the external power grid are considered unoptimized objectives. This is a multiobjective and mixed-variable problem, and the combination of the two properties poses an excellent challenge for evolutionary algorithms in optimizing the model. Therefore, we propose an efficient generating operator based on a fully connected weighted neural network, called the FCWN operator. This operator overcomes the challenge of the discrete variable space and broken neighborhood relationships in mixed-variable problems. During the algorithm search process, search history information is used to update the fully connected weight network for distribution estimation of the variable space. Offspring are generated based on the fully connected network to improve the algorithm’s search efficiency. In the experimental section, we construct a model with 20 variables and conduct simulation-based configuration optimization for scenarios with 3, 5, and 8 available equipment types. The final obtained Pareto frontier solution set is evaluated using the hypervolume metric, a widely used multiobjective evaluation metric. The Wilcoxon rank sum test on the experimental results shows that the proposed algorithm has better results than other state-of-the-art algorithms at a 95% confidence level.

Joint Trajectory Design and User Scheduling for Secure Aerial Underlay IoT Systems

Unmanned aerial vehicles (UAVs) have been widely employed to enhance the end-to-end performance of wireless communications since the links between UAVs and terrestrial nodes are line-of-sight (LoS) with high probability. However, the broadcast characteristics of signal propagation in LoS links make them vulnerable to being wiretapped by malicious eavesdroppers, which poses a considerable challenge to the security of wireless communications. In this work, we investigate the security of aerial underlay Internet of Things (IoT) systems by jointly designing trajectory and user scheduling. An airborne base station transmits confidential messages to secondary users utilizing the same spectrum as the primary network. An aerial jammer transmits jamming signals to suppress the eavesdropper to enhance secrecy performance. The uncertainty of eavesdropping node locations is considered, and the average secrecy rate of the secondary user is maximized by optimizing multiple users’ scheduling, the UAVs’ trajectory, and transmit power. To solve the non-convex optimization problem with mixed multi-integer variable problem, we propose an iterative algorithm based on block coordinate descent and successive convex approximation. Numerical results verify the effectiveness of our proposed algorithm and demonstrate that our scheme is beneficial in improving the secrecy performance of aerial underlay IoT systems.

Surrogate-Assisted Hybrid-Model Estimation of Distribution Algorithm for Mixed-Variable Hyperparameters Optimization in Convolutional Neural Networks.

The performance of a convolutional neural network (CNN) heavily depends on its hyperparameters. However, finding a suitable hyperparameters configuration is difficult, challenging, and computationally expensive due to three issues, which are 1) the mixed-variable problem of different types of hyperparameters; 2) the large-scale search space of finding optimal hyperparameters; and 3) the expensive computational cost for evaluating candidate hyperparameters configuration. Therefore, this article focuses on these three issues and proposes a novel estimation of distribution algorithm (EDA) for efficient hyperparameters optimization, with three major contributions in the algorithm design. First, a hybrid-model EDA is proposed to efficiently deal with the mixed-variable difficulty. The proposed algorithm uses a mixed-variable encoding scheme to encode the mixed-variable hyperparameters and adopts an adaptive hybrid-model learning (AHL) strategy to efficiently optimize the mixed-variables. Second, an orthogonal initialization (OI) strategy is proposed to efficiently deal with the challenge of large-scale search space. Third, a surrogate-assisted multi-level evaluation (SME) method is proposed to reduce the expensive computational cost. Based on the above, the proposed algorithm is named s urrogate-assisted hybrid-model EDA (SHEDA). For experimental studies, the proposed SHEDA is verified on widely used classification benchmark problems, and is compared with various state-of-the-art methods. Moreover, a case study on aortic dissection (AD) diagnosis is carried out to evaluate its performance. Experimental results show that the proposed SHEDA is very effective and efficient for hyperparameters optimization, which can find a satisfactory hyperparameters configuration for the CIFAR10, CIFAR100, and AD diagnosis with only 0.58, 0.97, and 1.18 GPU days, respectively.

Bayesian optimization for mixed-variable, multi-objective problems

Optimizing multiple, non-preferential objectives for mixed-variable, expensive black-box problems is important in many areas of engineering and science. The expensive, noisy, black-box nature of these problems makes them ideal candidates for Bayesian optimization (BO). Mixed-variable and multi-objective problems, however, are a challenge due to BO's underlying smooth Gaussian process surrogate model. Current multi-objective BO algorithms cannot deal with mixed-variable problems. We present MixMOBO, the first mixed-variable, multi-objective Bayesian optimization framework for such problems. Using MixMOBO, optimal Pareto-fronts for multi-objective, mixed-variable design spaces can be found efficiently while ensuring diverse solutions. The method is sufficiently flexible to incorporate different kernels and acquisition functions, including those that were developed for mixed-variable or multi-objective problems by other authors. We also present HedgeMO, a modified Hedge strategy that uses a portfolio of acquisition functions for multi-objective problems. We present a new acquisition function, SMC. Our results show that MixMOBO performs well against other mixed-variable algorithms on synthetic problems. We apply MixMOBO to the real-world design of an architected material and show that our optimal design, which was experimentally fabricated and validated, has a normalized strain energy density $10^4$ times greater than existing structures.

Advances in Computational Intelligence of Polymer Composite Materials: Machine Learning Assisted Modeling, Analysis and Design

The superior multi-functional properties of polymer composites have made them an ideal choice for aerospace, automobile, marine, civil, and many other technologically demanding industries. The increasing demand of these composites calls for an extensive investigation of their physical, chemical and mechanical behavior under different exposure conditions. Machine learning (ML) has been recognized as a powerful predictive tool for data-driven multi-physical modeling, leading to unprecedented insights and exploration of the system properties beyond the capability of traditional computational and experimental analyses. Here we aim to abridge the findings of the large volume of relevant literature and highlight the broad spectrum potential of ML in applications like prediction, optimization, feature identification, uncertainty quantification, reliability and sensitivity analysis along with the framework of different ML algorithms concerning polymer composites. Challenges like the curse of dimensionality, overfitting, noise and mixed variable problems are discussed, including the latest advancements in ML that have the potential to be integrated in the field of polymer composites. Based on the extensive literature survey, a few recommendations on the exploitation of various ML algorithms for addressing different critical problems concerning polymer composites are provided along with insightful perspectives on the potential directions of future research.

Optimal Solution of Reactive Power Dispatch in Transmission System to Minimize Power Losses Using Sine-Cosine Algorithm

Optimal reactive power dispatch (ORPD) has a crucial impact to enhance safety, reliability, and economical operation of the electric power system. ORPD is a non-linear, non-convex and mixed variable problem, which has been solved by many researchers via different meta-heuristic algorithms during the last decade. In this work, a novel algorithm named sine-cosine algorithm (SCA) is utilized to solve ORPD problem by considering both dependent and independent control variable constraints. SCA has been tested and validated on standard 14, 30 and 57-bus power systems. To validate the superiority of proposed algorithm, the outcomes obtained through SCA are compared with recent published results attained through particle swarm optimization (PSO), modified Gaussian barebones teaching–learning based optimization (BBTLBO), ant bee colony optimization (ABCO), whale optimization algorithm (WOA) and backtracking search algorithms (BSA). The results attained using SCA show the improvement in the power losses minimization. Thus, with standard 14-bus system, the power losses are minimized from 0.04% to 4.78%. While, using standard 30-bus, the power losses are minimized from 0.4% to 3.4% and with standard 57-bus, power losses are reduced from 0.9% to 1.99%. Furthermore, a comparative analysis with 30 independent runs on the above-mentioned bus systems is performed to examine the functioning of the proposed method in terms of probability density function (PDF) and cumulative density function (CDF). For such analysis, well-known meta-heuristic algorithms such as PSO, WOA, differential evolution (DE) are compared with proposed SCA in solving the ORPD problem. The results of this analysis clearly show that proposed algorithm is robust, effective, and computationally easy in solving the ORPD problem compared to the existing meta-heuristic algorithms.

Reliability-based Design Optimization of Laminated Composite Structures under Delamination and Material Property Uncertainties

Delamination is a major type of defects from manufacturing process in laminated composite structures, exhibiting uncertain characteristics which are represented by mixed random variables of continuous in-plane position and delamination size, as well as discrete through-the-thickness position, which should be considered in the design phase of composite structures. Thus, this paper presents a reliability-based design framework for optimal design of composite stacking sequence, for the first time to consider both delamination and material property uncertainties from manufacturing process. Mixed continuous-discrete random variables are involved, and a reliability analysis method is newly proposed to tackle this mixed-variable problem, maintaining high levels of both accuracy and efficiency. The formula of total probability is first employed to formulate the reliability constraint so that discrete and continuous random variables are decoupled; Monte Carlo simulation is then used for reliability analysis with respect to continuous random variables, and a surrogate modelling approach based on Gaussian process regression model is adopted to reduce computation costs. As design variables of ply angles are limited to a discrete set, a genetic algorithm with some proposed improvements is used to handle discrete design variables. Consequently, an optimization framework is devised for composite laminate design under uncertainties from manufacturing process, which is verified via case studies of a cantilever composite plate and can also be extended as a general solution for other composite laminates. With integrated consideration of manufacturing limitations and imperfections as well as structural performances, the presented framework provides a valuable tool in composite structure design.

Cohort intelligence with self-adaptive penalty function approach hybridized with colliding bodies optimization algorithm for discrete and mixed variable constrained problems

Recently, several socio-/bio-inspired algorithms have been proposed for solving a variety of problems. Generally, they perform well when applied for solving unconstrained problems; however, their performance degenerates when applied for solving constrained problems. Several types of penalty function approaches have been proposed so far for handling linear and non-linear constraints. Even though the approach is quite easy to understand, the precise choice of penalty parameter is very much important. It may further necessitate significant number of preliminary trials. To overcome this limitation, a new self-adaptive penalty function (SAPF) approach is proposed and incorporated into socio-inspired Cohort Intelligence (CI) algorithm. This approach is referred to as CI–SAPF. Furthermore, CI–SAPF approach is hybridized with Colliding Bodies Optimization (CBO) algorithm referred to as CI–SAPF–CBO algorithm. The performance of the CI–SAPF and CI–SAPF–CBO algorithms is validated by solving discrete and mixed variable problems from truss structure domain, design engineering domain, and several problems of linear and nonlinear in nature. Furthermore, the applicability of the proposed techniques is validated by solving two real-world applications from manufacturing engineering domain. The results obtained from CI–SAPF and CI–SAPF–CBO are promising and computationally efficient when compared with other nature inspired optimization algorithms. A non-parametric Wilcoxon’s rank sum test is performed on the obtained statistical solutions to examine the significance of CI–SAPF–CBO. In addition, the effect of the penalty parameter on pseudo-objective function, penalty function and constrained violations is analyzed and discussed along with the advantages over other algorithms.

On the implementation of a global optimization method for mixed-variable problems

We describe the optimization algorithm implemented in the open-source derivative-free solver RBFOpt. The algorithm is based on the radial basis function method of Gutmann and the metric stochastic response surface method of Regis and Shoemaker. We propose several modifications aimed at generalizing and improving these two algorithms: (i) the use of an extended space to represent categorical variables in unary encoding; (ii) a refinement phase to locally improve a candidate solution; (iii) interpolation models without the unisolvence condition, to both help deal with categorical variables, and initiate the optimization before a uniquely determined model is possible; (iv) a master-worker framework to allow asynchronous objective function evaluations in parallel. Numerical experiments show the effectiveness of these ideas.

Niching comprehensive learning gravitational search algorithm for multimodal optimization problems

Many niching algorithms have been proposed in recent times to cater complex real-world problems having several global/local optima. These algorithms help in locating as many global/local optima as possible. However, to explore the search space along with locating and maintaining a large number of global solutions is always a challenging problem. Hence, we propose a new niching strategy named, ‘niching Comprehensive Learning Gravitational Search algorithm’, in which the CLGSA algorithm is used to solve complex problems having multiple solutions. The proposed algorithm first efficiently tries to explore the search space without trapping in local optima’s and secondly it locates all possible global optima’s. This algorithm has an external memory that enhances its capability to find the maximum number of optima’s. The proposed approach is tested on the IEEE CEC-2013 Multimodal Optimization benchmark problems (Li et al. in Evolutionary computation and machine learning group, RMIT University, Melbourne, Australia, Technical Report, 2013). The results of the proposed scheme are compared with five other state-of-the-art methods. In order to show the efficiency of CLGSA algorithm over real-life multimodal optimization problems, it applied to non-linear, and a mixed variable problem this is, the Reactive Power Dispatch problem (RPD). A standard benchmark IEEE-57 bus system is considered to evaluate the efficiency of the CLGSA. The evaluated simulation results show that CLGSA algorithm successfully solve multimodal problems and the RPD problem with significant accuracy.

A novel ISLE-ESE method for mixed-variable experiment design

Discrete-continuous mixed-variable problem is an inevitable problem in engineering design. However, few researches pay attention to discrete variables. In this paper, a novel method is proposed for solving mixed-variable problem by combing the Enhanced Stochastic Evolution (ESE) method and the improved Successive Local Enumeration (SLE) method. The continuous part of design variables is generated first. And then the improved SLE method is utilized to generate the discrete part. Finally, an overall optimization is conducted utilizing ESE method taking the continuous part of design variables as input. Results of numerical tests show that the proposed method performs better in both space-filling property and orthogonality. Besides, the computation effort is also saved compared to the ESE method.

Surrogate-based optimization for mixed-integer nonlinear problems

Simulation-based optimization using surrogate models enables decision-making through the exchange of data from high-fidelity models and development of approximations. Many chemical engineering optimization problems, such as process design and synthesis, rely on simulations and contain both discrete and continuous decision variables. Surrogate-based optimization with continuous variables has been studied extensively; however, there are many open challenges for the case of mixed-variable inputs. In this work, we propose an algorithm for mixed-integer nonlinear simulation-based problems that uses adaptive sampling and surrogate modeling with one-hot encoding. We propose techniques for the design of experiments for mixed-variable problems, surrogate modeling for mixed-variable response surfaces, and iterative approximation-optimization procedure that leads to optimal solutions. Results show that one-hot encoding leads to accurate and robust mixed-variable Gaussian Process and Neural Network models that are effective surrogates for optimization. The proposed algorithm is tested on mixed-integer nonlinear benchmark problems and a chemical process synthesis case study.

Improved genetic algorithm with two-level multipoint approximation for complex frame structural optimization

In this paper, an improved structural topology and sizing optimization method is developed for the fast and efficient engineering design of complex frame structures where beam elements are mainly used in the structures. Discrete and continuous variables are included that the elimination or existence of beam elements are treated as discrete variables (0,1), and the continuous sizing variables of beam cross sections are considered to be continuous variables. To solve the mixed variable problem, the paper introduces a two-level multipoint approximation strategy (TMA). The first-level approximate problem is established by using the branched multipoint approximate function, which includes both two types of variables. Genetic algorithm (GA) is used to determine the absence or presence of beam members. The second-level approximate problem that only involving retained continuous size variables is made on this basis, which uses Taylor expansion and dual methods to solve the inner layer continuous optimization problem. Meanwhile, a strategy of adding a new complementary design point is adopted to expend the search scopes and improve the precision. Temporal deletion techniques are used to temporarily remove redundant constraints and local vibration modes processing techniques are used for continuum topology optimization under frequency constraints. Several representative examples are investigated to validate the effectiveness of the improved method.

Bayesian Optimization for Materials Design with Mixed Quantitative and Qualitative Variables

Although Bayesian Optimization (BO) has been employed for accelerating materials design in computational materials engineering, existing works are restricted to problems with quantitative variables. However, real designs of materials systems involve both qualitative and quantitative design variables representing material compositions, microstructure morphology, and processing conditions. For mixed-variable problems, existing Bayesian Optimization (BO) approaches represent qualitative factors by dummy variables first and then fit a standard Gaussian process (GP) model with numerical variables as the surrogate model. This approach is restrictive theoretically and fails to capture complex correlations between qualitative levels. We present in this paper the integration of a novel latent-variable (LV) approach for mixed-variable GP modeling with the BO framework for materials design. LVGP is a fundamentally different approach that maps qualitative design variables to underlying numerical LV in GP, which has strong physical justification. It provides flexible parameterization and representation of qualitative factors and shows superior modeling accuracy compared to the existing methods. We demonstrate our approach through testing with numerical examples and materials design examples. The chosen materials design examples represent two different scenarios, one on concurrent materials selection and microstructure optimization for optimizing the light absorption of a quasi-random solar cell, and another on combinatorial search of material constitutes for optimal Hybrid Organic-Inorganic Perovskite (HOIP) design. It is found that in all test examples the mapped LVs provide intuitive visualization and substantial insight into the nature and effects of the qualitative factors. Though materials designs are used as examples, the method presented is generic and can be utilized for other mixed variable design optimization problems that involve expensive physics-based simulations.

Iterative Most Probable Point Search Method for Problems With a Mixture of Random and Interval Variables

Abstract To represent input variability accurately, an input distribution model for random variables should be constructed using many data. However, for certain input variables, engineers may have only their intervals, which represent input uncertainty. In practical engineering applications, both random and interval variables could exist at the same time. To consider both input variability and uncertainty, inverse reliability analysis should be carried out considering both random and interval variables—mixed variables—and their mathematical correlation in a performance measure. In this paper, an iterative most probable point (MPP) search method has been developed for the mixed-variable problem. The update procedures for MPP search are developed considering the features of mixed variables in the inverse reliability analysis. MPP search for random and interval variables proceed simultaneously to consider the mathematical correlation. An interpolation method is introduced to find a better candidate MPP without additional function evaluations. Mixed-variable design optimization (MVDO) has been formulated to obtain cost-effective and reliable design in the presence of mixed variables. In addition, the design sensitivity of a probabilistic constraint has been developed for an effective and efficient MVDO procedure. Using numerical examples, it is found that the developed MPP search method finds an accurate MPP more efficiently than the generic optimization method does. In addition, it is verified that the developed method enables the MVDO process with a small number of function evaluations.

Efficient global optimization of constrained mixed variable problems

Due to the increasing demand for high performance and cost reduction within the framework of complex system design, numerical optimization of computationally costly problems is an increasingly popular topic in most engineering fields. In this paper, several variants of the Efficient Global Optimization algorithm for costly constrained problems depending simultaneously on continuous decision variables as well as on quantitative and/or qualitative discrete design parameters are proposed. The adaptation that is considered is based on a redefinition of the Gaussian Process kernel as a product between the standard continuous kernel and a second kernel representing the covariance between the discrete variable values. Several parameterizations of this discrete kernel, with their respective strengths and weaknesses, are discussed in this paper. The novel algorithms are tested on a number of analytical test-cases and an aerospace related design problem, and it is shown that they require fewer function evaluations in order to converge towards the neighborhoods of the problem optima when compared to more commonly used optimization algorithms.

Cohort intelligence algorithm for discrete and mixed variable engineering problems

In this study, the performance of an emerging socio inspired metaheuristic optimization technique referred to as Cohort Intelligence (CI) algorithm is evaluated on discrete and mixed variable nonlinear constrained optimization problems. The investigated problems are mainly adopted from discrete structural optimization and mixed variable mechanical engineering design domains. For handling the discrete solution variables, a round off integer sampling approach is proposed. Furthermore, in order to deal with the nonlinear constraints, a penalty function method is incorporated. The obtained results are promising and computationally more efficient when compared to the other existing optimization techniques including a Multi Random Start Local Search algorithm. The associated advantages and disadvantages of CI algorithm are also discussed evaluating the effect of its two parameters namely the number of candidates, and sampling space reduction factor.

A Hybrid-coded Human Learning Optimization for mixed-variable optimization problems

Human Learning Optimization (HLO) is an emerging meta-heuristic with promising potential, which is inspired by human learning mechanisms. Although binary algorithms like HLO can be directly applied to mixed-variable problems that contains both continuous values and discrete or Boolean values, the search efficiency and the performance of those algorithms may be significantly spoiled due to “the curse of dimensionality” caused by the binary coding strategy especially when the continuous parameters of problems require high accuracy. Therefore, this paper extends HLO and proposes a novel hybrid-coded HLO (HcHLO) framework to tackle mix-coded problems more efficiently and effectively, in which real-coded parameters are optimized by a new continuous HLO (CHLO) based on the linear learning mechanism of humans and the other variables are handled by the binary learning operators of HLO. Finally, HcHLO is adopted to solve 14 benchmark problems and its performance is compared with that of recent meta-heuristic algorithms. The experimental results show that the proposed HcHLO achieves the best-known overall performance so far on the test problems, which demonstrates the validity and superiority of HcHLO.