Adaptive operator selection in heuristic optimization utilizing generalized experience with reinforcement learning

Combinatorial optimisation problems are known as unpredictable and challenging due to their nature and complexity. One way to reduce the unpredictability of such problems is to identify features and the characteristics that can be utilised to guide the search using domain-knowledge and act accordingly. Many problem solving algorithms use multiple complementary operators in patterns to handle such unpredictable cases. A well-characterised search space may help to evaluate the problem states better and select/apply a neighbourhood operator to generate more productive new problem states that allow for a smoother path to the final/optimum solutions. This applies to the algorithms that use multiple operators to solve problems. However, the remaining challenge is determining how to select an operator in an optimal way from the set of operators while taking the search space conditions into consideration. Recent research shows the success of adaptive operator selection to address this problem. However, efficiency and scalability issues persist in this regard. In addition, selecting the most representative features remains crucial in addressing problem complexity and inducing commonality for transferring experience across domains. This paper investigates if a problem can be represented by a number of features identified by landscape analysis, and whether an adaptive operator selection scheme can be constructed using Machine Learning (ML) techniques to address the efficiency and scalability problem. The proposed method determines the optimal categorisation by analysing the predictivity of a set of features using the most well-known supervised ML techniques. The identified set of features is then used to construct an adaptive operator selection scheme. The findings of the experiments demonstrate that supervised ML algorithms are highly effective when building adaptable operator selectors.

The adaptive operator selection (AOS) and the adaptive parameter control are widely used to enhance the search power in many multiobjective evolutionary algorithms. This paper proposes a novel adaptive selection strategy with bandits for the multiobjective evolutionary algorithm based on decomposition (MOEA/D), named latest stored information based adaptive selection (LSIAS). An improved upper confidence bound (UCB) method is adopted in the strategy, in which the operator usage rate and abandonment of extreme fitness improvement are introduced to improve the performance of UCB. The strategy uses a sliding window to store recent valuable information about operators, such as factors, probabilities, and efficiency. Four common used DE operators are chosen with the AOS, and two kinds of assist information on operator are selected to improve the operators search power. The operator information is updated with the help of LSIAS and the resulting algorithmic combination is called MOEA/D-LSIAS. Compared to some well-known MOEA/D variants, the LSIAS demonstrates the superior robustness and fast convergence for various multiobjective optimization problems. The comparative experiments also demonstrate improved search power of operators with different assist information on different problems.

Adaptive operator selection (AOS) is used to dynamically select the appropriate genic operator for offspring reproduction, which aims to improve the performance of evolutionary algorithms (EAs) by producing high-quality offspring during the evolutionary process. This paper proposes a novel classification tree based adaptive operator selection strategy for multi-objective evolutionary algorithm based on decomposition (MOEA/D-CTAOS). In our proposal, the classification tree is trained by the recorded data set which contains the information on the historical offspring. Before the reproduction at each generation, the classifier is used to predict each possible result obtained by different operators, and only one operator with the best result is selected to generate offspring next. Meanwhile, a novel differential evolution based on search inertia (SiDE) is designed to steer the evolutionary process in a more efficient way. The experimental results demonstrate that proposed MOEA/D-CTAOS outperforms other MOEA/D variants on UF and LZ benchmarks in terms of IGD and HV value. Further investigation also confirms the advantage of direction-guided search strategy in SiDE.

Search operators play an important role in meta-heuristics. There are typically a variety of search operators available for solving a problem, and the selection and order of using the operators can greatly affect the algorithm performance. Adaptive operator selection (AOS) has been proposed to select operators during optimisation dynamically and adaptively. However, most existing studies focus on real-value optimisation problems, while combinatorial optimisation problems, especially complex routing problems, are seldom considered. Motivated by the effectiveness of AOS on real-value optimisation problems and the urgent need of efficiency in solving real routing problems, this paper investigates AOS in complex routing problems obtained from real-world scenarios, the multi-depot multi-disposal-facility multi-trip capacitated vehicle routing problems (M3CVRPs). Specifically, the stateless AOS, arguable the most classic, intuitive and commonly used category of AOS approaches, is integrated into the region-focused local search (RFLS), the state-of-the-art algorithm for solving M3CVRPs. Unexpectedly and yet within understanding, experimental results show that the original RFLS performs better than the RFLS embedded with stateless AOS approaches. To determine the causes, a novel neighbourhood analysis is conducted to investigate the characteristics of M3CVRP and the factors that affect the performance of the AOS. Experimental results indicate that the momentum assumption of stateless AOS, good operators in history will also work well in current stage, is not satisfied within most of the time during the optimisation of the complex problem, leading to the unstable performance of operators and the failure of stateless AOS.KeywordsAdaptive operator selectionVehicle routing problemNeighbourhood analysisLocal searchMeta-heuristics

This paper presents adaptive routing selection strategies suitable for network-on-chip (NoC). The main prototype presented in this paper uses contention information and bandwidth space occupancy to make routing decision at runtime during application execution time. The performance of the NoC router is compared to other NoC routers with queue-length-oriented adaptive routing selection strategies. The evaluation results show that the contention- and bandwidth-aware adaptive routing selection strategies are better than the queue-length-oriented adaptive selection strategies. Messages in the NoC are switched with a wormhole cut-through switching method, where different messages can be interleaved at flit-level in the same communication link without using virtual channels. Hence, the head-of-line blocking problem can be solved effectively and efficiently. The routing control concept and the VLSI microarchitecture of the NoC routers are also presented in this paper.

Adaptive operator selection with reinforcement learning

As well as new algorithms are constantly proposed, new test functions for these algorithms are also designed. In this paper we explore 15 new benchmark functions proposed for CEC-2018 Multiobjective Evolutionary Algorithms (MOEA) Competition for many-objective optimization. The functions have diverse properties which cover a good representation of various real-world scenarios. We propose many-objective approaches that were designed considering three schemes to perform adaptive operator selection with NSGA-III algorithm: Thompson Sampling, Probability Matching and Adaptive Pursuit. They select from a pool of candidates composed by DE mutations and a Genetic Algorithm crossover. Thompson Sampling is a multi-armed bandit approach, i.e., it was designed to deal with the exploration versus exploitation dilemma intrinsic to the adaptive operator selection problem. Its use in a many objective evolutionary algorithm is innovative and constitutes the main contribution of this work. As the CEC-2018 is composed by complex, potentially nonlinear functions, we also perform the analysis of the effects of the insertion of a nonlinear operator within the candidate pool of operators. Statistical analysis of the experiments were performed with Mann-Whitney and Friedman tests. The IGD indicator was used to infer the quality of the solutions. The results indicate the use of Thompson Sampling as an adaptive operator selection is promising and increases the optimization performance of NSGA-III. They also indicate that the use of the nonlinear operator is capable of improving the results of all adaptive versions.

The measurement precision of the optical encoder is influenced by coupling interferences from the system and the environment. To further improve the measurement precision of the optical encoder, in this article, a nonlinear error compensation method is proposed to extract the optimal underlying trend of the measurement error, which is based on the empirical mode decomposition (EMD) with a local-sinusoidal-assisted (LSA) scheme and an adaptive intrinsic mode function (IMF) selection scheme. To solve the inherent problem of mode-mixing phenomena existing in the decomposed IMFs from the measurement error, an LSA scheme is proposed to adaptively change the extreme distribution of the measurement error, which is based on an abnormal segment discrimination rule. An adaptive IMF selection scheme is proposed to construct the optimal underlying trend of the measurement error, which is first formulated as an optimization problem. Next, a significant factor is defined to select the eligible IMFs with more contributions to the underlying trend. Comparison experiments indicate that the eligible IMFs can be adaptively selected, and the proposed compensation method achieves an excellent compensation performance with the root-mean-square error of 0.380 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\mu$</tex-math></inline-formula> m in the 95% confidence interval of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$-$</tex-math></inline-formula> 0.017–0.016 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\mu$</tex-math></inline-formula> m, which is superior to the previous EMD-based compensation methods.

Adaptive modulation selection plays an important role in wireless communication and wireless sensor network since the wireless channel condition is time-varying due to some factors e.g. path-loss and multipath propagation. Therefore, using one modulation type cannot fulfill all the channel conditions and receivers. Conventional adaptive modulation selection schemes are based on packet loss statistics or SNR measurement. However, all of these methods are indirect approaches because they utilize the packet loss rate or SNR information to estimate the BER approximately and the modulation selection that based on these approximate results is not exactly. Therefore, in this paper, we propose to utilize the error estimating code (EEC) which directly represents the BER information to realize a new adaptive modulation selection scheme. Compared to traditional adaptive modulation type selection schemes, our scheme needs only a small amount of overhead and very little computational cost for the same performance.

In this paper, an adaptive relay selection and transmission scheme are proposed for cooperative communication. As the conventional relay selection schemes, the best harmonic mean and the threshold-based relay selection scheme require channel state information (CSI) in advance at the source. By using CSI, they can select the optimal or the sub-optimal relay. But, the source does not know CSI as a priori information in general environment. Additionally, the conventional system uses the fixed maximum throughput regardless of channel state. The proposed relay selection scheme does not require a priori information at the source. It uses a characteristic of Reed---Solomon codes. By using it, the source can select the optimal relay for cooperative communication. Additionally, the proposed adaptive transmission scheme can change the maximum throughput according to the result of optimal relay selection. By using the proposed relay selection and adaptive transmission scheme, the proposed system is more reliable and adaptable according to selected relay.

Real-world optimisation problems tends to be complex and have many conflicting objectives. Many of these problems have been solved satisfactorily by metaheuristics, but choosing the appropriate method to each problem is still an open problem, as it depends on the characteristics of the problem. Fitness Landscape Analysis (FLA) is a mechanism that can be used to identify the major characteristics of a problem and guide the choice of the algorithm. Recently, the use of Adaptive Operator Selection (AOS) schemes to permit the adaptation of an algorithm to the online characteristics of a problem have been successfully used to solve complex problems. Most AOS mechanisms are only guided by a reward based on the fitness values, which does not bring information about the problem characteristics. This work intends to incorporate FLA information about the problems into an AOS mechanism to improve its performance. The proposed approach is based on the MOEA/D-DRA framework and combines two FLA metrics (Fitness Cloud Index and Dispersion Metric) with the Probability Matching operator selection scheme. Five variants of the proposed approach are investigated in a multi-objective benchmark (CEC 2009) and a many-objective benchmark (CEC 2018). The results indicate that the addition of FLA information improves the behavior of the AOS mechanism, that multimodality information tends to be more important than searchability information, that using both metrics is better than using only the searchability metric and that using only the multimodality information is usually the best choice.

Considerable attention has been paid to selective hyper-heuristic frameworks for addressing computationally hard scheduling problems. By using selective hyper-heuristics, we can derive benefits from the strength of low level heuristics and their components at different stages of the heuristic search. In this paper, a simple, general and effective selective hyper heuristic is presented. We introduce an iterated local search based hyper-heuristic framework that incorporates the adaptive operator selection scheme to learn through the search process. The considered iterative approach employs an action selection model to decide the perturbation strategy to apply in each step and a credit assignment module to score its performance. The designed framework allows us to employ any action selection model and credit assignment mechanism used in the literature. Empirical results and an analysis of six different action selection models against state-of-the-art approaches, across 39 problem instances, highlight the significant potential of the proposed selection hyper-heuristics. Further analysis on the adaptive behavior of the model suggests that two of the six models are able to learn the best performing perturbation strategy, resulting in significant performance gains.

Evolvability metrics gauge the potential for fitness of an individual rather than fitness itself. They measure the local characteristics of the fitness landscape surrounding a solution. In adaptive operator selection the goal is to dynamically select from a given pool the operator to apply next during the search process. An important component of these adaptive schemes is credit assignment, whereby operators are rewarded according to their observed performance. This article brings the notion of evolvability to adaptive operator selection, by proposing an autonomous search algorithm that rewards operators according to their potential for fitness rather than their immediate fitness improvement. The approach is tested within an evolutionary algorithm framework featuring several mutation operators on binary strings. Three benchmark problems of increasing difficulty, Onemax, Royal Staircase and Multiple Knapsack are considered. Experiments reveal that evolvability metrics significantly improve the performance of adaptive operator selection, when compared against standard fitness improvement metrics.The main contribution is to effectively use fitness landscape metrics to guide a self-configuring algorithm.

Adaptive operator selection with dueling deep Q-network for evolutionary multi-objective optimization

A performance comparison of mobility anchor point selection schemes in Hierarchical Mobile IPv6 networks