Hyper-heuristic Framework Research Articles

A cooperative learning-aware dynamic hierarchical hyper-heuristic for distributed heterogeneous mixed no-wait flow-shop scheduling

The distributed heterogeneous mixed scheduling mode in the manufacturing systems emphasizes the cooperation between factories for the entire production cycle, which poses enormous challenges to the processing and assignment of jobs. Discrepancies in the processing environment and types of machines of each factory during various production stages cause diverse processing paths and scheduling. The distributed heterogeneous mixed no-wait flow-shop scheduling problem with sequence-dependent setup time (DHMNWFSP-SDST), abstracted from the industrial scenarios, is addressed in this paper. The mathematical model of DHMNWFSP-SDST is established. A cooperative learning-aware dynamic hierarchical hyper-heuristic (CLDHH) is proposed to solve the DHMNWFSP-SDST. In CLDHH, a cooperative initialization method is developed to promote diversity and quality of solutions. A hierarchical hyper-heuristic framework with reinforcement learning (RL) is designed to select the algorithm component automatically. Estimation of Distribution Algorithm (EDA) guides the upper-layer RL to select four neighborhood structures. A dynamic adaptive neighborhood switching constructs the lower-layer RL to accelerate exploitation with the dominant sub-neighborhoods. An elite-guided hybrid path relinking achieves local enhancement. The experimental results of CLDHH and six state-of-the-art algorithms on instances indicate that the proposed CLDHH is superior to the state-of-the-art algorithms in solution quality, robustness, and efficiency.

Energy-conserving cold chain with ambient temperature, path flexibility, and hybrid fleet: formulation and heuristic approach

Cold chain logistics networks represent intricate systems that require harmonisation of their influence on the economy, environment, and society. However, simultaneously achieving these goals is hard. This paper defined a comprehensive model aiming to achieve a tradeoff of these goals related to cost efficiency, product quality, delivery timeliness, and environmental impacts. Meanwhile, the influences of ambient temperature, path flexibility, and hybrid fleet on the proposed dual-mode location-routing problem-based cold chain logistics (DMLRPCCL) are analyzed. A meticulously crafted hyper-heuristic framework employing Q-learning has been developed to address the complexity of cold chains to obtain high-quality solutions. The numerical study has shown that the proposed model can analyze various scenarios for perishable products and evaluate their impact on cost, emissions, and quality. The proposed algorithm is efficient and effective in achieving competitive results compared to three tailored algorithms. Extensive analyses are performed to empirically assess the effect of path flexibility, hybrid fleet, and ambient temperature on the DMLRPCCL planning. Several managerial insights are presented.

A study of jamming resource allocation based on a hyperheuristic framework

There are massive data and rapidly changing battlefield situations in modern electronic warfare, which is a challenge to jamming resource allocation. It is difficult for the existing optimization algorithms to balance optimization capability and calculation speed at the same time. Above all, this study designs a fuzzy multiattribute evaluation model based on behavioral characteristics. The model comprehensively integrates the characteristics of signal behavior, motion behavior, and external intelligence and can effectively deal with unknown threat signals. Then, this study proposes an improved genetic selection electronic warfare operator (EWO) hyperheuristic (GAEWHH) algorithm. As an emergent optimization algorithm, the HH framework has not previously been applied to the problem of jamming resource allocation. This is a two-level algorithm framework that can isolate problem domains. The high level uses an improved genetic algorithm to search the heuristic space, and four EWOs based on the problem domain are designed for the low level to search the solution space. Combining different EWOs can change the population diversity, evolution direction, and algorithm complexity of the GAEWHH algorithm, which improves the algorithm performance to meet battlefield situation requirements. The experiment shows that for large-scale problems, the GAEWHH algorithm is better than the mainstream evolutionary algorithm in terms of optimization capability and better than Google OR-Tools in terms of calculation speed. In this way, the GAEWHH algorithm achieves a balance between optimization capability and calculation speed.

A reinforcement learning-based hyper-heuristic for AGV task assignment and route planning in parts-to-picker warehouses

Globally, e-commerce warehouses have begun implementing robotic mobile fulfillment systems (RMFS), which can improve order-picking efficiency by using automated guided vehicles (AGVs) to realize operations from parts to pickers. AGVs depart from their initial points, move to a target rack position, and subsequently transport racks to picking stations. The AGVs return the racks to their original positions after the workers pick them up. When all tasks are completed, the AGVs return to their starting point. In this context, the main challenge is the task assignment and route planning of multiple AGVs to minimize travel times. We formulate a mixed-integer linear programming (MILP) model with valid inequalities to solve small problem instances optimally. We introduce a reinforcement learning (RL)-based hyper-heuristic (HH) framework to solve large instances to near-optimality. A typical HH framework comprises two levels: high-level heuristics (HLH) and low-level heuristics (LLH). The framework starts from an initial solution and improves iteratively through LLHs, while the HLH invokes a selection strategy and an acceptance criterion to generate a new solution. We propose a novel selection strategy based on the improved Multi-Armed Bandits algorithm called Co-SLMAB and Exponential Monte Carlo with counters (EMCQ) as the acceptance criterion. The corresponding collision avoidance rules are then formulated for different conflicts to construct a conflict-free traveling route for AGVs. Besides testing the proposed framework’s effectiveness in real-life warehouse layouts, we perform extensive computational experiments and a thorough sensitivity analysis. The results show that (i) the proposed valid inequalities aid in obtaining better lower bounds and significantly speed up the solution process; (ii) the Co-SLMAB-HH framework is quite competitive compared to CPLEX, outperforming the other tested hyper-heuristics and the problem-specific heuristic regarding convergence and computation time; and (iii) a pool of LLHs consisting of a wide range of different operators is advantageous over a limited set of simple operators while solving problems using hyper-heuristics.

Dynamic metaheuristic selection via Thompson Sampling for online optimization

It is acknowledged that no single heuristic can outperform all the others in every optimization problem. This has given rise to hyper-heuristic methods for providing solutions to a wider range of problems. In this work, a set of five non-competing low-level heuristics is proposed in a hyper-heuristic framework. The multi-armed bandit problem analogy is efficiently leveraged and Thompson Sampling is used to actively select the best heuristic for online optimization. The proposed method is compared against ten population-based metaheuristic algorithms on the well-known CEC’05 optimizing benchmark consisting of 23 functions of various landscapes. The results show that the proposed algorithm is the only one able to find the global minimum of all functions with remarkable consistency.

Q-learning-based hyper-heuristic framework for estimating the energy consumption of electric buses for public transport

Q-learning-based hyper-heuristic framework for estimating the energy consumption of electric buses for public transport

Nested Markov chain hyper-heuristic (NMHH): a hybrid hyper-heuristic framework for single-objective continuous problems.

This article introduces a new hybrid hyper-heuristic framework that deals with single-objective continuous optimization problems. This approach employs a nested Markov chain on the base level in the search for the best-performing operators and their sequences and simulated annealing on the hyperlevel, which evolves the chain and the operator parameters. The novelty of the approach consists of the upper level of the Markov chain expressing the hybridization of global and local search operators and the lower level automatically selecting the best-performing operator sequences for the problem. Numerical experiments conducted on well-known benchmark functions and the comparison with another hyper-heuristic framework and six state-of-the-art metaheuristics show the effectiveness of the proposed approach.

Multi-period portfolio optimization using a deep reinforcement learning hyper-heuristic approach

Portfolio optimization concerns with periodically allocating the limited funds to invest in a variety of potential assets in order to satisfy investors’ appetites for risk and return goals. Recently, Deep Reinforcement Learning (DRL) has shown its promising capabilities in sequential decision making problems. However, traditional DRL algorithms directly operate in the space of low-level actions, which exhibits poor scalability and becomes intractable in real-world problem instances when the dimensionality of the environment increases. To deal with this, in this work, a novel DRL hyper-heuristic framework is proposed for multi-period portfolio optimization problem. Instead of exploiting the entire action domain, our proposed approach is more effective by searching for low-level well-developed trading strategies. In addition, our proposed approach is data-driven and respects the nature of the problem by taking advantage of expert domain knowledge and posing it multidimensional states to further leverage additional diverse information from alternative views of the environment. The proposed approach is evaluated on five real-world capital market problem instances and numerous experimental results demonstrate our proposed method can achieve notable performance gains compared to state-of-art trading strategies as well as traditional DRL baseline method. The data we used are from five stock indices, covering the period from the 2012 to 2022. Our study can have salient policy implications for investment strategy formulation and effective regulatory frameworks establishment.

A deep reinforcement learning hyper-heuristic with feature fusion for online packing problems

In recent years, deep reinforcement learning has shown great potential in solving computer games with sequential decision-making scenarios. Hyper-heuristic is a generic search framework, capable of intelligently selecting or generating algorithms to solve a class of optimisation problems with stochastic or dynamic settings. This paper proposes a new general framework for solving online packing problems using deep reinforcement learning hyper-heuristics. Although analytical approaches can address most offline packing problems successfully, their online versions have proved much more challenging and the performance of the existing methods is often not satisfactory. In this paper, we extend a recent deep reinforcement learning hyper-heuristic framework by fusing the visual information of real-time packing with distributional information of random parameters of the problem. Computational experiments show that our method outperforms the state of the art online methods with reductions in optimality gap between 2%–19% for knapsack problem and 0.7% for the online strip packing problem. In addition, a new visual analysis presentation is also devised to better interpret the learned packing strategies, which can reveal more information than the widely used landscape analysis. As online packing problems are widely available in production environments, the proposed approach can serve as an important reference to solve other similar combinatorial optimisation problems for which visual layout inputs would aid learning.

Big Data Cyber Security Using Machine Learning

It is well-known that cyber protection in the context of big data is a crucial issue and also presents a significant challenge to the research study field. The use of artificial intelligence algorithms has been suggested as a potential solution for complex information security issues. Support vector machines (SVMs), one of these techniques, have shown outstanding performance on several category problems. However, in order to construct an effective SVM, the user must first determine the correct SVM configuration, which is a difficult task that necessitates expert knowledge as well as a substantial amount of hands-on initiative for trial and error. In this study, precision and design intricacy are treated as two conflicting goals in our formulation of the SVM arrangement technique as a bi-objective optimisation problem. We propose a brand-new, issue-domain-independent hyper-heuristic framework for bi-objective optimisation. This is the first time a hyper-heuristic for this issue has been developed. The suggested hyper-heuristic framework consists of both high-level and low-level heuristics. The high-level strategy makes decisions on which low-level heuristics to employ in order to create a fresh SVM setup based on the search performance. Each of the low-level heuristics successfully locates the SVM layout search region using a distinct set of rules. The suggested structure adaptively integrates Pareto-based and disintegration-based strategy toughness to approximate the Pareto collection of SVM configurations in order to handle bi-objective optimisation. The effectiveness of the suggested structure has been evaluated in relation to two cyber security concerns: the detection of anomalous intrusions and Microsoft malware's large information category. The obtained results show that the suggested framework is quite effective, if not superior, when compared to its competitors and other formulas.

LS-HH: A Learning-Based Selection Hyper-Heuristic for Distributed Heterogeneous Hybrid Blocking Flow-Shop Scheduling

As the development of economic globalization, the distributed manufacturing has become common in modern industries. The scheduling of production resources in multiple production centers becomes an emerging topic. This paper is the first attempt to address a distributed heterogeneous hybrid blocking flow-shop scheduling problem (DHHFSP-B) with the minimization of makespan. Compared with the traditional single flow-shop scheduling, DHHFSP-B considers the collaborative production of multiple hybrid flow lines with heterogeneous layout and processing performance as well as no intermediate buffers. We firstly present a mixed-integer linear programming model to formulate DHHFSP-B and then propose a learning-based selection hyper-heuristic framework (LS-HH) for solving it. The LS-HH contains high-level strategy and low-level heuristics. In the high-level strategy, a learning probability model is built to provide the guidance to choose the suitable perturbation heuristic during the optimization process. A simulated annealing-like move acceptance is employed to determine the updating of incumbent domain solution and prevent the search from trapping into local optimum. In the low-level heuristics, a constructive heuristic is proposed based on a novel assignment rule to create the initial domain solution. Four problem-specific perturbation heuristics and a variable neighborhood search-based improvement operator are employed to search the solution space. A comprehensive computational experiment is conducted. The comparative results show that the LS-HH significantly outperforms the Gurobi solver and several closely relevant optimization methods in solving the DHHFSP-B.

A general deep reinforcement learning hyperheuristic framework for solving combinatorial optimization problems

Many problem-specific heuristic frameworks have been developed to solve combinatorial optimization problems, but these frameworks do not generalize well to other problem domains. Metaheuristic frameworks aim to be more generalizable compared to traditional heuristics, however their performances suffer from poor selection of low-level heuristics (operators) during the search process. An example of heuristic selection in a metaheuristic framework is the adaptive layer of the popular framework of Adaptive Large Neighborhood Search (ALNS). Here, we propose a selection hyperheuristic framework that uses Deep Reinforcement Learning (Deep RL) as an alternative to the adaptive layer of ALNS. Unlike the adaptive layer which only considers heuristics’ past performance for future selection, a Deep RL agent is able to take into account additional information from the search process, e.g., the difference in objective value between iterations, to make better decisions. This is due to the representation power of Deep Learning methods and the decision making capability of the Deep RL agent which can learn to adapt to different problems and instance characteristics. In this paper, by integrating the Deep RL agent into the ALNS framework, we introduce Deep Reinforcement Learning Hyperheuristic (DRLH), a general framework for solving a wide variety of combinatorial optimization problems and show that our framework is better at selecting low-level heuristics at each step of the search process compared to ALNS and a Uniform Random Selection (URS). Our experiments also show that while ALNS can not properly handle a large pool of heuristics, DRLH is not negatively affected by increasing the number of heuristics.

Hyper-heuristics with Path Relinking applied to the Generalised Time-Dependent ATSP in air travel

In this work we propose the use of path relinking within a hyper-heuristic framework to solve the Generalised Time-Dependent Asymmetric Traveling Salesman problem applied to Air Travel. We implemented several heuristic selection methods, such as Simple Random, Random Descent, Random Permutation and Reinforcement Learning. We were able to achieve very good solutions for 13 of the 14 instances from a well-known benchmark set, evidencing that the adequate use of a hyper-heuristic framework with path relinking can be very efficient in improving the final results.

Wireless edge device intelligent task offloading in mobile edge computing using hyper-heuristics

To overcome with the computation limitation of resource-constrained wireless IoT edge devices, providing an efficient task computation offloading and resource allocation in distributed mobile edge computing environment is consider as a challenging and promising solution. Hyper-heuristic in recent times is gaining popularity due to its general applicability of same solution to solve different types of problems. Hyper-heuristic is generally a heuristic method or framework which iteratively evaluates and chooses the best low-level heuristic, to solve different types of problems. In this paper, we try to solve wireless device task offloading in mobile edge computing, which is a non-convex and NP-Hard problem by using a proposed novel Hyper-Heuristic Framework using Stochastic Heuristic Selection (HHFSHS) using Contextual Multi-Armed Bandit (CMAB) with Epsilon-Decreasing strategy, considering two key Quality of Service (QoS) objectives computation time and energy consumption. These multiobjective criteria are modeled as single-objective optimization problem with the goal to minimize latency and energy consumption of wireless devices without losing the pareto optimality. Finally, evaluate its performance by comparing with other individual meta-heuristic algorithms.

Self-Adapting Particle Swarm Optimization for continuous black box optimization

This paper introduces a new version of a hyper-heuristic framework: Generalized Self-Adapting Particle Swarm Optimization with samples archive (M-GAPSO). This framework is based on the authors previous works on hybridization of optimization algorithms and enhancing population based optimization with model based optimization. The paper presents the structure of the proposed framework and analyzes the impact of its modules on the final system performance. M-GAPSO hybridizes Particle Swarm Optimization, Differential Evolution and model based optimizers. A ratio of particular algorithms within a population is regulated by an adaptation scheme. The applicability of the proposed hybrid method to black-box optimization is verified on 24 continuous benchmark functions from the COCO test set and 29 functions from the CEC-2017 test set. On the BBOB test set a hybrid of PSO and DE with adaptation obtained 11 significantly better and 2 significantly worse results on 5 and 20 dimensional functions than the basic DE. Further inclusion of the model based optimizers led to 15 significantly better and 2 significantly worse results compared to the PSO-DE hybrid. On the CEC-2017 test set, M-GAPSO was significantly better than both Red Fox Optimization and Dual Opposition-Based Learning for Differential Evolution (DOBL) on 7 functions in 30 dimensions and 12 functions in 50 dimensions.

A hyper-heuristic based reinforcement-learning algorithm to train feedforward neural networks

Artificial Neural Networks (ANNs) offer unique opportunities in numerous research fields. Due to their remarkable generalization capabilities, they have grabbed attention in solving challenging problems such as classification, function approximation, pattern recognition and image processing that can be quite complex to model mathematically in practice. One of the most vital issues regarding the ANNs is the training process. The aim at this stage is to find the optimum values of ANN parameters such as weights and biases, which indeed embed the whole information of the network. Traditional gradient-descent-based training methods include various algorithms, of which the backpropagation is one of the best-known. Such methods have been shown to exhibit outstanding results, however, they are known have two major theoretical and computational limitations, which are slow convergence speed and possible local minima issues. For this purpose, numerous stochastic search algorithms and heuristic methods have been individually used to train ANNs. However, methods, bringing diverse features of different optimizers together are still lacking in the related literature. In this regard, this paper aims to develop a training algorithm operating based on a hyper-heuristic (HH) framework, which indeed resembles reinforcement learning-based machine learning algorithm. The proposed method is used to train Feed-forward Neural Networks, which are specific forms of ANNs. The proposed HH employs individual metaheuristic algorithms such as Particle Swarm Optimization (PSO), Differential Evolution (DE) Algorithm and Flower Pollination Algorithm (FPA) as low-level heuristics. Based on a feedback mechanism, the proposed HH learns throughout epochs and encourages or discourages the related metaheuristic. Thus, due its stochastic nature, HH attempts to avoid local minima, while utilizing promising regions in search space more conveniently by increasing the probability of invoking relatively more promising heuristics during training. The proposed method is tested in both function approximation and classification problems, which have been adopted from UCI machine learning repository and existing literature. According to the comprehensive experimental study and statistically verified results, which point out significant improvements, the developed HH based training algorithm can achieve significantly superior results to some of the compared optimizers.

Hyperheuristic Approach Based on Reinforcement Learning for Air Traffic Complexity Mitigation

Airspace capacity has become a critical resource for air transportation. Complexity in traffic patterns is a structural problem, whereby airspace capacity is sometimes saturated before the number of aircraft has reached the capacity threshold. This paper addresses a strategic planning problem with an efficient optimization approach that minimizes traffic complexity based on linear dynamical systems in order to improve the traffic structure. Traffic structuring techniques comprise departure time adjustment, en route trajectory deviation, and flight-level allocation. The resolution approach relies on the hyperheuristic framework based on reinforcement learning to improve the searching strategy during the optimization process. The proposed methodology is implemented and tested with a full day of traffic in the French airspace. Numerical results show that the proposed approach can reduce air traffic complexity by 92.8%. The performance of the proposed algorithm is then compared with two different algorithms, including the random search and the standard simulated annealing. The proposed algorithm provides better results in terms of air traffic complexity and the number of modified trajectories. Further analysis of the proposed model was conducted by considering time uncertainties. This approach can be an innovative solution for capacity management in the future air traffic management system.

Evolution of group-theoretic cryptology attacks using hyper-heuristics

Abstract In previous work, we developed a single evolutionary algorithm (EA) to solve random instances of the Anshel–Anshel–Goldfeld (AAG) key exchange protocol over polycyclic groups. The EA consisted of six simple heuristics which manipulated strings. The present work extends this by exploring the use of hyper-heuristics in group-theoretic cryptology for the first time. Hyper-heuristics are a way to generate new algorithms from existing algorithm components (in this case, simple heuristics), with EAs being one example of the type of algorithm which can be generated by our hyper-heuristic framework. We take as a starting point the above EA and allow hyper-heuristics to build on it by making small tweaks to it. This adaptation is through a process of taking the EA and injecting chains of heuristics built from the simple heuristics. We demonstrate we can create novel heuristic chains, which when placed in the EA create algorithms that out perform the existing EA. The new algorithms solve a greater number of random AAG instances than the EA. This suggests the approach may be applied to many of the same kinds of problems, providing a framework for the solution of cryptology problems over groups. The contribution of this article is thus a framework to automatically build algorithms to attack cryptology problems given an applicable group.

A deep reinforcement learning based hyper-heuristic for combinatorial optimisation with uncertainties

In the past decade, considerable advances have been made in the field of computational intelligence and operations research. However, the majority of these optimisation approaches have been developed for deterministically formulated problems, the parameters of which are often assumed perfectly predictable prior to problem-solving. In practice, this strong assumption unfortunately contradicts the reality of many real-world problems which are subject to different levels of uncertainties. The solutions derived from these deterministic approaches can rapidly deteriorate during execution due to the over-optimisation without explicit consideration of the uncertainties. To address this research gap, a deep reinforcement learning based hyper-heuristic framework is proposed in this paper. The proposed approach enhances the existing hyper-heuristics with a powerful data-driven heuristic selection module in the form of deep reinforcement learning on parameter-controlled low-level heuristics, to substantially improve their handling of uncertainties while optimising across various problems. The performance and practicality of the proposed hyper-heuristic approach have been assessed on two combinatorial optimisation problems: a real-world container terminal truck routing problem with uncertain service times and the well-known online 2D strip packing problem. The experimental results demonstrate its superior performance compared to existing solution methods for these problems. Finally, the increased interpretability of the proposed deep reinforcement learning hyper-heuristic has been exhibited in comparison with the conventional deep reinforcement learning methods.

Solving examination timetabling problem within a hyper-heuristic framework

Scheduling exams in colleges are a complicated job that is difficult to solve conventionally. Exam timetabling is one of the combinatorial optimization problems where there is no exact algorithm that can answer the problem with the optimum solution and minimum time possible. This study investigated the University of Toronto benchmark dataset, which provides 13 real instances regarding the scheduling of course exams from various institutions. The hard constraints for not violate the number of time slots must be fulfilled while paying attention to fitness and running time. Algorithm of largest degree, hill climbing, and tabu search within a hyper-heuristic framework is investigated with regards to each performance. This study shows that the Tabu search algorithm produces much lower penalty value for all datasets by reducing 18-58% from the initial solution.