Collaborative Q-learning hyper-heuristic evolutionary algorithm for the production and transportation integrated scheduling of silicon electrodes

Abstract

Silicon electrodes are widely used in semiconductor etching machines. The periodic consumption of silicon electrodes has become an important consumable in wafer manufacturing. Due to the limited number of silicon electrode manufacturers and the increasing demand for silicon electrodes in global market, the newly processed silicon electrodes need to be immediately delivered to wafer manufacturers to ensure uninterrupted production. Driven by this, this paper introduces a new integrated scheduling problem called the silicon electrode production and transportation integrated scheduling problem (SEPTISP), whose criterion is to minimize the makespan. This problem is modeled as a combination of the production scheduling subproblem (PSS) and the transportation scheduling subproblem (TSS). These two subproblems are coupled with each other. Considering the SEPTISP's NP-hard and coupling properties, a novel collaborative Q-learning hyper-heuristic evolutionary algorithm integrated with reinforcement learning scheme (CQHEA_RLS) is proposed to handle them. In the CQHEA_RLS, the production agent and the transportation agent are focused on executing exploitation in the reduced but promising regions via their specific search actions, while the joint agent is centered on performing exploration in the whole solution space through its joint search actions and determining the suitable search regions for the other two agents. Meanwhile, the joint agent also integrates the historical performance information of its own and the other two agents' search actions to dynamically choose its subsequent search actions. Moreover, each agent is devised as an effective Q-learning hyper-heuristic evolutionary algorithm. This new collaborative framework ensures that the CQHEA_RLS can continuously search downwards to obtain truly high-quality solutions in a relatively short runtime. The experimental results demonstrate that the designed algorithm can achieve better performance than the existing state-of-the-art algorithms.