A Multiresolution Analysis-Assisted Reinforcement Learning Approach to Run-by-Run Control

Abstract

In recent years, the run-by run (RbR) control mechanism has emerged as a useful tool for keeping complex semiconductor manufacturing processes on target during repeated short production runs. Many types of RbR controllers exist in the literature of which the exponentially weighted moving average (EWMA) controller is widely used in the industry. However, EWMA controllers are known to have several limitations. For example, in the presence of multiscale disturbances and lack of accurate process models, the performance of EWMA controller deteriorates and often fails to control the process. Also, the control of complex manufacturing processes requires sensing of multiple parameters that may be spatially distributed. New control strategies that can successfully use spatially distributed sensor data are required. This paper presents a new multiresolution analysis (wavelet) assisted reinforcement learning (RL)-based control strategy that can effectively deal with both multiscale disturbances in processes and the lack of process models. The novel idea of a wavelet-aided RL-based controller represents a paradigm shift in the control of large-scale stochastic dynamic systems of which the control problem is a subset. Henceforth, we refer our new control strategy as a WRL-RbR controller. The WRL-RbR controller is tested on a multiple-input-multiple-output chemical mechanical planarization process of wafer fabrication for which the process model is available. Results show that the RL controller outperforms EWMA-based controllers for low autocorrelation. The new controller also performs quite well for strongly autocorrelated processes for which the EWMA controllers are known to fail. Convergence analysis of the new breed of the WRL-RbR controller is presented. Further enhancement of the controller to deal with model-free processes and for inputs coming from spatially distributed environments are also discussed. Note to Practitioners-This work was motivated by the need to develop an intelligent and efficient RbR process controller, especially for the control of processes with short production runs as in the case of the semiconductor manufacturing industry. A novel controller that is presented here is capable of generating optimal control actions in the presence of multiple time-frequency disturbances, and allows the use of realistic (often complex) process models without sacrificing robustness and speed of execution. Performance measures, such as reduction of variability in process output and control recipe, minimization of initial bias, and ability to control processes with high autocorrelations are shown to be superior in comparison to the commercially available exponentially weighted moving average controllers. The WRL-RbR controller is very generic, and can be easily extended to processes with drifts and sudden shifts in the mean and variance. The viability of extending the controller to distributed input parameter sensing environments, including those for which process models are not available, is also discussed