Neurodynamic Optimization: towards Nonconvexity

Abstract

Optimization is a ubiquitous phenomenon in nature and an important tool in engineering. As the counterparts of biological neural systems, properly designed artificial neural networks can serve as goal-seeking computational models for solving various optimization problems in many applications. In many engineering applications such as optimal control and signal processing, obtaining real-time locally optimal solutions is more important than taking time to search for globally optimal solutions. In such applications, recurrent neural networks are usually more competent than numerical optimization methods because of the inherent parallel nature. Since the seminal work of Tank and Hopfield in 1980s (Hopfield & Tank, 1986; Tank & Hopfield, 1986), recurrent neural networks for solving optimization problems have attracted much attention. In the past twenty years, many models have been developed for solving convex optimization problems, from the earlier proposals including the penalty method based neural network (Kennedy & Chua, 1988), the switched-capacitor neural network (Rodriģuez-Vazquez et al., 1990) and the deterministic annealing neural network (Wang, 1994), to the latest development including (Xia, 2004; Gao, 2004; Gao et al., 2005; Hu & Wang, 2007b; Hu & Wang, 2007c; Hu & Wang, 2008). These latest models have a common characteristic: they were all formulated based on optimality conditions of the problems and therefore their equilibria correspond exactly to the solutions of the problems. In addition, for ensuring this correspondence, in contrast to many earlier proposals such as the penaltybased neural network (Kennedy & Chua, 1988), there is no need to let any parameter go infinity. More importantly, if these neural networks are applied to solve nonconvex optimization problems, this nice property will be retained in the sense of critical points instead of global optima, e.g., Karush-Kuhn-Tucker (KKT) points (i.e., the equilibria will correspond no longer to the global optima but to these critical points). Naturally, one will ask if these models are suitable for searching for critical points, especially local optima, of general nonconvex optimization problems. Unfortunately, there is no guarantee that these optimality-conditions-based neural networks can be directly adopted to solve nonconvex optimization problems. In designing recurrent neural networks for optimization, letting the equilibria correspond to solutions is just one O pe n A cc es s D at ab as e w w w .ite ch on lin e. co m