Bad Local Optima Research Articles

Big Learning Expectation Maximization

Mixture models serve as one fundamental tool with versatile applications. However, their training techniques, like the popular Expectation Maximization (EM) algorithm, are notoriously sensitive to parameter initialization and often suffer from bad local optima that could be arbitrarily worse than the optimal. To address the long-lasting bad-local-optima challenge, we draw inspiration from the recent ground-breaking foundation models and propose to leverage their underlying big learning principle to upgrade the EM. Specifically, we present the Big Learning EM (BigLearn-EM), an EM upgrade that simultaneously performs joint, marginal, and orthogonally transformed marginal matchings between data and model distributions. Through simulated experiments, we empirically show that the BigLearn-EM is capable of delivering the optimal with high probability; comparisons on benchmark clustering datasets further demonstrate its effectiveness and advantages over existing techniques. The code is available at https://github.com/YulaiCong/Big-Learning-Expectation-Maximization.

Deep Fuzzy Variable C-Means Clustering Incorporated With Curriculum Learning

End-to-end deep clustering method utilizes deep neural networks to jointly learn representation features and clustering assignments. Although many k-means-friendly deep clustering models have been explored, the existing division-based methods tend to directly implement clustering with a specific number of clusters, which suffers from poor performance resulted from indistinguishable clusters, and contributes to bad local optimum. At the same time, the representation learning of fuzzy <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$c$</tex-math></inline-formula> -means clustering in the feature space still needs more research. In this paper, a Deep Fuzzy Curriculum Clustering (DFC <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> ) method with the learning strategy of clustering from easy to complex automatically is proposed to tackle above issues. Firstly, considering the soft flexible allocation of fuzzy <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$c$</tex-math></inline-formula> -means and the preservation of local structure of original data, the fuzzy clustering loss and the autoencoder's reconstruction loss are constructed to learn the embedded features and clustering centers simultaneously. Second, curriculum loss is introduced into the constraint to make clusters successively merge in line with implementing clustering from easy to complex, and realize the bottom-up deep aggregative clustering automatically. In addition, novel curriculum information is proposed as constraint to guide the merging of clusters belonging to the same class. Experimental results on four real-world datasets show the superiority of the proposal.

Distributed Soft Clustering Algorithm for IoT Based on Finite Time Average Consensus

Clustering is a common technique for statistical data analysis and it has been widely used in many fields. This article investigates data clustering over the Internet-of-Things (IoT) network. Facing the IoT network challenges, including data volume, communication latency, and information security, we here propose a distributed soft clustering algorithm for the IoT environments where each IoT node may have data from multiple clusters. Considering that the main task of soft clustering is to compute each cluster center in a weighted averaging fashion, our distributed clustering method resorts to an efficient finite-time average-consensus algorithm. Moreover, to make the distributed clustering algorithm more stable and be able to escape from some bad local optimum, we propose a distributed deterministic initialization method based on data variance partitioning. Experiments show that the proposed distributed soft clustering algorithm can offer the same performance as its centralized counterpart in terms of both convergence and clustering quality. Besides, unlike most clustering methods relying on probabilistic initialization, our algorithm could provide stable clustering quality which makes it more suitable for IoT networks. A real-world case study about the clustering analysis for distributed data sets collected by environmental monitoring stations is offered, which shows the potential of our algorithms in practical applications.

Sample Balancing for Deep Learning-Based Visual Recognition.

Sample balancing includes sample selection and sample reweighting. Sample selection aims to remove some bad samples that may lead to bad local optima. Sample reweighting aims to assign optimal weights to samples to improve performance. In this article, we integrate a sample selection method based on self-paced learning into deep learning frameworks and study the influence of different sample selection strategies on training deep networks. In addition, most of the existing sample reweighting methods mainly take per-class sample number as a metric, which does not fully consider sample qualities. To improve the performance, we propose a novel metric based on the multiview semantic encoders to reweight the samples more appropriately. Then, we propose an optimization mechanism to embed sample weights into loss functions of deep networks, which can be trained in end-to-end manners. We conduct experiments on the CIFAR data set and the ImageNet data set. The experimental results demonstrate that our proposed sample balancing method can improve the performances of deep learning methods in several visual recognition tasks.

A Novel Multi-Epoch Particle Swarm Optimization Technique

Abstract Since canonical PSO method has many disadvantages which do not allow to effectively reach the global minima of various functions it needs to be improved. The article refers to a novel Multi-Epoch Particle Swarm Optimization (ME-PSO) technique which has been developed by authors. ME-PSO algorithm is based on reinitializing of the stagnant swarm with low exploration efficiency. This approach provides a high rate of global best changing. As a result ME-PSO has great possibility of finding good local (or even global) optimum and does not trap in bad local optimum. In order to prove the advantages of the ME-PSO technique numerical experiments have been carried out with ten uni- and multimodal benchmark functions. Analysis of the obtained results convincingly showed significant superiority of ME-PSO over PSO and IA-PSO algorithms. It has been set that canonical PSO is a special case of ME-PSO.

Optimal prediction for sparse linear models? Lower bounds for coordinate-separable M-estimators

For the problem of high-dimensional sparse linear regression, it is known that an $\ell_{0}$-based estimator can achieve a $1/n$ “fast” rate for prediction error without any conditions on the design matrix, whereas in the absence of restrictive conditions on the design matrix, popular polynomial-time methods only guarantee the $1/\sqrt{n}$ “slow” rate. In this paper, we show that the slow rate is intrinsic to a broad class of M-estimators. In particular, for estimators based on minimizing a least-squares cost function together with a (possibly nonconvex) coordinate-wise separable regularizer, there is always a “bad” local optimum such that the associated prediction error is lower bounded by a constant multiple of $1/\sqrt{n}$. For convex regularizers, this lower bound applies to all global optima. The theory is applicable to many popular estimators, including convex $\ell_{1}$-based methods as well as M-estimators based on nonconvex regularizers, including the SCAD penalty or the MCP regularizer. In addition, we show that bad local optima are very common, in that a broad class of local minimization algorithms with random initialization typically converge to a bad solution.

On Unsupervised Training of Multi-Class Regularized Least-Squares Classifiers

In this work we present the first efficient algorithm for unsupervised training of multi-class regularized least-squares classifiers. The approach is closely related to the unsupervised extension of the support vector machine classifier known as maximum margin clustering, which recently has received considerable attention, though mostly considering the binary classification case. We present a combinatorial search scheme that combines steepest descent strategies with powerful meta-heuristics for avoiding bad local optima. The regularized least-squares based formulation of the problem allows us to use matrix algebraic optimization enabling constant time checks for the intermediate candidate solutions during the search. Our experimental evaluation indicates the potential of the novel method and demonstrates its superior clustering performance over a variety of competing methods on real world datasets. Both time complexity analysis and experimental comparisons show that the method can scale well to practical sized problems.

Multimodal Evolutionary Algorithm for Multidimensional Scaling with City-Block Distances

Multidimensional scaling with city-block distances is considered in this paper. The technique requires optimization of an objective function which has many local minima and can be non-differentiable at minimum points. This study is aimed at developing a fast and effective global optimization algorithm spanning the whole search domain and providing good solutions. A multimodal evolutionary algorithm is used for global optimization to prevent stagnation at bad local optima. Piecewise quadratic structure of the least squares objective function with city-block distances has been exploited for local improvement. The proposed algorithm has been compared with other algorithms described in literature. Through a comprehensive computational study, it is shown that the proposed algorithm provides the best results. The algorithm with fine-tuned parameters finds the global minimum with a high probability.

Evolutionary Computation and Its Applications in Neural and Fuzzy Systems

Neural networks and fuzzy systems are two soft‐computing paradigms for system modelling. Adapting a neural or fuzzy system requires to solve two optimization problems: structural optimization and parametric optimization. Structural optimization is a discrete optimization problem which is very hard to solve using conventional optimization techniques. Parametric optimization can be solved using conventional optimization techniques, but the solution may be easily trapped at a bad local optimum. Evolutionary computation is a general‐purpose stochastic global optimization approach under the universally accepted neo‐Darwinian paradigm, which is a combination of the classical Darwinian evolutionary theory, the selectionism of Weismann, and the genetics of Mendel. Evolutionary algorithms are a major approach to adaptation and optimization. In this paper, we first introduce evolutionary algorithms with emphasis on genetic algorithms and evolutionary strategies. Other evolutionary algorithms such as genetic programming, evolutionary programming, particle swarm optimization, immune algorithm, and ant colony optimization are also described. Some topics pertaining to evolutionary algorithms are also discussed, and a comparison between evolutionary algorithms and simulated annealing is made. Finally, the application of EAs to the learning of neural networks as well as to the structural and parametric adaptations of fuzzy systems is also detailed.

A genetic algorithm for the optimal sequential partitioning problem

AbstractThe optimal sequential problem is defined as the problem of finding the minimum cost partition of the nodes of a directed acyclic graph into subsets of a given size, subject to the constraint that the precedence relationships among the elements are satisfied. A heuristic algorithm based on tabu search has been proposed for this problem [2]. However, there is a tendency for the solutions obtained by tabu search to become trapped in bad local optima in parallel graphs with random edge costs.In this paper we present a genetic algorithm for the optimal sequential partitioning problem. We develop an effective two‐point partial order crossover satisfying sequential conditions, which preserve better blocks that have a larger sum of edge costs. In this crossover we introduce the roulette selection method to escape local optima. We also assess the effectiveness of the algorithm. The results show that this proposed algorithm outperforms any other algorithm using tabu search in terms of solution quality. © 2001 Scripta Technica, Electr Eng Jpn, 135(4): 43–51, 2001

遺伝的アルゴリズムによる最適系列分割問題の解法

The optimal sequential problem is defined as the problem of finding the minimum cost partition of the nodes of a directed acyclic graph into subsets of a given size, subject to the constraint that the precedence relationships among the elements are satisfied. A heuristic algorithm based on tabu search has been proposed for this problem [2]. However, there is a tendency for the solutions obtained by tabu search to become trapped in bad local optima in parallel graphs with random edge costs. In this paper we present a genetic algorithm for the optimal sequential partitioning problem. We develop an effective two-point partial order crossover satisfying sequential conditions, which preserve better blocks that have a larger sum of edge costs. In this crossover we introduce the roulette selection method to escape local optima. We also assess the effectiveness of the algorithm. The results show that this proposed algorithm outperforms any other algorithm using tabu search in terms of solution quality. © 2001 Scripta Technica, Electr Eng Jpn, 135(4): 43–51, 2001

Application of repetitive SA method to scheduling problems of chemical processes

A scheduling algorithm using simulated annealing (SA) method is proposed in this paper. In the proposed algorithm called repetitive SA method, in order to reduce the probability of being trapped in a bad local optimum, the scheduling using SA method is executed repeatedly, and a best schedule is selected at the final stage of repetition. The problem is how to determine the number of repetitions of scheduling and the number of schedules searched in a round of scheduling. In order to find out the best combination of those two numbers, first the probability distribution of the performance of the schedule is calculated for each case where the number of schedules searched in a round of scheduling is different from one another. Then, the best combination of those numbers is selected using the derived probability distributions. The results of applying the repetitive SA method to scheduling problems of chemical processes suggest that the proposed method is effective typically in reducing the deviation of the performances of the schedules.

Application of Repetitive SA Method to Scheduling Problems of Chemical Processes

One of the predominant characteristics of chemical batch processes is that the material leaving a batch unit is fluid. Therefore, the starting times of jobs at each unit must be determined by taking into account the availability of storage between two units. In this paper, for such chemical processes, a scheduling algorithm using simulated annealing (SA) method is proposed. In the proposed method called repetitive SA method, in order to reduce the probability of being trapped in a : bad local optimum, the scheduling using SA method is executed repeatedly, and a best schedule is selected at the final stage of repetition. The problem is how to determine the number of repetitions of scheduling and the number of schedules searched in a round of scheduling. In order to find out the best combination of those two numbers, first the probability distributions of the performances of the schedules are calculated for various cases where the number of schedules searched in a round of scheduling is different from one another. Then, the best combination of those numbers is selected using the derived probability distributions. The results of applying the repetitive SA method to scheduling problems of chemical processes suggest that the proposed method is effective typically in reducing the deviation of the performances of the schedules.

General reordering algorithm for scheduling of batch processes.

A sequencing algorithm applicable to various scheduling problems is proposed. It consists of two reordering operations, the insertion of a job and the exchange of two jobs. We propose a new reordering algorithm which can be used to solve scheduling problems of multiproduct processes with network structure.The proposed reordering algorithm is applied to the scheduling problem of a multiproduct batch process consisting of parallel production lines with a shared unit. It is shown that the simultaneous reordering of several jobs effectively avoids becoming trapped in a bad local optimum. To execute simultaneous reordering of jobs, an algorithm consisting of two steps, the aggregation of jobs and the reordering of aggregated jobs, is proposed.