Deterministic Annealing Approach Research Articles

Attack Identification for Cyber-Physical Security in Dynamic Games under Cognitive Hierarchy

This paper considers the problem of identifying the profiles and capabilities of attackers injecting adversarial inputs to a cyber-physical system. The system in question interacts with attackers of different levels of intelligence, each employing different feedback controllers against the system. Principles of behavioral game theory – specifically the concept of level-k thinking – is employed to construct a database of potential attack vectors. By observing the state trajectories under sequential interactions with different adversaries, the defender adaptively estimates both the number and profiles of the different attack signals using an online deterministic annealing approach. This information is used to dynamically estimate the level of intelligence of the attackers. Simulation results showcase the efficacy of the proposed method.

Deterministic annealing Gustafson-Kessel fuzzy clustering algorithm

The Gustafson and Kessel (GK) fuzzy clustering algorithm, proposed by Gustafson and Kessel in 1979, was the first important extension to the fuzzy c-means (FCM) algorithm. Up to now, the GK algorithm had become one of the most commonly used fuzzy clustering algorithms, where the Mahalanobis distance is used as a dissimilarity measure to provide more effectiveness and robustness than the FCM algorithm. Recently, Chaomurilige et al. (2015) proposed a theoretical analysis on the parameter selection for the GK algorithm in which they indicated that the parameter of the fuzziness index heavily influences the performance of the GK algorithm. In this paper we propose a novel GK fuzzy clustering algorithm based on the deterministic annealing approach for decreasing the effect of parameters. We first consider maximizing the Shannon's entropy of membership functions to the GK objective function, and then use deterministic annealing to adjust the annealing parameter. We also mathematically provide a theoretical initialization lower bound for the annealing parameter of the proposed deterministic annealing GK (DA-GK) algorithm. Comparisons between the DA-GK algorithm and other methods are made. The computational complexity of the proposed method is also provided. Experimental results and comparisons actually verify theoretical results and also indicate the superiority and effectiveness of the proposed DA-GK algorithm.

Robust deterministic annealing based EM algorithm

A deterministic annealing (DA)-based expectation-maximisation (EM) algorithm is proposed for robust learning of Gaussian mixture models. By combing the DA approach, trimmed likelihood function and Bayesian information criterion (BIC), the proposed algorithm can simultaneously perform model selection and outlier detection, and mitigate the problems of local optima and boundary of parameter space with the conventional EM algorithm. Experiments demonstrate that the proposed algorithm can determine the number of components correctly even though the data are contaminated by outliers.

Channel Optimized Distributed Multiple Description Coding

In this paper, channel optimized distributed multiple description vector quantization (CDMD) schemes are presented for distributed source coding in symmetric and asymmetric settings. The CDMD encoder is designed using a deterministic annealing approach over noisy channels with packet loss. A minimum mean squared error asymmetric CDMD decoder is proposed for effective reconstruction of a source, utilizing the side information (SI) and its corresponding received descriptions. The proposed iterative symmetric CDMD decoder jointly reconstructs the symbols of multiple correlated sources. Two types of symmetric CDMD decoders, namely the estimated-SI and the soft-SI decoders, are presented which respectively exploit the reconstructed symbols and a posteriori probabilities of other sources as SI in iterations. In a multiple source CDMD setting, for reconstruction of a source, three methods are proposed to select another source as its SI during the decoding. The methods operate based on minimum physical distance (in a wireless sensor network setting), maximum mutual information and minimum end-to-end distortion. The performance of the proposed systems and algorithms are evaluated and compared in detail.

Deterministic Annealing Approach to Fuzzy C-Means Clustering Based on Entropy Maximization

This paper is dealing with the fuzzy clustering method which combines the deterministic annealing (DA) approach with an entropy, especially the Shannon entropy and the Tsallis entropy. By maximizing the Shannon entropy, the fuzzy entropy, or the Tsallis entropy within the framework of the fuzzy c-means (FCM) method, membership functions similar to the statistical mechanical distribution functions are obtained. We examine characteristics of these entropy-based membership functions from the statistical mechanical point of view. After that, both the Shannon- and Tsallis-entropy-based FCMs are formulated as DA clustering using the very fast annealing (VFA) method as a cooling schedule. Experimental results indicate that the Tsallis-entropy-based FCM is stable with very fast deterministic annealing and suitable for this annealing process.

Generative topographic mapping by deterministic annealing

Abstract Generative Topographic Mapping (GTM) is an important technique for dimension reduction which has been successfully applied to many fields. However the usual Expectation-Maximization (EM) approach to GTM can easily get stuck in local minima and so we introduce a Deterministic Annealing (DA) approach to GTM which is more robust and less sensitive to initial conditions so we do not need to use many initial values to find good solutions. DA has been very successful in clustering, hidden Markov Models and Multidimensional Scaling but typically uses a fixed cooling schemes to control the temperature of the system. We propose a new cooling scheme which can adaptively adjust the choice of temperature in the middle of process to find better solutions. Our experimental measurements suggest that deterministic annealing improves the quality of GTM solutions.

A regularization framework for multiclass classification: A deterministic annealing approach

In this paper, we propose a general regularization framework for multiclass classification based on discriminant functions. Since the objective function in the primal optimization problem of this framework is always not differentiable, the optimal solution cannot be obtained directly. With the aid of the deterministic annealing approach, a differentiable objective function is derived subject to a constraint on the randomness of the solution. The problem can be approximated by solving a sequence of differentiable optimization problems, and such approximation converges to the original problem asymptotically. Based on this approach, class-conditional posterior probabilities can be calculated directly without assuming the underlying probabilistic model. We also notice that there is a connection between our approach and some existing statistical models, such as Fisher discriminant analysis and logistic regression.

Robust Distributed Source Coder Design by Deterministic Annealing

This paper considers the design of efficient quantizers for a <i xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">robust</i> distributed source coding system. The information is encoded at independent terminals and transmitted across separate channels, any of which may fail. The scenario subsumes a wide range of source and source-channel coding/quantization problems, including multiple descriptions and distributed source coding. Greedy descent methods depend heavily on initialization, and the presence of abundant (high density of) ¿poor¿ local optima on the cost surface strongly motivates the use of a global design algorithm. We propose a deterministic annealing approach for the design of all components of a generic robust distributed source coding system. Our approach avoids many poor local optima, is independent of initialization, and does not make any simplifying assumption on the underlying source distribution. Simulation results demonstrate a wide spread in the performance of greedy Lloyd-based algorithms, and considerable gains are achieved by using the proposed deterministic annealing approach.

Optimizing an organized modularity measure for topographic graph clustering: A deterministic annealing approach

This paper proposes an organized generalization of Newman and Girvan's modularity measure for graph clustering. Optimized via a deterministic annealing scheme, this measure produces topologically ordered graph clusterings that lead to faithful and readable graph representations based on clustering induced graphs. Topographic graph clustering provides an alternative to more classical solutions in which a standard graph clustering method is applied to build a simpler graph that is then represented with a graph layout algorithm. A comparative study on four real world graphs ranging from 34 to 1133 vertices shows the interest of the proposed approach with respect to classical solutions and to self-organizing maps for graphs.

Image change detection from difference image through deterministic simulated annealing

This paper proposes an automatic method based on the deterministic simulated annealing (DSA) approach for solving the image change detection problem between two images where one of them is the reference image. Each pixel in the reference image is considered as a node with a state value in a network of nodes. This state determines the magnitude of the change. The DSA optimization approach tries to achieve the most network stable configuration based on the minimization of an energy function. The DSA scheme allows the mapping of interpixel contextual dependencies which has been used favorably in some existing image change detection strategies. The main contribution of the DSA is exactly its ability for avoiding local minima during the optimization process thanks to the annealing scheme. Local minima have been detected when using some optimization strategies, such as Hopfield neural networks, in images with large amount of changes, greater than the 20%. The DSA performs better than other optimization strategies for images with a large amount of changes and obtain similar results for images where the changes are small. Hence, the DSA approach appears to be a general method for image change detection independently of the amount of changes. Its performance is compared against some recent image change detection methods.

Entropy Maximization and Deterministic Annealing Approach to Fuzzy C-means Clustering

Entropy Maximization and Deterministic Annealing Approach to Fuzzy C-means Clustering

Evaluation of a convex relaxation to a quadratic assignment matching approach for relational object views

We introduce a convex relaxation approach for the quadratic assignment problem to the field of computer vision. Due to convexity, a favourable property of this approach is the absence of any tuning parameters and the computation of high-quality combinatorial solutions by solving a mathematically simple optimization problem. Furthermore, the relaxation step always computes a tight lower bound of the objective function and thus can additionally be used as an efficient subroutine of an exact search algorithm. We report the results of both established benchmark experiments from combinatorial mathematics and random ground-truth experiments using computer-generated graphs. For comparison, a deterministic annealing approach is investigated as well. Both approaches show similarly good performance. In contrast to the convex approach, however, the annealing approach yields no problem relaxation, and four parameters have to be tuned by hand for the annealing algorithm to become competitive.

A Method for Segmentation of Switching Dynamic Modes in Time Series

A method to identify switching dynamics in time series, based on Annealed Competition of Experts algorithm (ACE), has been developed by Kohlmorgen et al. Incorrect selection of embedding dimension and time delay of the signal significantly affect the performance of the ACE method, however. In this paper, we utilize systematic approaches based on mutual information and false nearest neighbor to determine appropriate embedding dimension and time delay. Moreover, we obtained further improvements to the original ACE method by incorporating a deterministic annealing approach as well as phase space closeness measure. Using these improved implementations, we have enhanced the performance of the ACE algorithm in determining the location of the switching of dynamic modes in the time series. The application of the improved ACE method to heart rate data obtained from rats during control and administration of double autonomic blockade conditions indicate that the improved ACE algorithm is able to segment dynamic mode changes with pinpoint accuracy and that its performance is superior to the original ACE algorithm.

Reduced-Complexity Deterministic Annealing for Vector Quantizer Design

This paper presents a reduced-complexity deterministic annealing (DA) approach for vector quantizer (VQ) design by using soft information processing with simplified assignment measures. Low-complexity distributions are designed to mimic the Gibbs distribution, where the latter is the optimal distribution used in the standard DA method. These low-complexity distributions are simple enough to facilitate fast computation, but at the same time they can closely approximate the Gibbs distribution to result in near-optimal performance. We have also derived the theoretical performance loss at a given system entropy due to using the simple soft measures instead of the optimal Gibbs measure. We use the derived result to obtain optimal annealing schedules for the simple soft measures that approximate the annealing schedule for the optimal Gibbs distribution. The proposed reduced-complexity DA algorithms have significantly improved the quality of the final codebooks compared to the generalized Lloyd algorithm and standard stochastic relaxation techniques, both with and without the pairwise nearest neighbor (PNN) codebook initialization. The proposed algorithms are able to evade the local minima and the results show that they are not sensitive to the choice of the initial codebook. Compared to the standard DA approach, the reduced-complexity DA algorithms can operate over 100 times faster with negligible performance difference. For example, for the design of a 16-dimensional vector quantizer having a rate of 0.4375 bit/sample for Gaussian source, the standard DA algorithm achieved 3.60 dB performance in 16 483 CPU seconds, whereas the reduced-complexity DA algorithm achieved the same performance in 136 CPU seconds. Other than VQ design, the DA techniques are applicable to problems such as classification, clustering, and resource allocation.

Reduced-complexity deterministic annealing for vector quantizer design

This paper presents a reduced-complexity deterministic annealing (DA) approach for vector quantizer (VQ) design by using soft information processing with simplified assignment measures. Low-complex...

Multiple description quantization by deterministic annealing

The design of vector quantizers for diversity-based communication over two or more channels of possibly differing capacities and failure probabilities, is considered. The crucial dependence of current design techniques on initialization, especially of index assignment, is well recognized. Instead, we propose to pursue a deterministic annealing approach which is independent of initialization, does not assume any prior knowledge of the source density, and avoids many poor local minima of the cost surface. The approach consists of iterative optimization of a random encoder at gradually decreasing levels of randomness as measured by the Shannon entropy. At the limit of zero entropy, a hard multiple description (MD) quantizer is obtained. This process is directly analogous to annealing processes in statistical physics. Via an alternative derivation, we show that it may also be interpreted as approximating the minimum rate sums among points on the convex hull of the MD achievable rate-distortion region of El Gamal and Cover, subject to constraints on the sizes of the reproduction alphabets. To illustrate the potential of our approach, we present simulation results that show substantial performance gains over existing design techniques.

A deterministic annealing approach for parsimonious design of piecewise regression models

A new learning algorithm is proposed for piecewise regression modeling. It employs the technique of deterministic annealing to design space partition regression functions. While the performance of traditional space partition regression functions such as CART and MARS is limited by a simple tree-structured partition and by a hierarchical approach for design, the deterministic annealing algorithm enables the joint optimization of a more powerful piecewise structure based on a Voronoi partition. The new method is demonstrated to achieve consistent performance improvements over regular CART as well as over its extension to allow arbitrary hyperplane boundaries. Comparison tests, on several benchmark data sets from the regression literature, are provided.

Deterministic annealing for clustering, compression, classification, regression, and related optimization problems

The deterministic annealing approach to clustering and its extensions has demonstrated substantial performance improvement over standard supervised and unsupervised learning methods in a variety of important applications including compression, estimation, pattern recognition and classification, and statistical regression. The application-specific cost is minimized subject to a constraint on the randomness of the solution, which is gradually lowered. We emphasize the intuition gained from analogy to statistical physics. Alternatively the method is derived within rate-distortion theory, where the annealing process is equivalent to computation of Shannon's rate-distortion function, and the annealing temperature is inversely proportional to the slope of the curve. The basic algorithm is extended by incorporating structural constraints to allow optimization of numerous popular structures including vector quantizers, decision trees, multilayer perceptrons, radial basis functions, and mixtures of experts.

Pairwise data clustering by deterministic annealing

Partitioning a data set and extracting hidden structure from the data arises in different application areas of pattern recognition, speech and image processing. Pairwise data clustering is a combinatorial optimization method for data grouping which extracts hidden structure from proximity data. We describe a deterministic annealing approach to pairwise clustering which shares the robustness properties of maximum entropy inference. The resulting Gibbs probability distributions are estimated by mean-field approximation. A new structure-preserving algorithm to cluster dissimilarity data and to simultaneously embed these data in a Euclidian vector space is discussed which can be used for dimensionality reduction and data visualization. The suggested embedding algorithm which outperforms conventional approaches has been implemented to analyze dissimilarity data from protein analysis and from linguistics. The algorithm for pairwise data clustering is used to segment textured images.

A global optimization technique for statistical classifier design

A global optimization method is introduced that minimize the rate of misclassification. We first derive the theoretical basis for the method, on which we base the development of a novel design algorithm and demonstrate its effectiveness and superior performance in the design of practical classifiers for some of the most popular structures currently in use. The method, grounded in ideas from statistical physics and information theory, extends the deterministic annealing approach for optimization, both to incorporate structural constraints on data assignments to classes and to minimize the probability of error as the cost objective. During the design, data are assigned to classes in probability so as to minimize the expected classification error given a specified level of randomness, as measured by Shannon's entropy. The constrained optimization is equivalent to a free-energy minimization, motivating a deterministic annealing approach in which the entropy and expected misclassification cost are reduced with the temperature while enforcing the classifier's structure. In the limit, a hard classifier is obtained. This approach is applicable to a variety of classifier structures, including the widely used prototype-based, radial basis function, and multilayer perceptron classifiers. The method is compared with learning vector quantization, back propagation (BP), several radial basis function design techniques, as well as with paradigms for more directly optimizing all these structures to minimize probability of error. The annealing method achieves significant performance gains over other design methods on a number of benchmark examples from the literature, while often retaining design complexity comparable with or only moderately greater than that of strict descent methods. Substantial gains, both inside and outside the training set, are achieved for complicated examples involving high-dimensional data and large class overlap.