Bayesian Ying-Yang Learning Research Articles

Further advances on Bayesian Ying-Yang harmony learning

After a short tutorial on the fundamentals of Bayes approaches and Bayesian Ying-Yang (BYY) harmony learning, this paper introduces new progresses. A generic information harmonising dynamics of BYY harmony learning is proposed with the help of a Lagrange variety preservation principle, which provides Lagrange-like implementations of Ying-Yang alternative nonlocal search for various learning tasks and unifies attention, detection, problem-solving, adaptation, learning and model selection from an information harmonising perspective. In this framework, new algorithms are developed to implement Ying-Yang alternative nonlocal search for learning Gaussian mixture and several typical exemplars of linear matrix system, including factor analysis (FA), mixture of local FA, binary FA, nonGaussian FA, de-noised Gaussian mixture, sparse multivariate regression, temporal FA and temporal binary FA, as well as a generalised bilinear matrix system that covers not only these linear models but also manifold learning, gene regulatory networks and the generalised linear mixed model. These algorithms are featured with a favourable nature of automatic model selection and a unified formulation in performing unsupervised learning and semi-supervised learning. Also, we propose a principle of preserving multiple convex combinations, which leads alternative search algorithms. Finally, we provide a chronological outline of the history of BYY learning studies.

Adaptive electric load forecaster

In this paper, a methodology, Self-Developing and Self-Adaptive Fuzzy Neural Networks using Type-2Fuzzy Bayesian Ying-Yang Learning(SDSA-FNN-T2FBYYL) algorithm and multi-objective optimization is proposed.The features of this methodology are as follows:(1) A Bayesian Ying-Yang Learning(BYYL) algorithm is used to construct a compact but high-performance system automatically.(2) A novel multi-objective T2 FBYYL is presented that integrates the T2 fuzzy theory with BYYL to automatically construct its best structure and better tackle various data uncertainty problems simultaneously.(3) The weighted sum multi-objective optimization technique with combinations of different weightings is implemented to achieve the best trade-off among multiple objectives in the T2 FBYYL. The proposed methods are applied to electric load forecast using a real operational dataset collected from Macao electric utility. The test results reveal that the proposed method is superior to other existing relevant techniques.

Projection-embedded BYY learning algorithm for Gaussian mixture-based clustering

Abstract On learning the Gaussian mixture model, existing BYY learning algorithms are featured by a gradient-based line search with an appropriate stepsize. Learning becomes either unstable if the stepsize is too large or slow and gets stuck in a local optimal solution if the stepsize is too small. An algorithm without a learning stepsize has been proposed with expectation-maximization (EM) like two alternative steps. However, its learning process may still be unstable. This paper tackles this problem of unreliability by a modified algorithm called projection-embedded Bayesian Ying-Yang learning algorithm (pBYY). Experiments have shown that pBYY outperforms learning algorithms developed from not only minimum message length with Jeffreys prior (MML-Jef) and Variational Bayesian with Dirichlet-Normal-Wishart (VB-DNW) prior but also BYY with these priors (BYY-Jef and BYY-DNW). pBYY obtains the superiority with an easy implementation, while DNW prior-based learning algorithms suffer a complicated and tedious computation load. The performance of pBYY has also been demonstrated on the Berkeley Segmentation Dataset for the topic of unsupervised image segmentation. The resulted performances of semantic image segmentation have shown that pBYY outperforms not only MML-Jef, VB-DNW, BYY-Jef, and BYY-DNW but also three leading image segmentation algorithms, namely gPb-owt-ucm, MN-Cut, and mean shift.

INTELLIGENT SELF-DEVELOPING AND SELF-ADAPTIVE ELECTRIC LOAD FORECASTER BASED ON TYPE-2 FUZZY BAYESIAN YING-YANG LEARNING ALGORITHM

In new advocated “smart grid” development, an electric load forecaster should possess high-level intelligence in order to handle higher uncertainty, indefiniteness, and variability on electric load demand. The intelligence is referred to as self-learning, self-adaptability, and the highest capability of handling various uncertainties, which the forecaster should possess. In this study, a novel methodology, self-developing and self-adaptive fuzzy neural networks using type-2 fuzzy Bayesian Ying-Yang learning algorithm (SDSA-FNN-T2BYYL) is proposed. Its novelty is that (1) the Bayesian Ying-Yang learning algorithm (BYYL) is used to construct a compact system structure automatically. (2) Further, a novel T2 fuzzy BYYL is presented, which integrates type-2 (T2) fuzzy theory and BYYL in order to achieve two objectives simultaneously: compact system structure and better handling of data uncertainty. (3) Because a training dataset cannot include all possible operation conditions, the system should be able to restructure continuously for good generalization. Consequently, a T2 fuzzy BYY split-and-merge algorithm is proposed. The proposed method is validated using a real operational dataset collected from a Macao electric utility. Simulation and test results reveal that SDSA-FNN-T2BYYL has superior accuracy for load forecasting over other existing relevant techniques.

Image Segmentation with the EM and the BYY Learning

The aim of the study is to present an image segmentation method based on feature space clustering with the Gaussian Mixture Model (GMM) and the Expectation Maximization (EM) algorithm. To address the issue of selection of determining the number of clusters, the Bayesian Ying-Yang (BYY) learning is employed. Moreover, to improve the performance of the segmentation on noisy images, both intensity and spatial position information are employed as features to describe a pixel in the image. The simulation results on images with and without noises validate the performance of the proposed method.

KCMAC-BYY: Kernel CMAC using Bayesian Ying–Yang learning

The Cerebellar Model Articulation Controller (CMAC) possesses attractive properties of fast learning and simple computation. In application, the size of its association vector is always reduced to economize the memory requirement, greatly constraining its modeling capability. The kernel CMAC (KMAC), which provides an interpretation for the traditional CMAC from the kernel viewpoint, not only strengthens the modeling capability without increasing its complexity, but reinforces its generalization with the help of a regularization term. However, the KCMAC suffers from the problem of selecting its hyperparameter. In this paper, the Bayesian Ying–Yang (BYY) learning theory is incorporated into KCMAC, referred to as KCMAC-BYY, to optimize the hyperparameter. The proposed KCMAC-BYY achieves the systematic tuning of the hyperparameter, further improving the performance in modeling and generalization. The experimental results on some benchmark datasets show the prior performance of the proposed KCMAC-BYY to the existing representative techniques.

Transcription network analysis by a sparse binary factor analysis algorithm.

Transcription factor activities (TFAs), rather than expression levels, control gene expression and provide valuable information for investigating TF-gene regulations. The underlying bimodal or switch-like patterns of TFAs may play important roles in gene regulation. Network Component Analysis (NCA) is a popular method to deduce TFAs and TF-gene control strengths from microarray data. However, it does not directly examine the bimodality of TFAs and it needs TF-gene connection topology a priori known. In this paper, we modify NCA to model gene expression regulation by Binary Factor Analysis (BFA), which directly captures switch-like patterns of TFAs. Moreover, sparse technique is employed on the mixing matrix of BFA, and thus the proposed sparse BYY-BFA algorithm, developed under Bayesian Ying-Yang (BYY) learning framework, can not only uncover the latent TFA profile’s switch-like patterns, but also be capable of automatically shutting off the unnecessary connections. Simulation study demonstrates the effectiveness of BYY-BFA, and a preliminary application to Saccharomyces cerevisiae cell cycle data and Escherichia coli carbon source transition data shows that the reconstructed binary patterns of TFAs by BYY-BFA are consistent with the ups and downs of TFAs by NCA, and that BYY-BFA also works well when the network topology is unknown.

Transcription Network Analysis by A Sparse Binary Factor Analysis Algorithm

Summary Transcription factor activities (TFAs), rather than expression levels, control gene expres- sion and provide valuable information for investigating TF-gene regulations. The underly- ing bimodal or switch-like patterns of TFAs may play important roles in gene regulation. Network Component Analysis (NCA) is a popular method to deduce TFAs and TF-gene control strengths from microarray data. However, it does not directly examine the bimodal- ity of TFAs and it needs the TF-gene connection topology to be a priori known. In this paper, we modify NCA to model gene expression regulation by Binary Factor Analysis (BFA), which directly captures switch-like patterns of TFAs. Moreover, sparse technique is employed on the mixing matrix of BFA, and thus the proposed sparse BYY-BFA al- gorithm, developed under Bayesian Ying-Yang (BYY) learning framework, can not only uncover the latent TFA profile’s switch-like patterns, but also be capable of automatically shutting off the unnecessary connections. Simulation study demonstrates the effectiveness of BYY-BFA, and a preliminary application to Saccharomyces cerevisiae cell cycle data and Escherichia coli carbon source transition data shows that the reconstructed binary pat- terns of TFAs by BYY-BFA are consistent with the ups and downs of TFAs by NCA, and that BYY-BFA also works well when the network topology is unknown.

Optimization of fuzzy CMAC using evolutionary Bayesian Ying-Yang learning

Cerebellar model articulation controller (CMAC) is a popular associative memory neural network that imitates human’s cerebellum, which allows it to learn fast and carry out local generalization efficiently. This research aims to integrate evolutionary computation into fuzzy CMAC Bayesian Ying-Yang (FCMACBYY) learning, which is referred to as FCMAC-EBYY, to achieve a synergetic development in the search for optimal fuzzy sets and connection weights. Traditional evolutionary approaches are limited to small populations of short binary string length and as such are not suitable for neural network training, which involves a large searching space due to complex connections as well as real values. The methodology employed by FCMACEBYY is coevolution, in which a complex solution is decomposed into some pieces to be optimized in different populations/species and then assembled. The developed FCMAC-EBYY is compared with various neuro-fuzzy systems using a real application of traffic flow prediction.

Parameterizations make different model selections: Empirical findings from factor analysis

How parameterizations affect model selection performance is an issue that has been ignored or seldom studied since traditional model selection criteria, such as Akaike’s information criterion (AIC), Schwarz’s Bayesian information criterion (BIC), difference of negative log-likelihood (DNLL), etc., perform equivalently on different parameterizations that have equivalent likelihood functions. For factor analysis (FA), in addition to one traditional model (shortly denoted by FA-a), it was previously found that there is another parameterization (shortly denoted by FA-b) and the Bayesian Ying-Yang (BYY) harmony learning gets different model selection performances on FA-a and FA-b. This paper investigates a family of FA parameterizations that have equivalent likelihood functions, where each one (shortly denoted by FA-r) is featured by an integer r, with FA-a as one end that r = 0 and FA-b as the other end that r reaches its upper-bound. In addition to the BYY learning in comparison with AIC, BIC, and DNLL, we also implement variational Bayes (VB). Several empirical finds have been obtained via extensive experiments. First, both BYY and VB perform obviously better on FA-b than on FA-a, and this superiority of FA-b is reliable and robust. Second, both BYY and VB outperform AIC, BIC, and DNLL, while BYY further outperforms VB considerably, especially on FA-b. Moreover, with FA-a replaced by FA-b, the gain obtained by BYY is obviously higher than the one by VB, while the gain by VB is better than no gain by AIC, BIC, and DNLL. Third, this paper also demonstrates how each part of priors incrementally and jointly improves the performances, and further shows that using VB to optimize the hyperparameters of priors deteriorates the performances while using BYY for this purpose can further improve the performances.

Fuzzy CMAC With Incremental Bayesian Ying–Yang Learning and Dynamic Rule Construction

Inspired by the philosophy of ancient Chinese Taoism, Xu's Bayesian ying-yang (BYY) learning technique performs clustering by harmonizing the training data (yang) with the solution (ying). In our previous work, the BYY learning technique was applied to a fuzzy cerebellar model articulation controller (FCMAC) to find the optimal fuzzy sets; however, this is not suitable for time series data analysis. To address this problem, we propose an incremental BYY learning technique in this paper, with the idea of sliding window and rule structure dynamic algorithms. Three contributions are made as a result of this research. First, an online expectation-maximization algorithm incorporated with the sliding window is proposed for the fuzzification phase. Second, the memory requirement is greatly reduced since the entire data set no longer needs to be obtained during the prediction process. Third, the rule structure dynamic algorithm with dynamically initializing, recruiting, and pruning rules relieves the "curse of dimensionality" problem that is inherent in the FCMAC. Because of these features, the experimental results of the benchmark data sets of currency exchange rates and Mackey-Glass show that the proposed model is more suitable for real-time streaming data analysis.

Machine learning problems from optimization perspective

Both optimization and learning play important roles in a system for intelligent tasks. On one hand, we introduce three types of optimization tasks studied in the machine learning literature, corresponding to the three levels of inverse problems in an intelligent system. Also, we discuss three major roles of convexity in machine learning, either directly towards a convex programming or approximately transferring a difficult problem into a tractable one in help of local convexity and convex duality. No doubly, a good optimization algorithm takes an essential role in a learning process and new developments in the literature of optimization may thrust the advances of machine learning. On the other hand, we also interpret that the key task of learning is not simply optimization, as sometimes misunderstood in the optimization literature. We introduce the key challenges of learning and the current status of efforts towards the challenges. Furthermore, learning versus optimization has also been examined from a unified perspective under the name of Bayesian Ying-Yang learning, with combinatorial optimization made more effectively in help of learning.

An online Bayesian Ying–Yang learning applied to fuzzy CMAC

This paper proposes an online Bayesian Ying–Yang (OBYY) clustering algorithm, which is then applied to the fuzzy cerebellar model articulation controller (FCMAC). Inspired by ancient Chinese Ying–Yang philosophy, Xu's Bayesian Ying–Yang (BYY) learning has been successfully applied to clustering by harmonizing the visible input data (Yang) and the invisible clusters (Ying). In this research, the original BYY is advanced to dynamically recruit, adjust, and merge the fuzzy clusters to achieve maximum harmony and highest membership values. The proposed online FCMAC-BYY offers the following advantages. First, the antecedent of the fuzzy rules are dynamically constructed and optimized by the OBYY algorithm during the operation of the system. Second, the credit assignment is then employed in the learning process of the neurons to greatly speed up the learning process. These features make the entire online FCMAC-BYY an optimal structure with a fast learning speed that can perform online learning and suitable for real-time applications. The experimental results on some benchmark datasets show that the proposed model outperforms the existing representative techniques.

An experimental study of the extended NRBF regression model and its enhancement for classification problem

As an extension of the traditional normalized radial basis function (NRBF) model, the extended normalized RBF ( ENRBF) model was proposed by Xu [RBF nets, mixture experts, and Bayesian Ying-Yang learning, Neurocomputing 19 (1998) 223–257]. In this paper, we perform a supplementary study on ENRBF with several properly designed experiments and some further theoretical discussions. It is shown that ENRBF is able to efficiently improve the learning accuracies under some circumstances. Moreover, since the ENRBF model is initially proposed for the regression and function approximation problems, a further step is taken in this work to modify the ENRBF model to deal with the classification problems. Both the original ENRBF model and the new proposed ENRBF classifier (ENRBFC) can be viewed as the special cases of the mixture-of-experts (ME) model that is discussed in Xu et al. [An alternative model for mixtures of experts, in: Advances in Neural Information Processing Systems, MIT Press, Cambridge, MA, 1995]. Experimental results show the potentials of ENRBFC compared to some other related classifiers.

FCMAC-BYY: Fuzzy CMAC Using Bayesian Ying–Yang Learning

As an associative memory neural network model, the cerebellar model articulation controller (CMAC) has attractive properties of fast learning and simple computation, but its rigid structure makes it difficult to approximate certain functions. This research attempts to construct a novel neural fuzzy CMAC, in which Bayesian Ying-Yang (BYY) learning is introduced to determine the optimal fuzzy sets, and a truth-value restriction inference scheme is subsequently employed to derive the truth values of the rule weights of implication rules. The BYY is motivated from the famous Chinese ancient Ying-Yang philosophy: everything in the universe can be viewed as a product of a constant conflict between opposites-Ying and Yang, a perfect status is reached when Ying and Yang achieve harmony. The proposed fuzzy CMAC (FCMAC)-BYY enjoys the following advantages. First, it has a higher generalization ability because the fuzzy rule sets are systematically optimized by BYY; second, it reduces the memory requirement of the network by a significant degree as compared to the original CMAC; and third, it provides an intuitive fuzzy logic reasoning and has clear semantic meanings. The experimental results on some benchmark datasets show that the proposed FCMAC-BYY outperforms the existing representative techniques in the research literature.

Temporal BYY Encoding, Markovian State Spaces, and Space Dimension Determination

As a complementary to those temporal coding approaches of the current major stream, this paper aims at the Markovian state space temporal models from the perspective of the temporal Bayesian Ying-Yang (BYY) learning with both new insights and new results on not only the discrete state featured Hidden Markov model and extensions but also the continuous state featured linear state spaces and extensions, especially with a new learning mechanism that makes selection of the state number or the dimension of state space either automatically during adaptive learning or subsequently after learning via model selection criteria obtained from this mechanism. Experiments are demonstrated to show how the proposed approach works.

Investigations on Non-Gaussian Factor Analysis

This letter further explores the Bayesian Ying-Yang learning based non-Gaussian factor analysis (NFA) via investigating its key yet analytically intractable factor estimating step. Among the three suggested numerical approaches we empirically show that the so-called iterative fixed posteriori approximation approach is the most optimal, as well as theoretically prove that the iterative fixed posteriori approximation is another type of EM-algorithm, with the proof of its convergence also shown.

RBF nets, mixture experts, and Bayesian Ying–Yang learning

The connections of the alternative model for mixture of experts (ME) to the normalized radial basis function (NRBF) nets and extended normalized RBF (ENRBF) nets are established, and the well-known expectation-maximization (EM) algorithm for maximum likelihood learning is suggested to the two types of RBF nets. This new learning technique determines the parameters of the input layer (including the covariance matrices or so-called receptive fields) and the parameters of the output layer in the RBF nets globally, instead of separately training the input layer by the K-means algorithm and the output layer by the least-squares learning as done in most of the existing RBF learning methods. In addition, coordinated competitive learning (CCL) and adaptive algorithms are proposed to approximate the EM algorithm for considerably speeding up the learning of the original and alternative ME models as well as the NRBF and ENRBF nets. Furthermore, the two ME models are linked to the recent proposed Bayesian Ying–Yang (BYY) learning system and theory such that not only the architecture of ME and RBF nets is shown to be more preferred than multilayer architecture, but also a new model selection criterion has been obtained to determine the number of experts and basis functions. A number of experiments are made on the prediction of foreign exchange rate and trading investment as well as piecewise nonlinear regression and piecewise line fitting. As shown in these experiments, the EM algorithm for NRBF nets and ENRBF nets obviously outperforms the conventional RBF learning technique, CCL speeds up the learning considerably with only a slight sacrifice on performance accuracy, the adaptive algorithm gives significant improvements on financial predication and trading investment, as well as that the proposed criterion can select the number of basis functions successfully. In addition, the ENRBF net and the alternative ME model are also shown to be able to implement curve fitting and detection.

Bayesian Ying–Yang machine, clustering and number of clusters

It is shown that a particular case of the Bayesian Ying–Yang learning system and theory reduces to the maximum likelihood learning of a finite mixture, from which we have obtained not only the EM algorithm for its parameter estimation and its various approximate but fast algorithms for clustering in general cases (including Mahalanobis distance clustering or elliptic clustering), but also criteria for the selection of the number of densities in a mixture, and the number k in the conventional Mean Square Error clustering. Moreover, a Re-weighted EM algorithm is also proposed and shown to be more robust in learning. Finally, experimental results are provided.