Bayesian Ying-Yang Research Articles

Automated Model Selection of the Two-Layer Mixtures of Gaussian Process Functional Regressions for Curve Clustering and Prediction

As a reasonable statistical learning model for curve clustering analysis, the two-layer mixtures of Gaussian process functional regressions (TMGPFR) model has been developed to fit the data of sample curves from a number of independent information sources or stochastic processes. Since the sample curves from a certain stochastic process naturally form a curve cluster, the model selection of TMGPFRs, i.e., the selection of the number of mixtures of Gaussian process functional regressions (MGPFRs) in the upper layer, corresponds to the discovery of the cluster number and structure of the curve data. In fact, this is rather challenging because the conventional model selection criteria, such as BIC and cross-validation, cannot lead to a stable result in practice even with a heavy burden of repetitive computation. In this paper, we improve the original TMGPFR model and propose a Bayesian Ying-Yang (BYY) annealing learning algorithm for the parameter learning of the improved model with automated model selection. The experimental results of both synthetic and realistic datasets demonstrate that our proposed algorithm can make correct model selection automatically during parameter learning of the model.

BYY harmony learning of t-mixtures with the application to image segmentation based on contourlet texture features

In this paper, we extend Bayesian Ying-Yang (BYY) harmony learning to the case of multivariate t-mixtures and propose a gradient BYY harmony learning algorithm that can automatically determine the number of actual t-distributions in a dataset during parameter learning. It is demonstrated by simulation experiments that this proposed algorithm for t-mixtures is both effective and stable on model selection and parameter estimation. Moreover, by mainly utilizing certain contourlet texture features from an image, the proposed algorithm is successfully applied to unsupervised image segmentation, showing considerable advantages for both general and multi-texture images.

Further advances on Bayesian Ying-Yang harmony learning

After a short tutorial on the fundamentals of Bayes approaches and Bayesian Ying-Yang (BYY) harmony learning, this paper introduces new progresses. A generic information harmonising dynamics of BYY harmony learning is proposed with the help of a Lagrange variety preservation principle, which provides Lagrange-like implementations of Ying-Yang alternative nonlocal search for various learning tasks and unifies attention, detection, problem-solving, adaptation, learning and model selection from an information harmonising perspective. In this framework, new algorithms are developed to implement Ying-Yang alternative nonlocal search for learning Gaussian mixture and several typical exemplars of linear matrix system, including factor analysis (FA), mixture of local FA, binary FA, nonGaussian FA, de-noised Gaussian mixture, sparse multivariate regression, temporal FA and temporal binary FA, as well as a generalised bilinear matrix system that covers not only these linear models but also manifold learning, gene regulatory networks and the generalised linear mixed model. These algorithms are featured with a favourable nature of automatic model selection and a unified formulation in performing unsupervised learning and semi-supervised learning. Also, we propose a principle of preserving multiple convex combinations, which leads alternative search algorithms. Finally, we provide a chronological outline of the history of BYY learning studies.

Adaptive electric load forecaster

In this paper, a methodology, Self-Developing and Self-Adaptive Fuzzy Neural Networks using Type-2Fuzzy Bayesian Ying-Yang Learning(SDSA-FNN-T2FBYYL) algorithm and multi-objective optimization is proposed.The features of this methodology are as follows:(1) A Bayesian Ying-Yang Learning(BYYL) algorithm is used to construct a compact but high-performance system automatically.(2) A novel multi-objective T2 FBYYL is presented that integrates the T2 fuzzy theory with BYYL to automatically construct its best structure and better tackle various data uncertainty problems simultaneously.(3) The weighted sum multi-objective optimization technique with combinations of different weightings is implemented to achieve the best trade-off among multiple objectives in the T2 FBYYL. The proposed methods are applied to electric load forecast using a real operational dataset collected from Macao electric utility. The test results reveal that the proposed method is superior to other existing relevant techniques.

BYY harmony learning of log-normal mixtures with automated model selection

Bayesian Ying–Yang (BYY) harmony learning system is a powerful tool for statistical learning. Via the BYY harmony leaning of finite mixtures, model selection, i.e., the selection of an appropriate number of components for the mixture, can be made automatically during parameter learning on a given dataset. In this paper, an adaptive gradient BYY harmony learning algorithm is proposed for log-normal mixtures to implement parameter learning with automated model selection. It is demonstrated by the experiments on both synthetic and real-world datasets that the proposed BYY harmony learning algorithm not only has the ability of automated model selection, but also leads to a rather good estimation of the parameters in the original log-normal mixture.

Learning local factor analysis versus mixture of factor analyzers with automatic model selection

Considering Factor Analysis (FA) for each component of Gaussian Mixture Model (GMM), clustering and local dimensionality reduction can be addressed simultaneously by Mixture of Factor Analyzers (MFA) and Local Factor Analysis (LFA), which correspond to two FA parameterizations, respectively. This paper investigates the performance of Variational Bayes (VB) and Bayesian Ying-Yang (BYY) harmony learning on MFA/LFA for the problem of automatically determining the component number and the local hidden dimensionalities (i.e., the number of factors of FA in each component). Similar to the existing VB learning algorithm on MFA, we develop an alternative VB algorithm on LFA with a similar conjugate Dirichlet–Normal–Gamma (DNG) prior on all parameters of LFA. Also, the corresponding BYY algorithms are developed for MFA and LFA. A wide range of synthetic experiments shows that LFA is superior to MFA in model selection under either VB or BYY, while BYY outperforms VB reliably on both MFA and LFA. These empirical findings are consistently observed from real applications on not only face and handwritten digit images clustering, but also unsupervised image segmentation.

Learning binary factor analysis with automatic model selection

Binary Factor Analysis (BFA) uncovers the independent binary information sources from observations with wide applications. BFA learning hierarchically nests three levels of inverse problems, i.e., inference of binary code for each observation, parameter estimation and model selection. Under Bayesian Ying-Yang (BYY) framework, the first level becomes an intractable Binary Quadratic Programming (BQP) problem, while model selection can be conducted automatically during parameter learning. We conduct extensive experiments to reveal that the performance order of four BQP methods is reversed from making BQP optimization to making BYY automatic model selection, which implies that learning is not merely optimization. Moreover, the BFA learning algorithm is further developed with priors over parameters to improve the performance. Finally, based on BFA, we empirically compare BYY with Variational Bayes (VB) and Bayesian information criterion (BIC).

On the construction of the relevance vector machine based on Bayesian Ying-Yang harmony learning

Tipping’s relevance vector machine (RVM) applies kernel methods to construct basis function networks using a least number of relevant basis functions. Compared to the well-known support vector machine (SVM), the RVM provides a better sparsity, and an automatic estimation of hyperparameters. However, the performance of the original RVM purely depends on the smoothness of the presumed prior of the connection weights and parameters. Consequently, the sparsity is actually still controlled by the choice of kernel functions and/or kernel parameters. This may lead to severe underfitting or overfitting in some cases. In this research, we explicitly involve the number of basis functions into the objective of the optimization procedure, and construct the RVM by maximizing the harmony function between “hypothetical” probability distribution in the forward training pathway and “true” probability distribution in the backward testing pathway, using Xu’s Bayesian Ying-Yang (BYY) harmony learning technique. The experimental results have shown that our proposed methodology can achieve both the least complexity of structure and goodness of fit to data.

Image Segmentation with the EM and the BYY Learning

The aim of the study is to present an image segmentation method based on feature space clustering with the Gaussian Mixture Model (GMM) and the Expectation Maximization (EM) algorithm. To address the issue of selection of determining the number of clusters, the Bayesian Ying-Yang (BYY) learning is employed. Moreover, to improve the performance of the segmentation on noisy images, both intensity and spatial position information are employed as features to describe a pixel in the image. The simulation results on images with and without noises validate the performance of the proposed method.

Unsupervised Natural Image Segmentation via Bayesian Ying–Yang Harmony Learning Theory

An unsupervised image segmentation method for natural images is proposed in this paper. We assume that texture features in natural images are distributed as a mixture of Gaussians. In order to cluster the extracted feature vectors, we modify a clustering algorithm based on Bayesian Ying–Yang (BYY) harmony learning theory with Dirichlet-Normal-Wishart prior. This algorithm can determine the number of components automatically during the clustering procedure, as long as we give a large enough initial component number. Our works in this paper have presented a complete pipeline of clustering-based image segmentation including feature extraction, robust feature clustering and methodological effective post processing. The experiments reported in this paper demonstrate that the proposed method is efficient (in terms of visual evaluation and quantitative performance measures) and performs competitively compared to the existing state-of-the-art segmentation methods.

Clustering user behavior patterns of E-commerce search engine-based on mixture of Markov models

Clustering the behavior patterns of the customers is helpful to provide more specific services for E-commerce applications.A mixture model based on Markov models was proposed to solve this problem on the search engine of E-Commerce website.This model assumed that the behaviors of every customer who used the search engine can be represented by a Markov model and every user was assigned to a particular cluster randomly.Based on Bayesian Ying-Yang(BYY) harmony learning theory,a corresponding harmony function and an adaptive gradient algorithm were designed to deal with the parameter-learning and model-selection tasks.The experimental result shows that this adaptive gradient algorithm can achieve the model-selection and the parameter-learning more automatically and efficiently when compared with EM algorithm.At last,this clustering approach was applied on real-world click-through logs of the search engine on www.taobao.com and the result shows that this method can capture the nature of customers' behaviors effectively.

KCMAC-BYY: Kernel CMAC using Bayesian Ying–Yang learning

The Cerebellar Model Articulation Controller (CMAC) possesses attractive properties of fast learning and simple computation. In application, the size of its association vector is always reduced to economize the memory requirement, greatly constraining its modeling capability. The kernel CMAC (KMAC), which provides an interpretation for the traditional CMAC from the kernel viewpoint, not only strengthens the modeling capability without increasing its complexity, but reinforces its generalization with the help of a regularization term. However, the KCMAC suffers from the problem of selecting its hyperparameter. In this paper, the Bayesian Ying–Yang (BYY) learning theory is incorporated into KCMAC, referred to as KCMAC-BYY, to optimize the hyperparameter. The proposed KCMAC-BYY achieves the systematic tuning of the hyperparameter, further improving the performance in modeling and generalization. The experimental results on some benchmark datasets show the prior performance of the proposed KCMAC-BYY to the existing representative techniques.

Transcription network analysis by a sparse binary factor analysis algorithm.

Transcription factor activities (TFAs), rather than expression levels, control gene expression and provide valuable information for investigating TF-gene regulations. The underlying bimodal or switch-like patterns of TFAs may play important roles in gene regulation. Network Component Analysis (NCA) is a popular method to deduce TFAs and TF-gene control strengths from microarray data. However, it does not directly examine the bimodality of TFAs and it needs TF-gene connection topology a priori known. In this paper, we modify NCA to model gene expression regulation by Binary Factor Analysis (BFA), which directly captures switch-like patterns of TFAs. Moreover, sparse technique is employed on the mixing matrix of BFA, and thus the proposed sparse BYY-BFA algorithm, developed under Bayesian Ying-Yang (BYY) learning framework, can not only uncover the latent TFA profile’s switch-like patterns, but also be capable of automatically shutting off the unnecessary connections. Simulation study demonstrates the effectiveness of BYY-BFA, and a preliminary application to Saccharomyces cerevisiae cell cycle data and Escherichia coli carbon source transition data shows that the reconstructed binary patterns of TFAs by BYY-BFA are consistent with the ups and downs of TFAs by NCA, and that BYY-BFA also works well when the network topology is unknown.

Transcription Network Analysis by A Sparse Binary Factor Analysis Algorithm

Summary Transcription factor activities (TFAs), rather than expression levels, control gene expres- sion and provide valuable information for investigating TF-gene regulations. The underly- ing bimodal or switch-like patterns of TFAs may play important roles in gene regulation. Network Component Analysis (NCA) is a popular method to deduce TFAs and TF-gene control strengths from microarray data. However, it does not directly examine the bimodal- ity of TFAs and it needs the TF-gene connection topology to be a priori known. In this paper, we modify NCA to model gene expression regulation by Binary Factor Analysis (BFA), which directly captures switch-like patterns of TFAs. Moreover, sparse technique is employed on the mixing matrix of BFA, and thus the proposed sparse BYY-BFA al- gorithm, developed under Bayesian Ying-Yang (BYY) learning framework, can not only uncover the latent TFA profile’s switch-like patterns, but also be capable of automatically shutting off the unnecessary connections. Simulation study demonstrates the effectiveness of BYY-BFA, and a preliminary application to Saccharomyces cerevisiae cell cycle data and Escherichia coli carbon source transition data shows that the reconstructed binary pat- terns of TFAs by BYY-BFA are consistent with the ups and downs of TFAs by NCA, and that BYY-BFA also works well when the network topology is unknown.

On essential topics of BYY harmony learning: Current status, challenging issues, and gene analysis applications

As a supplementary of [Xu L. Front. Electr. Electron. Eng. China, 2010, 5(3): 281–328], this paper outlines current status of efforts made on Bayesian Ying-Yang (BYY) harmony learning, plus gene analysis applications. At the beginning, a bird’s-eye view is provided via Gaussian mixture in comparison with typical learning algorithms and model selection criteria. Particularly, semi-supervised learning is covered simply via choosing a scalar parameter. Then, essential topics and demanding issues about BYY system design and BYY harmony learning are systematically outlined, with a modern perspective on Yin-Yang viewpoint discussed, another Yang factorization addressed, and coordinations across and within Ying-Yang summarized. The BYY system acts as a unified framework to accommodate unsupervised, supervised, and semi-supervised learning all in one formulation, while the best harmony learning provides novelty and strength to automatic model selection. Also, mathematical formulation of harmony functional has been addressed as a unified scheme for measuring the proximity to be considered in a BYY system, and used as the best choice among others. Moreover, efforts are made on a number of learning tasks, including a mode-switching factor analysis proposed as a semi-blind learning framework for several types of independent factor analysis, a hidden Markov model (HMM) gated temporal factor analysis suggested for modeling piecewise stationary temporal dependence, and a two-level hierarchical Gaussian mixture extended to cover semi-supervised learning, as well as a manifold learning modified to facilitate automatic model selection. Finally, studies are applied to the problems of gene analysis, such as genome-wide association, exome sequencing analysis, and gene transcriptional regulation.

A binary matrix factorization algorithm for protein complex prediction

BackgroundIdentifying biologically relevant protein complexes from a large protein-protein interaction (PPI) network, is essential to understand the organization of biological systems. However, high-throughput experimental techniques that can produce a large amount of PPIs are known to yield non-negligible rates of false-positives and false-negatives, making the protein complexes difficult to be identified.ResultsWe propose a binary matrix factorization (BMF) algorithm under the Bayesian Ying-Yang (BYY) harmony learning, to detect protein complexes by clustering the proteins which share similar interactions through factorizing the binary adjacent matrix of a PPI network. The proposed BYY-BMF algorithm automatically determines the cluster number while this number is pre-given for most existing BMF algorithms. Also, BYY-BMF’s clustering results does not depend on any parameters or thresholds, unlike the Markov Cluster Algorithm (MCL) that relies on a so-called inflation parameter. On synthetic PPI networks, the predictions evaluated by the known annotated complexes indicate that BYY-BMF is more robust than MCL for most cases. On real PPI networks from the MIPS and DIP databases, BYY-BMF obtains a better balanced prediction accuracies than MCL and a spectral analysis method, while MCL has its own advantages, e.g., with good separation values.

Parameterizations make different model selections: Empirical findings from factor analysis

How parameterizations affect model selection performance is an issue that has been ignored or seldom studied since traditional model selection criteria, such as Akaike’s information criterion (AIC), Schwarz’s Bayesian information criterion (BIC), difference of negative log-likelihood (DNLL), etc., perform equivalently on different parameterizations that have equivalent likelihood functions. For factor analysis (FA), in addition to one traditional model (shortly denoted by FA-a), it was previously found that there is another parameterization (shortly denoted by FA-b) and the Bayesian Ying-Yang (BYY) harmony learning gets different model selection performances on FA-a and FA-b. This paper investigates a family of FA parameterizations that have equivalent likelihood functions, where each one (shortly denoted by FA-r) is featured by an integer r, with FA-a as one end that r = 0 and FA-b as the other end that r reaches its upper-bound. In addition to the BYY learning in comparison with AIC, BIC, and DNLL, we also implement variational Bayes (VB). Several empirical finds have been obtained via extensive experiments. First, both BYY and VB perform obviously better on FA-b than on FA-a, and this superiority of FA-b is reliable and robust. Second, both BYY and VB outperform AIC, BIC, and DNLL, while BYY further outperforms VB considerably, especially on FA-b. Moreover, with FA-a replaced by FA-b, the gain obtained by BYY is obviously higher than the one by VB, while the gain by VB is better than no gain by AIC, BIC, and DNLL. Third, this paper also demonstrates how each part of priors incrementally and jointly improves the performances, and further shows that using VB to optimize the hyperparameters of priors deteriorates the performances while using BYY for this purpose can further improve the performances.

Learning Gaussian mixture with automatic model selection: A comparative study on three Bayesian related approaches

Three Bayesian related approaches, namely, variational Bayesian (VB), minimum message length (MML) and Bayesian Ying-Yang (BYY) harmony learning, have been applied to automatically determining an appropriate number of components during learning Gaussian mixture model (GMM). This paper aims to provide a comparative investigation on these approaches with not only a Jeffreys prior but also a conjugate Dirichlet-Normal-Wishart (DNW) prior on GMM. In addition to adopting the existing algorithms either directly or with some modifications, the algorithm for VB with Jeffreys prior and the algorithm for BYY with DNW prior are developed in this paper to fill the missing gap. The performances of automatic model selection are evaluated through extensive experiments, with several empirical findings: 1) Considering priors merely on the mixing weights, each of three approaches makes biased mistakes, while considering priors on all the parameters of GMM makes each approach reduce its bias and also improve its performance. 2) As Jeffreys prior is replaced by the DNW prior, all the three approaches improve their performances. Moreover, Jeffreys prior makes MML slightly better than VB, while the DNW prior makes VB better than MML. 3) As the hyperparameters of DNW prior are further optimized by each of its own learning principle, BYY improves its performances while VB and MML deteriorate their performances when there are too many free hyper-parameters. Actually, VB and MML lack a good guide for optimizing the hyper-parameters of DNW prior. 4) BYY considerably outperforms both VB and MML for any type of priors and whether hyper-parameters are optimized. Being different from VB and MML that rely on appropriate priors to perform model selection, BYY does not highly depend on the type of priors. It has model selection ability even without priors and performs already very well with Jeffreys prior, and incrementally improves as Jeffreys prior is replaced by the DNW prior. Finally, all algorithms are applied on the Berkeley segmentation database of real world images. Again, BYY considerably outperforms both VB and MML, especially in detecting the objects of interest from a confusing background.

An investigation of several typical model selection criteria for detecting the number of signals

Based on the problem of detecting the number of signals, this paper provides a systematic empirical investigation on model selection performances of several classical criteria and recently developed methods (including Akaike’s information criterion (AIC), Schwarz’s Bayesian information criterion, Bozdogan’s consistent AIC, Hannan-Quinn information criterion, Minka’s (MK) principal component analysis (PCA) criterion, Kritchman & Nadler’s hypothesis tests (KN), Perry & Wolfe’s minimax rank estimation thresholding algorithm (MM), and Bayesian Ying-Yang (BYY) harmony learning), by varying signal-to-noise ratio (SNR) and training sample size N. A family of model selection indifference curves is defined by the contour lines of model selection accuracies, such that we can examine the joint effect of N and SNR rather than merely the effect of either of SNR and N with the other fixed as usually done in the literature. The indifference curves visually reveal that all methods demonstrate relative advantages obviously within a region of moderate N and SNR. Moreover, the importance of studying this region is also confirmed by an alternative reference criterion by maximizing the testing likelihood. It has been shown via extensive simulations that AIC and BYY harmony learning, as well as MK, KN, and MM, are relatively more robust than the others against decreasing N and SNR, and BYY is superior for a small sample size.

Optimization of fuzzy CMAC using evolutionary Bayesian Ying-Yang learning

Cerebellar model articulation controller (CMAC) is a popular associative memory neural network that imitates human’s cerebellum, which allows it to learn fast and carry out local generalization efficiently. This research aims to integrate evolutionary computation into fuzzy CMAC Bayesian Ying-Yang (FCMACBYY) learning, which is referred to as FCMAC-EBYY, to achieve a synergetic development in the search for optimal fuzzy sets and connection weights. Traditional evolutionary approaches are limited to small populations of short binary string length and as such are not suitable for neural network training, which involves a large searching space due to complex connections as well as real values. The methodology employed by FCMACEBYY is coevolution, in which a complex solution is decomposed into some pieces to be optimized in different populations/species and then assembled. The developed FCMAC-EBYY is compared with various neuro-fuzzy systems using a real application of traffic flow prediction.