Efficient Structure Learning Research Articles

Effective and efficient structure learning with pruning and model averaging strategies

Learning the structure of a Bayesian Network (BN) with score-based solutions involves exploring the search space of possible graphs and moving towards the graph that maximises a given objective function. Some algorithms offer exact solutions that guarantee to return the graph with the highest objective score, while others offer approximate solutions in exchange for reduced computational complexity. This paper describes an approximate BN structure learning algorithm, which we call Model Averaging Hill-Climbing (MAHC), that combines two novel strategies with hill-climbing search. The algorithm starts by pruning the search space of graphs, where the pruning strategy can be viewed as an aggressive version of the pruning strategies that are typically applied to combinatorial optimisation structure learning problems. It then performs model averaging in the hill-climbing search process and moves to the neighbouring graph that maximises the objective function, on average, for that neighbouring graph and over all its valid neighbouring graphs. Comparisons with other algorithms spanning different classes of learning suggest that the combination of aggressive pruning with model averaging is both effective and efficient, particularly in the presence of data noise.

Efficient Structured Learning for Personalized Diversification

This paper is concerned with the problem of personalized diversification of search results, with the goal of enhancing the performance of both plain diversification and plain personalization algorithms. In previous work, the problem has mainly been tackled by means of unsupervised learning. To further enhance the performance, we propose a supervised learning strategy. Specifically, we set up a structured learning framework for conducting supervised personalized diversification, in which we add features extracted directly from tokens of documents and those utilized by unsupervised personalized diversification algorithms, and, importantly, those generated from our proposed user-interest latent Dirichlet topic model. We also define two constraints in our structured learning framework to ensure that search results are both diversified and consistent with a user's interest. To further boost the efficiency of training, we propose a fast training framework for our proposed method by adding additional multiple highly violated but also diversified constraints at every training iteration of the cutting-plane algorithm. We conduct experiments on an open dataset and find that our supervised learning strategy outperforms unsupervised personalized diversification methods as well as other plain personalization and plain diversification methods. Our fast training framework significantly saves training time while it maintains almost the same performance.

Nonparametric Feature Matching Based Conditional Random Fields for Gesture Recognition from Multi-Modal Video.

We present a new gesture recognition method that is based on the conditional random field (CRF) model using multiple feature matching. Our approach solves the labeling problem, determining gesture categories and their temporal ranges at the same time. A generative probabilistic model is formalized and probability densities are nonparametrically estimated by matching input features with a training dataset. In addition to the conventional skeletal joint-based features, the appearance information near the active hand in an RGB image is exploited to capture the detailed motion of fingers. The estimated likelihood function is then used as the unary term for our CRF model. The smoothness term is also incorporated to enforce the temporal coherence of our solution. Frame-wise recognition results can then be obtained by applying an efficient dynamic programming technique. To estimate the parameters of the proposed CRF model, we incorporate the structured support vector machine (SSVM) framework that can perform efficient structured learning by using large-scale datasets. Experimental results demonstrate that our method provides effective gesture recognition results for challenging real gesture datasets. By scoring 0.8563 in the mean Jaccard index, our method has obtained the state-of-the-art results for the gesture recognition track of the 2014 ChaLearn Looking at People (LAP) Challenge.

A diversity-driven structure learning algorithm for building hierarchical neuro-fuzzy classifiers

In this article, an efficient structure learning algorithm is proposed for the development of self-organizing neuro-fuzzy multilayered classifiers (SONeFMUC). These classifiers are hierarchical structures comprising small-scale fuzzy-neuron classifiers (FNCs), interconnected along multiple layers. At each layer, parent FNCs are combined to construct a descendant FNC at the next layer with enhanced classification qualities. The SONeFMUC structure is progressively expanded by generating new layers based on the principles of the Group Method of Data Handling (GMDH) algorithm, which is appropriately adapted to handle classification tasks. Traditional GMDH proceeds blindly to the construction of all possible parent FNC pairs from the previous layer to obtain the individuals in the next layer without paying due attention to the diversity of the FNC combinations. However, previous experimentation shows that a large number of descendant FNCs exhibit similar or slightly better classification capabilities than their parent FNCs. This causes an increase of the computational cost required for structure learning, without a direct impact on the accuracy of the obtained models. In this paper, a modified version of GMDH is devised for effective identification of the SONeFMUC structure. We incorporate the Proportion of Specific Agreement (Ps) as a means to evaluate the diversity of the FNC pairs. In the devised method, only complementary FNCs are combined, i.e., FNCs which commit errors at different pattern subspaces, to construct a descendant FNC at the next layer. Accordingly, a computational reduction is achieved while high classification accuracy is maintained. The efficiency of the proposed structure learning is tested on a diverse set of benchmark datasets using land cover classification from multispectral images as a real-world application.

Efficient Structure Learning of Bayesian Networks using Constraints

This paper addresses the problem of learning Bayesian network structures from data based on score functions that are decomposable. It describes properties that strongly reduce the time and memory costs of many known methods without losing global optimality guarantees. These properties are derived for different score criteria such as Minimum Description Length (or Bayesian Information Criterion), Akaike Information Criterion and Bayesian Dirichlet Criterion. Then a branch-and-bound algorithm is presented that integrates structural constraints with data in a way to guarantee global optimality. As an example, structural constraints are used to map the problem of structure learning in Dynamic Bayesian networks into a corresponding augmented Bayesian network. Finally, we show empirically the benefits of using the properties with state-of-the-art methods and with the new algorithm, which is able to handle larger data sets than before.

A hybrid random field model for scalable statistical learning

This paper introduces hybrid random fields, which are a class of probabilistic graphical models aimed at allowing for efficient structure learning in high-dimensional domains. Hybrid random fields, along with the learning algorithm we develop for them, are especially useful as a pseudo-likelihood estimation technique (rather than a technique for estimating strict joint probability distributions). In order to assess the generality of the proposed model, we prove that the class of pseudo-likelihood distributions representable by hybrid random fields strictly includes the class of joint probability distributions representable by Bayesian networks. Once we establish this result, we develop a scalable algorithm for learning the structure of hybrid random fields, which we call 'Markov Blanket Merging'. On the one hand, we characterize some complexity properties of Markov Blanket Merging both from a theoretical and from the experimental point of view, using a series of synthetic benchmarks. On the other hand, we evaluate the accuracy of hybrid random fields (as learned via Markov Blanket Merging) by comparing them to various alternative statistical models in a number of pattern classification and link-prediction applications. As the results show, learning hybrid random fields by the Markov Blanket Merging algorithm not only reduces significantly the computational cost of structure learning with respect to several considered alternatives, but it also leads to models that are highly accurate as compared to the alternative ones.

Problem solving through imitation

This paper presents an approach to problem solving through imitation. It introduces the Statistical and Temporal Percept Action Coupling (ST-PAC) System which statistically models the dependency between the perceptual state of the world and the resulting actions that this state should elicit. The ST-PAC system stores a sparse set of experiences provided by a teacher. These memories are stored to allow efficient recall and generalisation over novel systems states. Random exploration is also used as a fall-back “brute-force” mechanism should a recalled experience fail to solve a scenario. Statistical models are used to couple groups of percepts with similar actions and incremental learning used to incorporate new experiences into the system. The system is demonstrated within the problem domain of a children’s shape sorter puzzle. The ST-PAC system provides an emergent architecture where competence is implicitly encoded within the system. In order to train and evaluate such emergent architectures, the concept of the Complexity Chain is proposed. The Complexity Chain allows efficient structured learning in a similar fashion to that used in biological system and can also be used as a method for evaluating a cognitive system’s performance. Tests demonstrating the Complexity Chain in learning are shown in both simulated and live environments. Experimental results show that the proposed methods allowed for good generalisation and concept refinement from an initial set of sparse examples provided by a tutor.