Connectionist Systems Research Articles

A self-watching model of analogy-making and perception

This paper describes Metacat, an extension of the Copycat model of analogy-making. The development of Copycat focused on modelling context-sensitive concepts and the ways in which they interact with perception within an abstract microworld of analogy problems. This approach differs from most other models of analogy in its insistence that concepts acquire their semantics from within the system itself, through perception, rather than being imposed from the outside. The present work extends these ideas by incorporating self-perception, episodic memory, and reminding into the model. These mechanisms enable Metacat to explain the similarities and differences that it perceives between analogies, and to monitor and respond to patterns that occur in its own behaviour as it works on analogy problems. This introspective capacity overcomes several limitations inherent in the earlier model, and affords the program a powerful degree of self-control. Metacat's architecture includes aspects of both symbolic and connectionist systems. The paper outlines the principal components of the architecture, analyses several sample runs and examples of program-generated commentary about analogies, and discusses Metacat's relation to some other well-known models of analogy.

On the Integration of Connectionist and Logic-Based Systems

It is a long-standing and important problem to integrate logic-based systems and connectionist systems. In brief, this problem is concerned with how each of these two paradigms interacts with the other and how each complements the other: how one may give a logical interpretation of neural networks, how one may interpret connectionism within a logical framework, and how one may combine the advantages of each within a single integrated system. In this paper, the computation and approximate computation by neural networks of semantic operators TP determined by logic programs P is studied; the converse of this problem, namely, the extraction of logic programs from given neural networks is also briefly considered. The foundations of the relevant notions employed in this problem are revisited and clarified and new definitions are presented which avoid embedding spaces of interpretations in the real line. In particular, such definitions are formulated relating to (1) pointwise and uniform approximation of TP, and (2) approximation and computation of (least) fixed points of TP. There are related notions of approximation and convergence of neural networks, and related notions of approximation and convergence of programs and these are discussed briefly, although the focus here is on (1) and (2). Necessary and sufficient conditions for uniform approximation of TP by neural networks are given in terms of continuity. Finally, the class of programs for which these methods can be employed to compute fixed points is greatly extended from the rather small class of acyclic programs to the (computationally adequate) class of all definite programs.

TWNFI — a transductive neuro-fuzzy inference system with weighted data normalization for personalized modeling

This paper introduces a novel transductive neuro-fuzzy inference model with weighted data normalization (TWNFI). In transductive systems a local model is developed for every new input vector, based on a certain number of data that are selected from the training data set and the closest to this vector. The weighted data normalization method (WDN) optimizes the data normalization ranges of the input variables for the model. A steepest descent algorithm is used for training the TWNFI models. The TWNFI is compared with some other widely used connectionist systems on two case study problems: Mackey-Glass time series prediction and a real medical decision support problem of estimating the level of renal function of a patient. The TWNFI method not only results in a "personalized" model with a better accuracy of prediction for a single new sample, but also depicts the most significant input variables (features) for the model that may be used for a personalized medicine.

A Connectionist Computational Model for Epistemic and Temporal Reasoning

The importance of the efforts to bridge the gap between the connectionist and symbolic paradigms of artificial intelligence has been widely recognized. The merging of theory (background knowledge) and data learning (learning from examples) into neural-symbolic systems has indicated that such a learning system is more effective than purely symbolic or purely connectionist systems. Until recently, however, neural-symbolic systems were not able to fully represent, reason, and learn expressive languages other than classical propositional and fragments of first-order logic. In this article, we show that nonclassical logics, in particular propositional temporal logic and combinations of temporal and epistemic (modal) reasoning, can be effectively computed by artificial neural networks. We present the language of a connectionist temporal logic of knowledge (CTLK). We then present a temporal algorithm that translates CTLK theories into ensembles of neural networks and prove that the translation is correct. Finally, we apply CTLK to the muddy children puzzle, which has been widely used as a test-bed for distributed knowledge representation. We provide a complete solution to the puzzle with the use of simple neural networks, capable of reasoning about knowledge evolution in time and of knowledge acquisition through learning.

Cognitive vision: The case for embodied perception

This paper considers arguments for the necessity of embodiment in cognitive vision systems. We begin by delineating the scope of cognitive vision, and follow this by a survey of the various approaches that can be taken to the realization of artificial cognitive vision systems, focussing on cognitive aspects. These range from the cognitivist symbolic representational paradigm, through connectionist systems and self-organizing dynamical systems, to the enactive cognition paradigm. We then consider various arguments for embodiment, beginning with paradigm-specific cases, and concluding with a paradigm-independent argument for embodied perception and cognition. We explore briefly different forms of embodiment and their relevance to the foregoing viewpoints. We highlight some of the key problems associated with embodied cognitive vision, including the phylogeny/ontogeny trade-off in artificial systems and the developmental limitations imposed by real-time environmental coupling. Finally, we conclude by considering some aspects of natural cognitive systems to see how they can provide insights to help in addressing these problems.

Feature Extraction and Connectionist Classification of SODAR Echograms

Sonic detection and ranging (SODAR) systems are efficient and economical tool to probe the lower planetary boundary layer on a continuous basis. The lower atmospheric patterns (each depicting a different atmospheric condition) recorded by this system can prove to be extremely useful if classified and interpreted correctly. The manual identification of these SODAR patterns is a laborious task and requires considerable expertise. A connectionist system has already been developed by the authors to automate the process to some extent. In this letter, we enhance its generalization of performance, by incorporating feature extraction using the fast Fourier transform. The results are compared with that in earlier work to demonstrate its effectiveness.

NFI: a neuro-fuzzy inference method for transductive reasoning

This paper introduces a novel neural fuzzy inference method-NFI for transductive reasoning systems. NFI develops further some ideas from DENFIS-dynamic neuro-fuzzy inference systems for both online and offline time series prediction tasks. While inductive reasoning is concerned with the development of a model (a function) to approximate data in the whole problem space (induction), and consecutively-using this model to predict output values for a new input vector (deduction), in transductive reasoning systems a local model is developed for every new input vector, based on some closest to this vector data from an existing database (also generated from an existing model). NFI is compared with both inductive connectionist systems (e.g., MLP, DENFIS) and transductive reasoning systems (e.g., K-NN) on three case study prediction/identification problems. The first one is a prediction task on Mackey Glass time series; the second one is a classification on Iris data; and the last one is a real medical decision support problem of estimating the level of renal function of a patient, based on measured clinical parameters for the purpose of their personalised treatment. The case studies have demonstrated better accuracy obtained with the use of the NFI transductive reasoning in comparison with the inductive reasoning systems.

AN INTEGRATED FEATURE SELECTION AND CLASSIFICATION METHOD TO SELECT MINIMUM NUMBER OF VARIABLES ON THE CASE STUDY OF GENE EXPRESSION DATA

This paper introduces a novel generic approach for classification problems with the objective of achieving maximum classification accuracy with minimum number of features selected. The method is illustrated with several case studies of gene expression data. Our approach integrates filter and wrapper gene selection methods with an added objective of selecting a small set of non-redundant genes that are most relevant for classification with the provision of bins for genes to be swapped in the search for their biological relevance. It is capable of selecting relatively few marker genes while giving comparable or better leave-one-out cross-validation accuracy when compared with gene ranking selection approaches. Additionally, gene profiles can be extracted from the evolving connectionist system, which provides a set of rules that can be further developed into expert systems. The approach uses an integration of Pearson correlation coefficient and signal-to-noise ratio methods with an adaptive evolving classifier applied through the leave-one-out method for validation. Datasets of gene expression from four case studies are used to illustrate the method. The results show the proposed approach leads to an improved feature selection process in terms of reducing the number of variables required and an increased in classification accuracy.

Adaptive anomaly detection with evolving connectionist systems

Anomaly detection holds great potential for detecting previously unknown attacks. In order to be effective in a practical environment, anomaly detection systems have to be capable of online learning and handling concept drift. In this paper, a new adaptive anomaly detection framework, based on the use of unsupervised evolving connectionist systems, is proposed to address these issues. It is designed to adapt to normal behavior changes while still recognizing anomalies. The evolving connectionist systems learn a subject's behavior in an online, adaptive fashion through efficient local element tuning. Experiments with the KDD Cup 1999 network data and the Windows NT user profiling data show that our adaptive anomaly detection systems, based on Fuzzy Adaptive Resonance Theory (ART) and Evolving Fuzzy Neural Networks (EFuNN), can significantly reduce the false alarm rate while the attack detection rate remains high.

Evolving connectionist system versus algebraic formulas for prediction of renal function from serum creatinine

In clinical trials, equation 7 from the Modification of Diet in Renal Disease (MDRD) Study is the most accurate formula for the prediction of glomerular filtration rate (GFR) from serum creatinine. An alternative approach has been developed using evolving connectionist systems (ECOS), which are novel computing structures that can be trained to generate accurate output from a given set of input variables. This study aims to compare the prediction errors associated with each method, using data that reproduce routine clinical practice as opposed to the artificial setting of clinical trials. The methods were compared using 441 radioisotope measurements of GFR in 178 chronic kidney disease patients from 12 centers in Australia and New Zealand. All clinical and laboratory measurements were obtained from the patients' center rather than central laboratories, as would be the case in routine clinical practice. Both the MDRD formula and ECOS used the same predictive variables, and both were optimized to the study cohort by stepwise regression and training, respectively. Mean measured GFR in the cohort was 22.6 mL/min/1.73 m(2). The bias and precision of the MDRD formula were -3.5 mL/min/1.73 m(2) and 34.5%, respectively, improving to -1.2 mL/min/1.73 m(2) and 31.1% after maximal optimization of the formula to study data. The bias and precision of the ECOS were 0.7 mL/min/1.73 m(2) and 32.6%, respectively, improving to -0.1 mL/min/1.73 m(2) and 16.6% after maximal optimization of the system to study data. The prediction of GFR using ECOS was improved by accounting for the center from where clinical and laboratory measurements originated within the connectionist model. Algebraic formulas will be associated with greater prediction error in routine clinical practice than in the original trials, and machine intelligence is more likely to predict GFR accurately in this setting.

Adaptation and interaction in dynamical systems: Modelling and rule discovery through evolving connectionist systems

The paper presents a methodology for adaptive modelling and discovery of dynamic relationship rules from continuous data streams. In dynamic processes, underlying rules may change over time and tracing these changes is a difficult task for computer modelling. Evolving fuzzy neural networks (EFuNN) are used for this purpose here. EFuNNs belong to the group of evolving connectionist systems (ECOS). These are information systems that learn from data in a supervised mode through on-line adaptive clustering and allow for rule extraction, each rule representing input-output relationship within a cluster of data. Extracted rules, after each consecutive chunk of data is entered into the system, are compared in order to discover new patterns of interaction between input and output variables. Thus the stability and plasticity of the investigated process are evaluated. The rules are also used for the prediction of future events. To illustrate the methodology, a mathematical example is used, along with two real case studies. The first case study is from Macroeconomics and the second one is from Bioinformatics.

Evolving Connectionist Systems: Methods and Applications in Bioinformatics, Brain Study and Intelligent Machines

Prologue.- Part I: Evolving Connectionist Systems: Methods and Techniques.- Evolving Processes and Evolving Connectionist Systems.- Evolving Connectionist Systems for Unsupervised Learning.- Evolving Connectionist Systems for Supervised Learning.- Recurrent Evolving Systems, Reinforcement Learning and Evolving Automata.- Evolving Neuro-Fuzzy Inference Systems.- Evolutionary Computation and Evolving Connectionist Systems.- Evolving Connectionist Machines: A Framework, Biological Motivation.- and Implementation Issues.- Part II: Evolving connectionist systems: Applications in Bioinformatics, Brain Study and Intelligent Systems.- Data Analysis, Modelling and Knowledge Discovery in Bioinformatics.- Dynamic Modelling of Brain Functions and Cognitive Processes.- Modelling the Emergence of Acoustic Segments (Phonemes) in Spoken Languages.- On-Line Adaptive Speech Recognition.- On-Line Image and Video Data Processing.- Evolving Systems for Integrated Multi-Modal Information Processing.- Epilogue.- References.- Extended Glossary.- Subject Index.

EVOLUTIONARY COMPUTATION FOR ON-LINE AND OFF-LINE PARAMETER TUNING OF EVOLVING FUZZY NEURAL NETWORKS

This work applies Evolutionary Computation to achieve completely self-adapting Evolving Fuzzy Neural Networks (EFuNNs) for operating in both incremental (on-line) and batch (off-line) modes. EFuNNs belong to a class of Evolving Connectionist Systems (ECOS), capable of performing clustering-based, on-line, local area learning and rule extraction. Through Evolutionary Computation, its parameters such as learning rates and membership functions are continuously adjusted to reflect the changes in the dynamics of incoming data. The proposed methods are tested on the Mackey–Glass series and the results demonstrate a substantial improvement in EFuNN's performance.

Logic programs and connectionist networks

One facet of the question of integration of Logic and Connectionist Systems, and how these can complement each other, concerns the points of contact, in terms of semantics, between neural networks and logic programs. In this paper, we show that certain semantic operators for propositional logic programs can be computed by feedforward connectionist networks, and that the same semantic operators for first-order normal logic programs can be approximated by feedforward connectionist networks. Turning the networks into recurrent ones allows one also to approximate the models associated with the semantic operators. Our methods depend on a well-known theorem of Funahashi, and necessitate the study of when Funahashi's theorem can be applied, and also the study of what means of approximation are appropriate and significant.

An evolutionary system for neural logic networks using genetic programming and indirect encoding

Nowadays, intelligent connectionist systems such as artificial neural networks have been proved very powerful in a wide area of applications. Consequently, the ability to interpret their structure was always a desirable feature for experts. In this field, the neural logic networks (NLN) by their definition are able to represent complex human logic and provide knowledge discovery. However, under contemporary methodologies, the training of these networks may often result in non-comprehensible or poorly designed structures. In this work, we propose an evolutionary system that uses current advances in genetic programming that overcome these drawbacks and produces neural logic networks that can be arbitrarily connected and are easily interpretable into expert rules. To accomplish this task, we guide the genetic programming process using a context-free grammar and we encode indirectly the neural logic networks into the genetic programming individuals. We test the proposed system in two problems of medical diagnosis. Our results are examined both in terms of the solution interpretability that can lead in knowledge discovery, and in terms of the achieved accuracy. We draw conclusions about the effectiveness of the system and we propose further research directions.

Logic programs, iterated function systems, and recurrent radial basis function networks

Graphs of the single-step operator for first-order logic programs—displayed in the real plane—exhibit self-similar structures known from topological dynamics, i.e., they appear to be fractals, or more precisely, attractors of iterated function systems. We show that this observation can be made mathematically precise. In particular, we give conditions which ensure that those graphs coincide with attractors of suitably chosen iterated function systems, and conditions which allow the approximation of such graphs by iterated function systems or by fractal interpolation. Since iterated function systems can easily be encoded using recurrent radial basis function networks, we eventually obtain connectionist systems which approximate logic programs in the presence of function symbols.

A Connectionist Approach to SODAR Pattern Classification

SODAR (or acoustic radar) systems are a useful tool to efficiently probe the lower planetary boundary layer (LPBL). The observations obtained by these systems can prove to be extremely useful if classified and interpreted correctly. The manual identification of different types of SODAR-recorded lower atmospheric microstructures is a laborious task and can be performed only by an expert having wide experience with the system and the variety of observations recorded by it. In this letter, we have developed a connectionist system to classify or identify SODAR patterns. The results demonstrate that the multilayer perceptron-based model is capable of successfully identifying the different SODAR patterns.

A Low-Frequency Counter Using Time-Discriminant Connectionist Systems

A new simple, accurate, and instantaneous low-frequency counter based on the time-discriminant connectionist system , is proposed. The new counter could be used as a general-purpose frequency counter to measure a wide range of low frequencies. The accuracy of the proposed counter can be increased by increasing the number and the size of the EPROMS used and by increasing the clock frequency. Measurements from a prototype system agree well with the analysis.

An Ontology Engineering Approach For Knowledge Discovery From Data In Evolving Domains

Knowledge discovery in evolving domains presents several challenges in information extraction and knowledge acquisition from heterogeneous, distributed, dynamic data sources. We define an evolving process if the process is developing, changing over time in a continuous manner. Examples of such domains include biological sciences, medical sciences, and social sciences, among others. This paper describes research in progress on a new methodology for leveraging the semantic content of ontologies to improve knowledge discovery in complex and dynamical domains. We consider in this initial stage the problem of how to acquire previous knowledge from data and then use this information in the context of ontology engineering. The first part of this paper concerns some aspects that help to understand the differences and similarities between ontologies and data models , followed by an analysis of some of the methods and ongoing researches in the process of building ontology from databases in evolving domains, or ontology learning from databases. In the second part we describe our approach to build a framework able to enhance ontology learning and discovery from data and present future directions of our research integrating ontology and evolving connectionist systems that is being developed in the Knowledge Engineering & Discovery Research Institute -Kedri.

Adaptive speech recognition with evolving connectionist systems

The paper presents a novel approach towards building adaptive speech recognition systems based on the evolving connectionist systems paradigm (ECoS). The simple evolving connectionist systems are the minimalist implementation of the ECoS. They can accommodate new input data and new classes through local element tuning. New connections and neurons are created during the adaptive learning process of the system. Experiments are conducted to illustrate this concept. It is demonstrated that a system can adapt to new speakers data and add new output classes on-line, e.g. new words, added at any time of its operation without having to rebuild the network from “scratch”. The system is robust to forgetting when new words are added.