Connectionist Networks Research Articles

The nonlinear dynamics of connectionist networks: The basis of motor control

The nonlinear dynamics of connectionist networks: The basis of motor control

Forecasting the behavior of multivariate time series using neural networks

This paper presents a neural network approach to multivariate time-series analysis. Real world observations of flour prices in three cities have been used as a benchmark in our experiments. Feedforward connectionist networks have been designed to model flour prices over the period from August 1972 to November 1980 for the cities of Buffalo, Minneapolis, and Kansas City. Remarkable success has been achieved in training the networks to learn the price curve for each of these cities and in making accurate price predictions. Our results show that the neural network approach is a leading contender with the statistical modeling approaches.

Of Shape Shifters and Theories

Hanson, S. J., & Burr, D. J. (1990). What connectionist models learn: Learning and representation in connectionist networks. Behavioral and Brain Sciences, 13, 529-576. Hintzman, D. L. (in press). 25 years of learning and memory: Was the cognitive revolution a mistake? In D. E. Myers & S. Kornblum (Eds.), Attention and performance XIV. Hillsdale, NJ: Lawrence Erlbaum Associates, Inc. Jones, E. E. (1985). Major developments in social psychology during the past five decades. In G. Lindzey & E. Aronson (Eds.), The handbook of social psychology (3rd ed., pp. 47-107). New York: Random House. Kamin, L. J. (1968). Attention-like processes in classical conditioning. In M. R. Jones (Ed.), Miami Symposium on the Prediction of Behavior: Aversive stimulation (pp. 9-33). Miami: University of Miami Press. Lieberman, D. A. (1990). Learning, behavior, and cognition. Belmont, CA: Wadsworth. Lord, C. G. (1989). The disappearance of dissonance in an age of relativism. Personality and Social Psychology Bulletin, 15, 513-518. Lord, C. G., Ross, L., & Lepper, M. R. (1979). Biased assimilation and attitude polarization: The effects of prior theories on subsequently considered evidence. Journal of Personality and Social Psychology, 37, 2098-2109. McLaren, P. L., Kaye, H., & Macintosh, N. J. (1990). An associative theory of the representation of stimuli: Applications to perceptual learning and latent inhibition. In R. G. M. Morris (Ed.), Parallel distributed processing: Implications for psychology and neurobiology (pp. 102-130). New York: Oxford University Press. Rescorla, R. A., & Wagner, A. R. (1972). A theory of Pavlovian conditioning: Variations in the effectiveness of reinforcement and nonreinforcement. In A. H. Black & W. F. Prokasy (Eds.), Classical conditioning: II. Current research and theory (pp. 64-99). New York: Appleton-Century-Crofts. Rumelhart, D. E., & McClelland, J. L. (1986). Parallel distributed processing (Vol. 1). Cambridge, MA: MIT Press. Schlenker, B. R. (1982). Translating actions into attitudes: An identityanalytic approach to the explanation of social conduct. In L. Berkowitz (Ed.), Advances in experimental social psychology (Vol. 15, pp. 193247). New York; Academic. Shanks, D. R. (1990). Connectionism and the learning of probabilistic concepts. Quarterly Journal of Experimental Psychology, 42A, 209237. Zanna, M. P., & Cooper, J. (1974). Dissonance and the pill: An attributional approach to studying the arousal properties of dissonance. Journal of Personality and Social Psychology, 29, 703-709.

Structured networks for adaptive language acquisition

In this research on adaptive language acquisition, connectionist networks are explored that map input messages into machine actions. In this work, the capability of such networks to learn associations between messages and meaningful responses to them as the set of machine actions increases in size and complexity is investigated. Specifically, how to reflect task structure in a network architecture is considered in order to provide improved generalization capability. A method for constructing networks from component subnetworks, namely a productnetwork, provides improved generalization by factoring the associations between words and action through an intermediate layer of semanticprimitives. A two-dimensional product network was evaluated in a 1000-action data retrieval system, the object of which is to answer questions about 20 attributes of the 50 states of the US. The system was tested by 13 subjects over a 2-week period, during which over 1000 natural language dialogues were recorded. The experiment was conducted using typed input with unconstrained vocabulary and syntax. During the course of performing its task, the system acquired over 500 words and retained 92% of what it learned. The results of this experiment demonstrate the scalability of previous results obtained on smaller, less complex tasks [Gorin etal., Comput. Speech Lang. 5, 101–132 (1991)] [Miller and Gorin, Proc. ICASSP 201–204 (1992)].

On-line learning without forgetting for language acquisition

In this paper, a new method for on-linetraining that overcomes interference in neural networks is presented. The language acquisition problem is modeled as a system of linear equations as first given by Gorin etal. [A. Gorin etal., ‘‘Adaptive Acquisition of Language,’’ in NeuralNetworks, TheoryandApplications, edited by R. J. Mammone and Y. Y. Zeevi (Academic, New York), p. 125]. The new method encodes data patterns or words that have been learned previously as a set of convex constraints. On-line training is then implemented using the row action projection (RAP) algorithm to readjust the weights subject to these convex constraints. The RAP algorithm asymptotically converges to the pseudoinverse solution in the case of simple linear constraints. The computational complexity of conventional methods of calculating the pseudoinverse make them unsuitable for on-line applications. The previously learned patterns are viewed as long-term memory while the on-line data is considered to reside in short-term memory. Memory retention is obtained in the new approach by storing the patterns that are close to the feature space boundaries in long-term memory. These stored patterns are then used as convex constraints for any subsequent on-line learning. In contrast to previous block learning strategies, the new approach learns the meaning of words sequentially. This enables the on-line training of a connectionist network. Computer simulations are presented that demonstrate the memory retention capabilities of the new on-line training algorithm.

A connectionist approach for peak detection in Hough space

Peak detection in multidimensional parameter space is an important aspect of feature detection using the Hough transform. In this paper a connectionist network is presented for detecting peaks in multidimensional Hough space. The neural network implementation successfully uses circumstantial evidence and detects multiple winners over the parameter space such that these winners correspond to parameters of features in the image. The dynamics of the network has been analysed and the conditions for convergence have been established. Experimental results obtained by applying the network for detecting straight lines in three images are presented. The results obtained are promising.

A connectionist expert system for fault diagnosis

The design of a connectionist expert system for the fault diagnosis of a distribution system is proposed. The heuristic rules acquired from distribution dispatchers' experiences are incorporated into the connectionist network which can be utilized as the expert system knowledge base. The data sets used to train the connectionist network are the current waveforms recorded upon the occurrence of miscellaneous faults. The interconnection weights of this network are adjusted by the back-propagation learning algorithm. The back-propagation connectionist network can then be used as an expert system for diagnosing distribution faults. A fault is identified and located by recognizing its associated current waveform. The connectionist expert system is built on a 32-bit personal computer. It is tested for diagnosing faults on a real distribution system. From the experimental results, it is found that the connectionist expert system has the following advantages: it is able to perform on-line diagnoses in a very short operation time of less than 1 second, and it makes inferences about fault types and fault locations from the locally recorded current waveforms only. This system is applicable to dispatching centres. It may assist dispatchers in judging emergency situations, which is the first step in the restoration procedure.

Connectionist generalization for production: An example from GridFont

Generalization for production is a difficult problem solved by intelligent systems. One such problem might be “paint a portrait of George Bush in the style of Vincent van Gogh.” In these tasks, a low dimensional representation (“George Bush” and “style of van Gogh”) is expanded into a (nonunique) output of extremely high dimension, for instance represented by color, line, shading, etc. We designed a connectionist network for generalization of production in such a way—to generate letterforms in a new font given just a few exemplars from that font. During learning, our network constructs distributed internal representations of different fonts and letters, even though each training instance had both font characteristics and letter characteristics. Separate hidden representations for “letter” and “font” were necessary for success of the network. The integration of information from font and letter representations had to be at a proper, intermediate level of abstraction. The limitations of the network can be attributed, in part, to limited training corpus, and lack of translation and scale invariances.

Practical issues in temporal difference learning

This paper examines whether temporal difference methods for training connectionist networks, such as Sutton's TD(λ) algorithm, can be successfully applied to complex real-world problems. A number of important practical issues are identified and discussed from a general theoretical perspective. These practical issues are then examined in the context of a case study in which TD(λ) is applied to learning the game of backgammon from the outcome of self-play. This is apparently the first application of this algorithm to a complex non-trivial task. It is found that, with zero knowledge built in, the network is able to learn from scratch to play the entire game at a fairly strong intermediate level of performance, which is clearly better than conventional commercial programs, and which in fact surpasses comparable networks trained on a massive human expert data set. This indicates that TD learning may work better in practice than one would expect based on current theory, and it suggests that further analysis of TD methods, as well as applications in other complex domains, may be worth investigating.

Simple statistical gradient-following algorithms for connectionist reinforcement learning

This article presents a general class of associative reinforcement learning algorithms for connectionist networks containing stochastic units. These algorithms, called REINFORCE algorithms, are shown to make weight adjustments in a direction that lies along the gradient of expected reinforcement in both immediate-reinforcement tasks and certain limited forms of delayed-reinforcement tasks, and they do this without explicitly computing gradient estimates or even storing information from which such estimates could be computed. Specific examples of such algorithms are presented, some of which bear a close relationship to certain existing algorithms while others are novel but potentially interesting in their own right. Also given are results that show how such algorithms can be naturally integrated with backpropagation. We close with a brief discussion of a number of additional issues surrounding the use of such algorithms, including what is known about their limiting behaviors as well as further considerations that might be used to help develop similar but potentially more powerful reinforcement learning algorithms.

Lexical and pragmatic disambiguation and re-interpretation in connectionist networks

Lexical and pragmatic ambiguity is a major source of uncertainty in natural language understanding. Symbolic models can make high-level inferences necessary for understanding text, but handle ambiguity poorly, especially when later context requires a re-interpretation of the input. Structured connectionist networks, on the other hand, can use their graded levels of activation to perform lexical disambiguation, but have trouble performing the variable bindings and inferencing necessary for language understanding. We have previously described a structured connectionist model, ROBIN, which overcomes many of these problems and allows the massively-parallel application of a large class of general knowledge rules. This paper describes how ROBIN uses these abilities and the contextual evidence from its semantic networks to disambiguate words and infer the most plausible plan/goal analysis of the input, while using the same mechanism to smoothly re-interpret the input if later context makes an alternative interpretation more likely. We present several experiments illustrating these abilities and comparing them to those of other connectionist models, and discuss several directions in which we are extending the model.

Models of central pattern generators as oscillators: the lamprey locomotor CPG

Locomotion in the lamprey is characterised by rhythmic bursts of muscle activity which alternate left and right and exhibit a rostral-to-caudal delay. This paper will describe several ways in which modelling has been applied to the study of the central pattern generator (CPG) for this system. A simple network model of the basic rhythm generator has been investigated using both a large biophysical network simulation and a simple connectionist algorithm. Intersegmental coordination has been studied using mathematical analysis of the lamprey spinal cord as a chain of coupled nonlinear oscillators. The correlation between these two levels of modelling has been investigated by applying the mathematical analysis to a simulated connectionist network. The contributions each kind of modelling makes to our understanding of the locomotor CPG will be discussed.

Semi-distributed Representations and Catastrophic Forgetting in Connectionist Networks

Abstract A major problem with connectionist networks is that newly-learned information may completely destroy previously-learned information unless the network is continually retrained on the old information. This phenomenon, known as catastrophic forgetting, is unacceptable both for practical purposes and as a model of mind. This paper advances the claim that catastrophic forgetting is in part the result of the overlap of system's distributed representations and can be reduced by reducing this overlap. A simple algorithm, called activation sharpening, is presented that allows a standard feed-forward backpropagation network to develop semi-distributed representations, thereby reducing the problem of catastrophic forgetting. Activation sharpening is discussed in tight of recent work done by other researchers who have experimented with this and other techniques for reducing catastrophic forgetting.

On Variable Binding in Connectionist Networks

This paper deals with the problem of variable binding in connectionist networks. Specifically, a more thorough solution to the variable binding problem based on the Discrete Neuron formalism is proposed and a number of issues arising in the solution are examined in relation to logic: consistency checking, binding generation, unification and functions. We analyze what is needed in order to resolve these issues and, based on this analysis, a procedure is developed for systematically setting up connectionist networks for variable binding based on logic rules. This solution compares favorably to similar solutions in simplicity and completeness.

SELECTIVE ATTENTION TO PERCEPTUAL GROUPS: THE PHASE TRACKING MECHANISM

We introduce a new mechanism of selective attention among perceptual groups as part of a computational model of early vision. In this model, selection of objects is a two-stage process: perceptual grouping is first performed in connectionist networks which dynamically bind together the neural activities triggered in response to related features in the image; secondly, by locking its output on the quasi-periodic bursts of activity associated with a single perceptual group, a system called the phase-tracker produces a temporal filter which retains the selected group for further processing while rejecting the unattended ones. The properties of our model are investigated in a number of simulations and interpreted in the context of current theories of visual attention developed in psychology.

Connectionism, Processing, Memory

Abstract computing devices such as Turing machines resolve the dilemma between the necessary finitude of effective procedures and the potential infinity of a function's domain by distinguishing between a finite-state processing part, defined over finitely many representation types, and a memory sufficiently large to contain representation tokens for any of the function's arguments and values. Connectionist networks have been shown to be (at least) Turing-equivalent if provided with infinitely many nodes or infinite-precision activation values and weights. Physical computation, however, is necessarily finite

A CONNECTIONIST APPROACH TO VERTEX COVERING PROBLEMS

This paper describes a connectionist approach to solving computationally difficult minimum vertex covering problems. This approach uses the graph representing the vertex covering problem as the connectionist network without any modifications (nodes of the connectionist network represent vertices and links represent edges of the given graph). The activation rule governing node behavior is derived by breaking down the global constraints on a solution into local constraints on individual nodes. The resulting model uses a competitive activation mechanism to carry out the computation where vertices compete not by explicit inhibitory links but through common resources (edges). Convergence and other properties of this model are formally established by introducing a monotonically non-increasing global energy function. Simulation results show that this model yields very high accuracy, significantly outperforming a well-known sequential approximation algorithm.

STRUCTURAL CONNECTIONIST LEARNING WITH COMPLEMENTARY CODING

We propose a new learning algorithm, structural learning with the complementary coding for concept learning problems. We introduce the new grouping measure that forms the similarity matrix over the training set and show this similarity matrix provides a sufficient condition for the linear separability of the set. Using the sufficient condition one should figure out a suitable composition of linearly separable threshold functions that classify exactly the set of labeled vectors. In the case of the nonlinear separability, the internal representation of connectionist networks, the number of the hidden units and value-space of these units, is pre-determined before learning based on the structure of the similarity matrix. A three-layer neural network is then constructed where each linearly separable threshold function is computed by a linear-threshold unit whose weights are determined by the one-shot learning algorithm that requires a single presentation of the training set. The structural learning algorithm proceeds to capture the connection weights so as to realize the pre-determined internal representation. The pre-structured internal representation, the activation value spaces at the hidden layer, defines intermediate-concepts. The target-concept is then learned as a combination of those intermediate-concepts. The ability to create the pre-structured internal representation based on the grouping measure distinguishes the structural learning from earlier methods such as backpropagation.

Minimum mean-square error transformations of categorical data to target positions

A new algorithm is described for transforming multidimensional data such that all the data points in each of several predefined categories map toward a category target position in the transformed space. The procedure is based on minimizing the mean-square error between specified category target positions and actual transformed locations of the data. Least squares estimation techniques are used to derive linear equations for computing the transformation coefficients and for determining an origin offset in the transformed space. However, for additional flexibility in the transformation, a method is presented for combining the linear transformation with a nonlinear connectionist network transformation. This procedure can, among other things, be used as a tool to evaluate the precision with which physical measurements of psychophysical stimuli correlate with the perceptual configuration of those stimuli. Potential speech science applications are identified. Experimental results illustrate some of these applications with vowel data.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Networks and Theories: The Place of Connectionism in Cognitive Science

This article considers how connectionist modeling can contribute to understanding of human cognition. J argue that connectionist networks should not be thought of as theories or simulations of theories, but may nevertheless contribute to the development of theories.