Connectionist Architecture Research Articles

Competition and recency in a hybrid network model of syntactic disambiguation

The competitive attachment model of human parsing is a hybrid connectionist architecture consisting of a distributed feature passing method for establishing syntactic relations within the network, and a numeric competition mechanism for resolving ambiguities, which applies to all syntactic relations. Because the approach employs a uniform mechanism for establishing syntactic relations, and a single competition mechanism for disambiguation, the model can capture general behaviors of the human parser that hold across a range of syntactic constructions. In particular, attachment and binding relations are similarly processed and are therefore subject to the very same influences of disambuguation and processing over time. An important influence on the competitive disambiguation process is distance within the network. Decay of numeric activation, along with distributed feature passing through the network structure, has an unavoidable effect on the outcome of attachment and binding competitions. Inherent properties of the model thus lead to a principled explanation of recency effects in the human parsing of both attachment and filler/gap ambiguities.

Learning in small connectionist networks does not generalize to large networks

A number of recent models of human information processing have been based on connectionist architectures. Such models are designed to illustrate specific psychological principles and are usually implemented in small networks. The assumption implicit in the work is that principles illustrated in small networks can be applied easily to brain-size networks by scaling up as required. To consider the scaling question, we used a back-propagation algorithm to compare learning in both large and small networks and found that learning depended on the size of the network. In small networks, increasing η (the rate-of-learning parameter) beyond 1 increased the rate of learning; in large networks, the same manipulation reduced the rate of learning. The example illustrates the difficulty of generalizing across network size and calls into question the assumption that principles illustrated in small networks can be applied to the brain by expansion of the network.

Recovering 3-D form features by a connectionist architecture

An original approach to the recovery of 3-D shape information from 2-D images is described. The first network is an implementation of the Boundary Contour system aimed to extract a brightness gradient map from the image. The second is a backpropagation network that estimates the geometric parameters of the object parts present in the acquired scene. A description of simulations of the implemented architecture and the quite satisfactory experimental results are reported as well as the comparisons with some classic approaches to the problem.

Connectionist approaches to inexact reasoning and learning systems for executive and decision support: Conceptual design

Connectionist approaches to the representation and reasoning about problems addressed by executive and decision support systems are described. It is shown that analog object structures coupled with learning produce mechanisms for managerial problem diagnoses. These mechanisms are neural models with multiple-layer structures that support continuous input/output. Because a connectionist architecture is assumed, the prototype system is called the Connectionist Inexact Reasoning System or CIROS. Although uncharacteristic of neural architectures, an explanation capability is developed within CIROS and described herein. The primary contribution of CIROS is that it provides principles for design of inexact reasoning systems for managerial problem diagnosis.

Parametric modeling of the temporal dynamics of neuronal responses using connectionist architectures

1. We describe an exploratory approach to the parametric modeling of dynamical (time-varying) neurophysiological data. The models use stimulus data from a window of time to predict the neuronal firing rate at the end of that window. The most successful models were feedforward three-layered networks of input, hidden, and output "nodes" connected by weights that were adjusted during a training phase by the backpropagation algorithm. The memory in these models was represented by delay lines of varying length propagating activation between the layers. Connectionist models with no memory (1 sequential node per layer) as well as zero-memory nonlinear nonconnectionist models were also tested. 2. Models were tested with recordings of neuronal activity from the auditory thalamic nucleus ovoidalis of urethane-anesthetized zebra finches (Taeniopygia guttata). All cells reported here showed phasic/tonic responses. Extensive modeling of one neuron (cell 1) defined a "canonical" architecture, which was most successful in modeling this cell. The canonical model had a zero-memory input layer, a hidden layer with 29 bins representing 185.6 ms, and a single output node whose value as a function of sequential bin position represented the output of the neuron as a function of time. The canonical model achieved convergence on the entire data set for cell 1, including responses to single tone bursts and zebra finch songs. The "frequency" weights of the canonical model matched well excitatory and inhibitory frequencies for cell 1 as determined by the cell's frequency tuning curve. The "memory" weights of the canonical model were dominated by excitation over the first 25 ms followed by inhibition. 3. When trained with only the dynamical responses to tone bursts, the canonical model also accurately predicted the responses to song (average R2 = 0.823). Thus for this neuron the responses to single tone bursts were sufficient to predict most of the responses to six different songs, each presented at three different amplitudes, although a Monte Carlo procedure indicated the residual variance was not just due to noise (P < 0.001). 4. The model's ability to predict the responses to song was further explored by altering the tone burst (training) data. For cell 1, the responses to songs were most strongly related to the phasic/tonic details of the temporal responses to tone bursts. Changes in the characteristic frequency, frequency tuning curves, and rate/intensity function of the neuron had less effect. These experiments would be difficult or impossible to conduct electrophysiologically.(ABSTRACT TRUNCATED AT 400 WORDS)

Integrating Neural and Symbolic Approaches: A Symbolic Learning Scheme for a Connectionist Associative Memory

This paper deals with the integration of neural and symbolic approaches. It focuses on associative memories where a connectionist architecture tries to provide a storage and retrieval component for...

An efficient feature-based connectionist inheritance scheme

A connectionist model that deals with the inheritance problem in an efficient and natural way is described. Based on the connectionist architecture CONSYDERR, the author analyses the problem of property inheritance and formulates it in ways facilitating conceptual clarity and connectionist implementation. A set of benchmarks is specified for ensuring the correctness of solution mechanisms. Parameters of CONSYDERR are formally derived to satisfy these benchmark requirements. The author also discusses how chaining of is-a links and multiple inheritance can be handled in this architecture. It is shown that CONSYDERR with a two-level dual (localist and distributed) representation can handle inheritance and cancellation of inheritance correctly and extremely efficiently, in constant time instead of proportional to the length of a chain in an inheritance hierarchy. The utility of a meaning-oriented intensional approach (with features) is demonstrated for supplementing and enhancing extensional approaches. >

Connectionism: Explanation or Implementation?

EXCHANGE Connectionism: Explanation or Implementation? Cheryl Fantuzzi University of California, Los Angeles Although purely empiricist, or environment-based, theories of language acquisition suffered some serious setbacks with the rise of generative grammar, they have very recently come into vogue again with a new brand of cognitive modeling known as connectionism. Connectionism represents the strongest form of empiricism: radical connectionists typically argue that all learning is based on the processing of input, and that there is no need to posit any a priori internal structure to the processing system at all. What linguists describe as rule-governed behavior, radical connectionists say is only a description of the emergent behavior of the processor. Learning is simply a matter of strengthening and weakening neural connections in response to the statistical frequency of patterns in the input, and the abstract symbols and rules that are so crucial to current linguistic theory have no place at all in a connectionist system. There have been varied responses to these strong claims. Some have been wildly enthusiastic about the new approach (e.g., Sampson, 1987) while others have severely Mehler, criticized many of its claims (e.g., the papers in Pinker extensive middle ground between the two 1989). There is also an extremes however. While pure Parallel Distributed Processing (PDP) models seem to work best with problems involving motor control or the earliest stages of visual processing, connectionists working with more complex cognitive processes such as language or problem solving have often incorporated symbols into their connectionist architectures (e.g., the papers in Hinton, 1991). There has recendy been some interest in the applicability of connectionist models to Second Language Acquisition (SLA) theory and research (Schmidt, 1988; Gasser, 1990; Sokolik, 1990). In the last issue of ML, Yas Shirai (1992) added his voice Issues in Applied Linguistics © Regents of the University of California ISSN 1050-4273 Vol.3 No. 2 1992 319-340

A connectionist model for commonsense reasoning incorporating rules and similarities

For the purpose of modelling commonsense reasoning, we investigate connectionist models of rule-based reasoning, and show that while such models can usually carry out reasoning in exactly the same way as symbolic systems, they have more to offer in terms of commonsense reasoning. A connectionist architecture, CONSYDERR, is proposed for capturing certain commonsense reasoning competence, which partially remedies the brittleness problem in traditional rule-based systems. The architecture employs a two-level, dual representational scheme, which utilizes both localist and distributed representations and explores the synergy resulting from the interaction between the two. CONSYDERR is therefore capable of accounting for many difficult patterns in commonsense reasoning with this simple combination of the two levels. This work shows that connectionist models of reasoning are not just “implementations” of their symbolic counterparts, but better computational models of commonsense reasoning.

A SURVEY ON THE PARALLEL DISTRIBUTED PROCESSING OF PRODUCTION SYSTEMS

The importance of production systems in artificial intelligence (AI) has been repeatedly demonstrated by a large number of expert systems. As the number and size of expert systems grow, there has however been an emerging obstacle in such AI applications: the large processing time. The need for faster execution of production systems has spurred research in both the software and hardware domains, including connectionist architectures. This paper surveys various aspects of parallel distributed processing of production systems. Approaches taken to date to solve the problems associated with production systems are classified here into three levels: the algorithmic level, the parallel implementation level, and the connectionist level. Several pattern matchers and multiple rule firing principles are presented to demonstrate the algorithm level improvement. Several parallel implementation efforts are surveyed along with experimental results on real machines or with simulators. The presentation of three different types of connectionist production systems (local, distributed, and hierarchical representation) completes this survey. Finally, we explore some potential avenues towards the implementation of a true asynchronous parallel production system.

Decision making on creditworthiness, using a fuzzy connectionist model

In this paper we present an unpolished expert system development tool, based on a connectionist architecture for knowledge representation. Our work is centered around a connectionist expert system, which can be expanded, and updated through learning of sample domain specific cases [1,4]. A cell recruitment learning algorithm [2] capable of forgetting previously learned facts by learning new ones is incorporated. Using this learning mechanism, we let the system learn a knowledge base on classifying the creditworthiness of credit applicants. The knowledge base consisted of 50 credit cases and was obtained from [11]. It will be shown that the fuzznet system is capable of learning such a model, with only a small amount of the cases being presented to it for learning. In all examples learned, an acceptable model was derived (based on the average prognostic error of actual and expected output for creditworthiness). The input features and their corresponding outputs (which were learned) are all fuzzy (uncertain) at the time of learning. So far implementations of connectionist expert systems, either only allowed for crisp inputs when placed in the learning mode, or only supported uncertainty when simulated [1, 8, 9, 10]. This example will further demonstrate the versatility of the fuzznet system, when dealing with uncertainty in the inputs and outputs when being placed in the learn mode, or while being simulated.

LIGHTNESS CONSTANCY FROM LUMINANCE CONTRAST

To discount effects of uneven illumination we have designed and tested a neural network that can adaptively control light sensitivity at the photosensor level. Our neural network architecture simulates the ON channel response of the visual system using multiple layers of hexagonally arranged nodes having partially overlapping receptive fields of different spatial frequencies. Feedforward connections are excitatory while feedback pathways subserve lateral inhibition. The outputs of these nodes are combined so as to maximize the signal to noise ratio while providing constant feedback that resets photosensor thresholds to maintain high sensitivity. A sparse primitive interpolation technique was applied to the ensemble output of the sensitivity control module to determine if it sufficiently encodes surface reflectance. The motivation is to determine to what extent the ratio principle, as captured by the sensitivity control system, explains the lightness constancy phenomenon and whether information contained within an ON channel response is adequate to reconstruct the surface lightness. Our connectionist architecture can account for many characteristics attributed to the lightness constancy phenomenon observed in biological systems. The results suggest that our module maintains high sensitivity across a large range of intensities without interfering with the transmission of visual information embedded in the spatial discontinuities of intensity. However, the amplitude of the luminance derivative as encoded in ON channel responses is not sufficient to approximate surface reflectance.

A connectionist approach to the quadratic assignment problem

The possibilities of applying a Boltzmann machine, and a related connectionist model in which the escape from local optima is performed in a deterministic way using tabu search, are tested for the quadratic assignment problem (QAP). Inefficiences with this approach led to an improved computational model for the QAP which is based on connectionist architecture. Computational results for problems of dimensions ranging from 5 up to 90 are given.

Connectionism and the mind: an introduction to parallel processing in networks

Networks versus symbol systems - two approaches to modelling cognition connectionist architectures learning pattern recognition - connectionism's forte connectionism and nonpropositional representations of knowledge two simulations of higher cognitive processes are rules and symbols needed? - critiques and defences of connectionism connectionism and the disciplines of cognitive science. Appendices: notation glossary.

SLUG: A connectionist architecture for inferring the structure of finite-state environments

Consider a robot wandering around an unfamiliar environment, performing actions and sensing the resulting environmental states. The robot's task is to construct an internal model of its environment, a model that will allow it to predict the consequences of its actions and to determine what sequences of actions to take to reach particular goal states. Rivest and Schapire (1987a, 1987bs Schapire, 1988) have studied this problem and have designed a symbolic algorithm to strategically explore and infer the structure of “finite state” environments. The heart of this algorithm is a clever representation of the environment called an update graph. We have developed a connectionist implementation of the update graph using a highly-specialized network architecture. With back propagation learning and a trivial exploration strategy—choosing random actions—the network can outperform the Rivest and Schapire algorithm on simple problems. Perhaps the most interesting consequence of the connectionist approach is that, by relaxing the constraints imposed by a symbolic description, it suggests a more general representation of the update graph, thus allowing for greater flexibility in expressing potential solutions.

Task decomposition through competition in a modular connectionist architecture: The what and where vision tasks

Task decomposition through competition in a modular connectionist architecture: The what and where vision tasks

Task Decomposition Through Competition in a Modular Connectionist Architecture: The What and Where Vision Tasks

A novel modular connectionist architecture is presented in which the networks composing the architecture compete to learn the training patterns. An outcome of the competition is that different networks learn different training patterns and, thus, learn to compute different functions. The architecture performs task decomposition in the sense that it learns to partition a task into two or more functionally independent tasks and allocates distinct networks to learn each task. In addition, the architecture tends to allocate to each task the network whose topology is most appropriate to that task. The architecture's performance on “what” and “where” vision tasks is presented and compared with the performance of two multilayer networks. Finally, it is noted that function decomposition is an underconstrained problem, and, thus, different modular architectures may decompose a function in different ways. A desirable decomposition can be achieved if the architecture is suitably restricted in the types of functions that it can compute. Finding appropriate restrictions is possible through the application of domain knowledge. A strength of the modular architecture is that its structure is well suited for incorporating domain knowledge.

A browser for large knowledge bases based on a hybrid distributed/local connectionist architecture

A browser concept based on a connectionist architecture is presented. The concept utilizes both distributed and local representations. A proof-of-concept system is implemented for an integrally developed, Honeywell-proprietary knowledge acquisition tool. In the browser, concepts and relations in a knowledge base are represented using microfeatures. The microfeatures can encode semantic attributes, structural features, contextual information, etc. Desired portions of the knowledge base can then be associatively retrieved based on a structured cue. An ordered list of partial matches is presented to the user for selection. Microfeatures can also be used as bookmarks-they can be placed dynamically at appropriate points in the knowledge base and subsequently used as retrieval cues. The browser concept can be applied wherever there is a need for conveniently inspecting and manipulating structured information.< <ETX xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">&gt;</ETX>

Sequence Manipulation Using Parallel Mapping Networks

We describe a parallel mapping matrix that performs several types of sequence manipulations that are the building blocks of well-known phonological processes. Our results indicate that human phonological behavior can by modeled by a highly constrained connectionist architecture, one that uses purely feedforward circuitry and imposes tight limits on depth of derivations.

Towards connectionist production systems

Abstract This article describes a connectionist architecture, CAPS, for implementing production systems programmed in the OPS5 language. CAPS illustrates the potential benefits and obstacles in using artificial neural networks (i.e., connectionist networks) in expert systems. The main objective of CAPS is to enhance the performance of existing rule-based OPS5 programs (e.g., expert systems) by implementing them on connectionist networks. CAPS supports a subset of the OPS5 language which includes variables, negation, conjunction, and disjunction of conditions. It uses a translation program to transform an OPS5 program into a limitedly interconnected, fully trained neural network. The translation program eliminates the learning/training phase in creating a network. CAPS resolves the variable binding problems normally associated with connectionist architectures by using local representations. Local representations also simplify the overall architecture and reduce the number of interconnections within the network, thus increasing the feasibility of hardware implementations. A hardware implementations of a CAPS network can potentially provide an 80- to 200- fold increase in parallelism over serial implementations, allowing significant increases in production system performance. Although CAPS advances the state of connectionist production systems, it still has some problems and limitations, particularly in handling simultaneously matched variables and performing conflict resolution. The CAPS architecture has been tested by transforming small OPS5 programs into connectionist networks and simulating them on a connectionist simulator. In addition, various cases of the classic Monkey and Bananas program have been successfully simulated. CAPS demonstrates that connectionist architectures can perform rule-based symbolic reasoning and support dynamic variable bindings.