Connectionist Learning Research Articles

Acquisition by Processing Theory: A theory of everything?

Truscott and Sharwood Smith (henceforth T&SS) propose a novel theory of language acquisition, ACQUISITION BY PROCESSING THEORY (APT), designed to account for both first and second language acquisition, monolingual and bilingual speech perception and parsing, and speech production. This is a tall order. Like any theoretically ambitious enterprise, the APT shares certain properties with much that has gone before. Like the Competition Model (CM; MacWhinney, 1987, 1997; MacWhinney and Bates, 1989, inter alia) and other associative network connectionist learning models, the APT eschews a Language Acquisition Device (LAD) by treating acquisition as the strengthening of levels of representation activation. A parser can produce multiple representations of a parse string in parallel, which then ‘compete’ as analyses for an input string. Unlike the CM, however, the APT is not motivated by a solid program of empirical studies in language acquisition or cross-language processing. Nor does it strike me as theoretically coherent, for the APT, unlike the CM, assumes that knowledge of language involves knowledge of grammatical structure and that the parser makes deterministic use of Universal Grammar in the form of a Minimalist grammar. The determinism is important here; the claim to eliminate LAD hinges on it.

Flexible on‐line Modeling and Control of pH in Waste Neutralization Reactors

AbstractThe control of pH in waste neutralization processes presents a challenging highly nonlinear and time‐varying problem in which the reactor also suffers from inaccessible state information. The ability to characterize the changing dynamics of such reactors is essential to the success of advanced control schemes for these applications. In this work, flexible on‐line modeling of a pH reactor simulating nonstationary behavior was studied. This entailed a comparison of the most popular connectionist learning algorithm, the “Widrow‐Hoff delta rule”, with a classical tool in adaptive identification and control, recursive least squares (RLS). The modeling was pursued within the framework of neural networks using the ADALINE neural network. Further, two heuristically defined first‐principles‐based transforms were investigated for providing “general globally linearizing” information to the ADALINE. The comparisons of the learning algorithms for different neural network information vectors has led to a critical understanding of the flexibility of each algorithm for on‐line learning of the diverse process gain characteristics encountered in pH reactors.

A knowledge-resonance (KRES) model of category learning.

This article introduces a connectionist model of category learning that takes into account the prior knowledge that people bring to new learning situations. In contrast to connectionist learning models that assume a feedforward network and learn by the delta rule or backpropagation, this model, the knowledge-resonance model, or KRES, employs a recurrent network with bidirectional symmetric connection whose weights are updated according to a contrastive Hebbian learning rule. We demonstrate that when prior knowledge is represented in the network, KRES accounts for a considerable range of empirical results regarding the effects of prior knowledge on category learning, including (1) the accelerated learning that occurs in the presence of knowledge, (2) the better learning in the presence of knowledge of category features that are not related to prior knowledge, (3) the reinterpretation of features with ambiguous interpretations in light of error-corrective feedback, and (4) the unlearning of prior knowledge when that knowledge is inappropriate in the context of a particular category.

Do current connectionist learning models account for reading development in different languages?

Learning to read a relatively irregular orthography, such as English, is harder and takes longer than learning to read a relatively regular orthography, such as German. At the end of grade 1, the difference in reading performance on a simple set of words and nonwords is quite dramatic. Whereas children using regular orthographies are already close to ceiling, English children read only about 40% of the words and nonwords correctly. It takes almost 4 years for English children to come close to the reading level of their German peers. In the present study, we investigated to what extent recent connectionist learning models are capable of simulating this cross-language learning rate effect as measured by nonword decoding accuracy. We implemented German and English versions of two major connectionist reading models, Plaut et al.'s (Plaut, D. C., McClelland, J. L., Seidenberg, M. S., & Patterson, K. (1996). Understanding normal and impaired word reading: computational principles in quasi-regular domains. Psychological Review, 103, 56–115) parallel distributed model and Zorzi et al.'s (Zorzi, M., Houghton, G., & Butterworth, B. (1998a). Two routes or one in reading aloud? A connectionist dual-process model. Journal of Experimental Psychology: Human Perception and Performance, 24, 1131–1161); two-layer associative network. While both models predicted an overall advantage for the more regular orthography (i.e. German over English), they failed to predict that the difference between children learning to read regular versus irregular orthographies is larger earlier on. Further investigations showed that the two-layer network could be brought to simulate the cross-language learning rate effect when cross-language differences in teaching methods (phonics versus whole-word approach) were taken into account. The present work thus shows that in order to adequately capture the pattern of reading acquisition displayed by children, current connectionist models must not only be sensitive to the statistical structure of spelling-to-sound relations but also to the way reading is taught in different countries.

Redefining the rules: Providing race models with a connectionist learning rule

In this paper, we describe the Parallel Race Network (PRN), a race model with the ability to learn stimulus-response associations using a formal framework that is very similar to the one used by the traditional connectionist networks. The PRN assumes that the connections represent abstract units of time rather than strengths of association. Consequently, the connections in the network indicate how rapidly the information should be sent to an output unit. The decision is based on a race between the outputs. To make learning functional and autonomous, the Delta rule was modified to fit the time-based assumption of the PRN. Finally, the PRN is used to simulate an identification task and the implications of its mode of representation are discussed.

Learning to divide the labor: an account of deficits in light and heavy verb production

Theories of sentence production that involve a convergence of activation from conceptual-semantic and syntactic-sequential units inspired a connectionist model that was trained to produce simple sentences. The model used a learning algorithm that resulted in a sharing of responsibility (or “division of labor”) between syntactic and semantic inputs for lexical activation according to their predictive power. Semantically rich, or “heavy”, verbs in the model came to rely on semantic cues more than on syntactic cues, whereas semantically impoverished, or “light”, verbs relied more on syntactic cues. When the syntactic and semantic inputs were lesioned, the model exhibited patterns of production characteristic of agrammatic and anomic aphasic patients, respectively. Anomic models tended to lose the ability to retrieve heavy verbs, whereas agrammatic models were more impaired in retrieving light verbs. These results obtained in both sentence production and single-word naming simulations. Moreover, simulated agrammatic lexical retrieval was more impaired overall in sentences than in single-word tasks, in agreement with the literature. The results provide a demonstration of the division-of-labor principle, as well as general support for the claim that connectionist learning principles can contribute to the understanding of non-transparent neuropsychological dissociations.

FREQUENCY EFFECTS IN LANGUAGE PROCESSING

This article shows how language processing is intimately tuned to input frequency. Examples are given of frequency effects in the processing of phonology, phonotactics, reading, spelling, lexis, morphosyntax, formulaic language, language comprehension, grammaticality, sentence production, and syntax. The implications of these effects for the representations and developmental sequence of SLA are discussed. Usage-based theories hold that the acquisition of language is exemplar based. It is the piecemeal learning of many thousands of constructions and the frequency-biased abstraction of regularities within them. Determinants of pattern productivity include the power law of practice, cue competition and constraint satisfaction, connectionist learning, and effects of type and token frequency. The regularities of language emerge from experience as categories and prototypical patterns. The typical route of emergence of constructions is from formula, through low-scope pattern, to construction. Frequency plays a large part in explaining sociolinguistic variation and language change. Learners' sensitivity to frequency in all these domains has implications for theories of implicit and explicit learning and their interactions. The review concludes by considering the history of frequency as an explanatory concept in theoretical and applied linguistics, its 40 years of exile, and its necessary reinstatement as a bridging variable that binds the different schools of language acquisition research.

The Value of Symbolic Computation

Standard generative linguistic theory, which uses discrete symbolic models of cognition, has some strengths and weaknesses. It is strong on providing a network of outposts that make scientific travel in the jungles of natural language feasible. It is weak in that it currently depends on the elaborate and unformalized use of intuition to develop critical supporting assumptions about each data point. In this regard, it is not in a position to characterize natural language systems in the lawful terms that ecological psychologists strive for. Connectionist learning models offer some help: They define lawful relations between linguistic environments and language systems. But our understanding of them is currently weak, especially when it comes to natural language syntax. Fortunately, symbolic linguistic analysis can help connectionism if the two meet via dynamical systems theory. I discuss a case in point: Insights from linguistic explorations of natural language syntax appear to have identified information structures that are particularly relevant to understanding ecologically appealing but analytically mysterious connectionist learning models.

Connectionist learning: A comparison of neural networks and an optical thin-film multilayer model

Current work on connectionist models has been focused largely on artificial neural networks that are inspired by the networks of biological neurons in the human brain. However, there are also other connectionistarchitectures that differ significantly from this biological exemplar. We proposed a novel connectionist learning architecture inspired by the physics associated with optical coatings of multiple layers of thin-films in a previous paper (Li and Purvis 1999, Annals of Mathematics and Artificial Intelligence, 26: 1-4). The proposed model differs significantly from the widely used neuron-inspired models. With thin-film layer thicknesses serving as adjustable parameters (as compared with connection weights in a neural network) for the learning system, the optical thin-film multilayer model (OTFM) is capable of approximating virtually any kind of highly nonlinear mappings. The OTFM is not a physical implementation using optical devices. Instead, it is proposed as a new connectionist learning architecture with its distinct optical properties as compared with neural networks. In this paper we focus on a detailed comparison of neural networks and the OTFM (Li 2001, Proceedings ofINNS-IEEE International Joint Conference on Neural Networks, Washington, DC, pp. 1727-1732). We describe the architecture of the OTFM and show how it can be viewed as a connectionist learning model. We then present experimental results on solving a classification problem and a time series prediction problem that are typical of conventional connectionist architectures to demonstrate the OTFM's learning capability.

Artificial nonmonotonic neural networks

In this paper, we introduce Artificial Nonmonotonic Neural Networks (ANNNs), a kind of hybrid learning systems that are capable of nonmonotonic reasoning. Nonmonotonic reasoning plays an important role in the development of artificial intelligent systems that try to mimic common sense reasoning, as exhibited by humans. On the other hand, a hybrid learning system provides an explanation capability to trained Neural Networks through acquiring symbolic knowledge of a domain, refining it using a set of classified examples along with Connectionist learning techniques and, finally, extracting comprehensible symbolic information. Artificial Nonmonotonic Neural Networks acquire knowledge represented by a multiple inheritance scheme with exceptions, such as nonmonotonic inheritance networks, and then can extract the refined knowledge in the same scheme. The key idea is to use a special cell operation during training in order to preserve the symbolic meaning of the initial inheritance scheme. Methods for knowledge initialization, knowledge refinement and knowledge extraction are introduced. We, also, prove that these methods address perfectly the constraints imposed by nonmonotonicity. Finally, performance of ANNNs is compared to other well-known hybrid systems, through extensive empirical tests.

Connectionist learning of regular graph grammars

This paper presents a new connectionist approach to grammatical inference. Using only positive examples, the algorithm learns regular graph grammars, representing two-dimensional iterative structures drawn on a discrete Cartesian grid. This work is intended as a case study in connectionist symbol processing and geometric concept formation. A grammar is represented by a self-configuring connectionist network that is analogous to a transition diagram except that it can deal with graph grammars as easily as string grammars. Learning starts with a trivial grammar. expressing no grammatical knowledge, which is then refined, by a process of successive node splitting and merging, into a grammar adequate to describe the population of input patterns. In conclusion. I argue that the connectionist style of computation is, in some ways, better suited than sequential computation to the task of representing and manipulating recursive structures.

Symbolic vs. connectionist learning: an experimental comparison in a structured domain

During the last two decades, the attempts to find effective solutions to the problem of learning any kind of structured information have been splitting the scientific community. A "holy war" has been fought between the advocates of a symbolic approach to learning and the advocates of a connectionist approach. One of the most repeated claims of the symbolic party has been that symbolic methods are able to cope with structured information while connectionist ones are not. However, in the last few years, the possibility of employing connectionist methods for structured data has been widely investigated and several approaches have been proposed. A novel algorithm for learning structured descriptions, ascribable to the category of symbolic techniques, is proposed. It faces the problem directly in the space of graphs by defining the proper inference operators, as graph generalization and graph specialization, and obtains general and consistent prototypes with a low computational cost with respect to other symbolic learning systems. The proposed algorithm is compared with a recent connectionist method for learning structured data (P. Frasconi et al., 1998), with reference to a problem of handwritten character recognition from a standard database on the Web. The orthogonality of the two approaches strongly suggests their combination in a multiclassifier system so as to retain the strengths of both of them, while overcoming their weaknesses. The results on an experimental case study demonstrated that the adoption of a parallel combination scheme of the two algorithms could improve the recognition performance by about 10 percent. A truce or an alliance between the symbolic and the connectionist worlds?.

Knowledge Representation: Two Kinds Of Emergence

Two different but closely related issues in current cognitive science will be considered in this essay. One is the controversial and extensively discussed question of how connectionist and symbolic representations of knowledge are related to each other. The other concerns the notion of connectionist learning and its relevance for the understanding of the distinction between propositional and nonpropositional knowledge. More specifically, I shall give an overview of a result in Rantala and Vaden (1994) establishing a limiting case correspondence between symbolic and connectionist representations and, on the other hand, study the problem, preliminarily investigated in Rantala (1998), of how propositional knowledge may arise from nonpropositional knowledge. I shall also try to point out that on some more or less plausible assumptions, often made by cognitive scientists, these results may have some significance when we try to comprehend the nature of human knowledge representation. Some of these assumptions are rather hypothethical and debatable for the time being and they will become justified in the future only if there will be more progress in the empirical and theoretical research on the brain and on artificial networks. The assumptions concern, besides some questions of the behavior of neural networks, such things as the relevance of pattern recognition for modelling human cognition, in particular, knowledge acquisition, and the relation between emergence and reduction.

Using cognitive science methods to assess the role of social information processing in sexually coercive behavior.

Seventy-four undergraduate men completed cognitive performance tasks assessing perceptual organization, classification, and category learning, as well as self-report measures relevant to sexual coercion. The stimuli were slides of Caucasian women who varied along affect and physical exposure (i.e., sensuality) dimensions. Data were analyzed using a weighted multidimensional scaling model, signal-detection theory analyses, and a connectionist learning model (RASHNL; J. K. Kruschke & M. K. Johansen, 1999). Individual differences in performance on the classification and category-learning tasks were congruent with individual differences in perceptual organization. Additionally, participants who showed relatively more attention to exposure than to affect were less sensitive to women's negative responses to unwanted sexual advances. Overall, the study demonstrates the feasibility and utility of cognitive science methods for studying information processing in psychopathology.

FRACTALS IN CELLULAR SYSTEMS AS SOLUTIONS FOR COGNITIVE PROBLEMS

A cellular method for the solution of function approximation and classification problems is introduced. It solves problems of the type solved by connectionist networks, but has an advantage of visualization. Solutions of complex, high-dimensional and non-linear problems correspond to sets of units that define fractal, two-dimensional patterns of aesthetic attractiveness. These sets, in turn, can be considered as patterns, and stored with a classic connectionist learning rule in a network. Then, one obtains a network in which the memorized patterns correspond to meta-patterns in the sense that they correspond to mappings between input and output patterns.

A model of response time effects in symbolic comparison.

A cognitive process model is developed that predicts the 3 major symbolic comparison response time effects (distance, end, and semantic congruity) found in the results of the linear syllogistic reasoning task. The model includes a simple connectionist learning component and dual evidence accumulation decision-making components. It assumes that responses can be based either on information concerning the positional difference between the presented stimulus items or on information concerning the endpoint status of each of these items. The model provides an excellent quantitative account of the mean correct response times obtained from 16 participants who performed paired comparisons of 6 ordered symbolic stimuli (3-letter names).

A computational and neuropsychological account of object‐oriented behaviours in infancy

Infants under 7 months of age fail to reach behind an occluding screen to retrieve a desired toy even though they possess sufficient motor skills to do so. However, even by 3.5 months of age they show surprise if the solidity of the hidden toy is violated, suggesting that they know that the hidden toy still exists. We describe a connectionist model that learns to predict the position of objects and to initiate a response towards these objects. The model embodies the dual‐route principle of object information processing characteristic of the cortex. One route develops a spatially invariant surface feature representation of the object whereas the other route develops a feature blind spatial–temporal representation of the object. The model provides an account of the developmental lag between infants’ knowledge of hidden objects and their ability to demonstrate that knowledge in an active retrieval task, in terms of the need to integrate information across multiple object representations using (associative) connectionist learning algorithms. Finally, the model predicts the presence of an early dissociation between infants’ ability to use surface features (e.g. colour) and spatial–temporal features (e.g. position) when reasoning about hidden objects. Evidence supporting this prediction has now been reported.

How do Simple Rules `Fit to Reality' in a Complex World?

The theory of fast and frugal heuristics, developed in a new book called Simple Heuristics that make Us Smart (Gigerenzer, Todd, and the ABC Research Group, in press), includes two requirements for rational decision making. One is that decision rules are bounded in their rationality –- that rules are frugal in what they take into account, and therefore fast in their operation. The second is that the rules are ecologically adapted to the environment, which means that they `fit to reality.' The main purpose of this article is to apply these ideas to learning rules–-methods for constructing, selecting, or evaluating competing hypotheses in science, and to the methodology of machine learning, of which connectionist learning is a special case. The bad news is that ecological validity is particularly difficult to implement and difficult to understand. The good news is that it builds an important bridge from normative psychology and machine learning to recent work in the philosophy of science, which considers predictive accuracy to be a primary goal of science.

A Hybrid Architecture for Situated Learning of Reactive Sequential Decision Making

In developing autonomous agents, one usually emphasizes only (situated) procedural knowledge, ignoring more explicit declarative knowledge. On the other hand, in developing symbolic reasoning models, one usually emphasizes only declarative knowledge, ignoring procedural knowledge. In contrast, we have developed a learning model CLARION, which is a hybrid connectionist model consisting of both localist and distributed representations, based on the two-level approach proposed in [40]. CLARION learns and utilizes both procedural and declarative knowledge, tapping into the synergy of the two types of processes, and enables an agent to learn in situated contexts and generalize resulting knowledge to different scenarios. It unifies connectionist, reinforcement, and symbolic learning in a synergistic way, to perform on-line, bottom-up learning. This summary paper presents one version of the architecture and some results of the experiments.

On the momentum term in gradient descent learning algorithms

A momentum term is usually included in the simulations of connectionist learning algorithms. Although it is well known that such a term greatly improves the speed of learning, there have been few rigorous studies of its mechanisms. In this paper, I show that in the limit of continuous time, the momentum parameter is analogous to the mass of Newtonian particles that move through a viscous medium in a conservative force field. The behavior of the system near a local minimum is equivalent to a set of coupled and damped harmonic oscillators. The momentum term improves the speed of convergence by bringing some eigen components of the system closer to critical damping. Similar results can be obtained for the discrete time case used in computer simulations. In particular, I derive the bounds for convergence on learning-rate and momentum parameters, and demonstrate that the momentum term can increase the range of learning rate over which the system converges. The optimal condition for convergence is also analyzed.