PDP Networks Research Articles

Modeling Morphological Priming in German With Naive Discriminative Learning

Both localist and connectionist models, based on experimental results obtained for English and French, assume that the degree of semantic compositionality of a morphologically complex word is reflected in how it is processed. Since priming experiments using English and French morphologically related prime-target pairs reveal stronger priming when complex words are semantically transparent (e.g., {\em refill--fill\/}) compared to semantically more opaque pairs (e.g., {\em restrain--strain}), localist models set up connections between complex words and their stems only for semantically transparent pairs. Connectionist models have argued that the effect of transparency should arise as an epiphenomenon in PDP networks. However, for German, a series of studies has revealed equivalent priming for both transparent and opaque prime-target pairs, which suggests mediation of lexical access by the stem, independent of degrees of semantic compositionality. This study reports a priming experiment that replicates equivalent priming for transparent and opaque pairs. We show that these behavioral results can be straightforwardly modeled by a computational implementation of Word and Paradigm Morphology ({\sc wpm}), Naive Discriminative Learning ({\sc ndl}). Just as {\sc wpm}, {\sc ndl} eschews the theoretical construct of the morpheme. {\sc Ndl} succeeds in modeling the German priming data by inspecting the extent to which a discrimination network pre-activates the target lexome from the orthographic properties of the prime. Measures derived from an {\sc ndl} network, complemented with a semantic similarity measure derived from distributional semantics, predict lexical decision latencies with somewhat improved precision compared to classical measures such as word frequency, prime type, and human association ratings. We discuss both the methodological implications of our results, as well as their implications for models of the mental lexicon.

Modeling design as situated and distributed process

In recent years, design learning environments have received considerable attention as contexts for learning diverse subjects such as mathematics, science, or engineering. In this paper, I show how designing can be modeled as a distributed process. Empirical examples are used to show three core aspects of situated design. First, material aspects of the setting facilitate negotiation of design concepts, a process modeled as constraint satisfaction in PDP networks. Second, materials and artifacts afford the creation of virtual design concepts. Finally, fact construction in school science laboratories is intimately tied to the artifacts that constitute students' design elements. As part of the analysis, constraint satisfaction models (implemented in PDP networks) and ontological maps are introduced as modeling tools that formalize the distributed aspects of designing.

On the Subsymbolic Nature of a PDP Architecture that Uses a Nonmonotonic Activation Function

PDP networks that use nonmonotonic activation functions often produce hidden unit regularities that permit the internal structure of these networks to be interpreted (Berkeley et al., 1995; McCaughan, 1997; Dawson, 1998). In particular, when the responses of hidden units to a set of patterns are graphed using jittered density plots, these plots organize themselves into a set of discrete stripes or bands. In some cases, each band is associated with a local interpretation. On the basis of these observations, Berkeley (2000) has suggested that these bands are both subsymbolic and symbolic in nature, and has used the analysis of one network to support the claim that there are fewer differences between symbols and subsymbols than one might expect. We suggest below that this conclusion is premature. First, in many cases the local interpretation of each band is difficult to relate to the interpretation of a network's response; a more appropriate relationship only emerges when a band associated with one hidden unit is considered in the context of other bands associated with other hidden units (i.e., interpretations of distributed representations are more useful than interpretations of local representations). Second, the content that a band designates to an external observer (i.e., the interpretation assigned to a band by the researcher) can be quite different from the content that a band designates to the output units of the network itself.. We use two different network simulations – including the one described by Berkeley (2000) – to illustrate these points. We conclude that current evidence involving interpretations of nonmonotonic PDP networks actually illustrates the differences between symbolic and subsymbolic processing.

Using Extra Output Learning to Insert a Symbolic Theory into a Connectionist Network

This paper examines whether a classical model could be translated into a PDP network using a standard connectionist training technique called extra output learning. In Study 1, standard machine learning techniques were used to create a decision tree that could be used to classify 8124 different mushrooms as being edible or poisonous on the basis of 21 different Features (Schlimmer, 1987). In Study 2, extra output learning was used to insert this decision tree into a PDP network being trained on the identical problem. An interpretation of the trained network revealed a perfect mapping from its internal structure to the decision tree, representing a precise translation of the classical theory to the connectionist model. In Study 3, a second network was trained on the mushroom problem without using extra output learning. An interpretation of this second network revealed a different algorithm for solving the mushroom problem, demonstrating that the Study 2 network was indeed a proper theory translation.

Representing space in a PDP network: Coarse allocentric coding can mediate metric and nonmetric spatial judgements

In one simulation, an artificial neural networkwas trained to rate the distances between pairsof cities on the map of Alberta, given onlyplace names as input. Distance ratings rangedfrom 0 (when the network rated the distancebetween a city and itself) to 10. The questionof interest was the nature of therepresentations developed by the network's sixhidden units after it successfully learned tomake the desired responses. Analyses indicatedthat the network used coarse allocentric codingto solve this problem. Each hidden unit couldbe described as occupying a position on the mapof Alberta, and each connection weight feedinginto a hidden unit was related to the distanceon the map between the hidden unit and one ofthe stimulus cities. On its own, a singlehidden unit's response was a relativelyinaccurate distance measure. However, bycombining all six hidden unit responses in acoarse coding scheme, accurate responses weregenerated by the network. In a secondsimulation, a second network was trained tomake similar judgements, but was trained toviolate the minimality constraint on metricspace when trained to judge the distancebetween a city and itself. An analysis of thisnetwork indicated that it too was using coarseallocentric coding.

Densmore and Dennett on Virtul Machines and Consciousness

In the preceding article, Densmore and Dennett (hereafter, D&D) urge me to note the parallels, and the potentially complementary fit, between Dennett's views on consciousness and my own. They are correct in arguing that I had not fully appreciated the parallels. I am pleased and honored by every one of them. Still, it will be more illuminating to press the remaining differences between us. They are several and they are important, quite aside from the question of which of our views is correct. Like good functionalists, D&D's first and almost unrestrained impulse is to see the mechanism of recurrent PDP network as (welcome and perhaps correct) story of how the abstract cognitive organization of humans and animals gets implemented in the physical The familiar metaphors of virtual and explanatory levels are once again confidently deployed, and the mind is assimilated to a software Program that is installed upon the parallel network of the brain. D&D are of course aware that these metaphors now need (at least) an additional layer of cautionary commentary, and, just as Dennett did in Consciousness Explained (CE), they do not hesitate to offer it. But the commentary still seems to me to be self-deceptive and uncomprehending. Those metaphors do not need to be qualified, they need to be junked. Though we have all grown up on them, they should be ushered swiftly into our history, at least where talk of minds is concerned. Those metaphors derive their primary meanings from perfectly sensible theory, theory that is perfectly adequate to classical computational procedures and to classical computing machines (and, let us add, to classical functionalism in philosophy). But that well-polished theory, according to some of us, has turned out to be entirely the wrong model for the cognitive activity of biological brains. It is wrong in its story of our basic computational mechanisms, it is wrong in its account of how occurrent information is coded and manipulated, it is wrong in its account of how skills are embodied and how regular behavior

The Virtues of Virtual Machines

Paul Churchland's book, The Engine of Reason, The Seat of the Soul (hereafter ER), is an entertaining and instructive advertisement for a neurocomputational vision of how the brain (and mind) works. While we agree with its general thrust, and commend its lucid pedagogy on a host of difficult topics, we note that such pedagogy often exploits artificially heightened contrast, and sometimes the result is a misleading caricature instead of a helpful simplification. In particular, Churchland is eager to contrast the explanation of consciousness that can be accomplished by his aspiring new structural and dynamic cognitive prototype: recurrent PDP networks (p. 266) with what strikes him as the retrograde introduction by Dennett of a virtual von Neumannesque machine-a failed prototype -as the key element in an explanation of human consciousness (in Consciousness Explained, 1991, hereafter, CE). We will try to show that by oversimplifying Dennett's alternative, he has taken a potential supplement to his own view-a much needed supplement-and transformed it in his imagination into a subversive threat. In part 1, we will expose and correct the mistaken contrasts. In part 2, we will compare the performance of the two views on Churchland's list of seven features of consciousness any theory must account for, showing that Dennett's account provides more than Churchland has recognized, and indeed offers answers to key questions that Churchland's account is powerless to address. At that point, Churchland's project and Dennett's could be seen to collaborate in a useful division of labor instead of being in mortal combat, were it not for what appears to be a fairly major disagreement about consciousness in non-human animals. Part 3 briefly examines this issue. It may be due to a misunderstanding, which when cleared up might restore the happy prospect of unification.

What about consciousness during learning?

Though we fully agree that unconscious processing produces explicit representations that form the conscious phenomenal experience of the subject, identifying phenomenal experience with stable patterns of activation in a PDP network seriously limits O'Brien & Opie's thesis. They fail to recognize the constructive role of consciousness during the learning episode itself, reducing consciousness to a resulting outcome of the learning episode. We illustrate how consciousness can guide and shape the formation of increasingly structured representations of the world by presenting a brief outline of a model for speech segmentation.

Quantities of qualia

We address two points in this commentary. First, we question the extent to which O'Brien & Opie have established that the classical approach is unable to support a viable vehicle theory of consciousness. Second, assuming that connectionism does have the resources to support a vehicle theory, we explore how the activity of the units of a PDP network might sum together to form phenomenal experience (PE).

PDP networks can provide models that are not mere implementations of classical theories

There is widespread belief that connectionist networks are dramatically different from classical or symbolic models. However, connectionists rarely test this belief by interpreting the internal structure of their nets. A new approach to interpreting networks was recently introduced by Berkeley et al. (1995). The current paper examines two implications of applying this method: (1) that the internal structure of a connectionist network can have a very classical appearance, and (2) that this interpretation can provide a cognitive theory that cannot be dismissed as a mere implementation.

The Story Gestalt: A model of knowledge-intensive processes in text comprehension

How are knowledge-intensive, text-comprehension processes computed? Specifically, how are (1) explicit propositions remembered correctly, (2) pronouns resolved, (3) coherence and prediction inferences drawn, (4) on-going interpretations revised as more information becomes available, and (5) information learned in specific contexts generalized to novel texts? A constraint satisfaction model is presented that offers a number of advantages over previous models: Each of the previous processes can be seen as examples of the same process of constraint satisfaction, constraints can have strengths to represent the degrees of correlation among information, and the independence of constraints provides insight into generalization. In the model, propositions describing a simple event, such as going to the beach or a restaurant, are sequentially presented to a recurrent PDP network. The model is trained through practice processing a large number of example texts and answering questions. Questions are predicates from propositions explicit or inferable from the text, and the model has to answer with the proposition that fits that predicate. The model learns to perform well, though some processes require substantial training. A second simulation shows how the combinatorics in the training corpus can increase generalization. This effect is explained by introducing the concept of identity and associative constraints that are learned from a corpus. Overall, the model provides a number of insights into how a graded constraint-satisfaction model can compute knowledge-intensive processes in text comprehension.

The Story Gestalt: A Model Of Knowledge‐Intensive Processes in Text Comprehension

How are knowledge‐intensive, text‐comprehension processes computed? Specifically, how are (1) explicit propositions remembered correctly, (2) pronouns resolved, (3) coherence and prediction inferences drawn, (4) on‐going interpretations revised as more information becomes available, and (5) information learned in specific contexts generalized to novel texts? A constraint satisfaction model is presented that offers a number of advantages over previous models: Each of the previous processes can be seen as examples of the same process of constraint satisfaction, constraints can have strengths to represent the degrees of correlation among information, and the independence of constraints provides insight into generalization. In the model, propositions describing a simple event, such as going to the beach or a restaurant, are sequentially presented to a recurrent PDP network. The model is trained through practice processing a large number of example texts and answering questions. Questions are predicates from propositions explicit or inferable from the text, and the model has to answer with the proposition that fits that predicate. The model learns to perform well, though some processes require substantial training. A second simulation shows how the combinatorics in the training corpus can increase generalization. This effect is explained by introducing the concept of identity and associative constraints that are learned from a corpus. Overall, the model provides a number of insights into how a graded constraint‐satisfaction model can compute knowledge‐intensive processes in text comprehension.

Natural language processing with modular pdp networks and distributed lexicon

An approach to connectionist natural language processing is proposed, which is based on hierarchically organized modular parallel distributed processing (PDP) networks and a central lexicon of distributed input/output representations. The modules communicate using these representations, which are global and publicly available in the system. The representations are developed automatically by all networks while they are learning their processing tasks. The resulting representations reflect the regularities in the subtasks, which facilitates robust processing in the face of noise and damage, supports improved generalization, and provides expectations about possible contexts. The lexicon can be extended by cloning new instances of the items, that is, by generating a number of items with known processing properties and distinct identities. This technique combinatorially increases the processing power of the system. The recurrent FGREP module, together with a central lexicon, is used as a basic building block in modeling higher level natural language tasks. A single module is used to form case-role representations of sentences from word-by-word sequential natural language input. A hierarchical organization of four recurrent FGREP modules (the DISPAR system) is trained to produce fully expanded paraphrases of script-based stories, where unmentioned events and role fillers are inferred.

Natural Language Processing With Modular Pdp Networks and Distributed Lexicon

An approach to connectionist natural language processing is proposed, which is based on hierarchically organized modular parallel distributed processing (PDP) networks and a central lexicon of distributed input/output representations. The modules communicate using these representations, which are global and publicly available in the system. The representations are developed automatically by all networks while they are learning their processing tasks. The resulting representations reflect the regularities in the subtasks, which facilitates robust processing in the face of noise and damage, supports improved generalization, and provides expectations about possible contexts. The lexicon can be extended by cloning new instances of the items, that is, by generating a number of items with known processing properties and distinct identities. This technique combinatorially increases the processing power of the system. The recurrent FGREP module, together with a central lexicon, is used as a basic building block in modeling higher level natural language tasks. A single module is used to form case‐role representations of sentences from word‐by‐word sequential natural language input. A hierarchical organization of four recurrent FGREP modules (the DISPAR system) is trained to produce fully expanded paraphrases of script‐based stories, where unmentioned events and role fillers are inferred.

Intracellular signalling as a parallel distributed process

Living cells respond to their environment by means of an interconnected network of receptors, second messengers, protein kinases and other signalling molecules. This article suggests that the performance of cell signalling pathways taken as a whole has similarities to that of the parallel distributed process networks (PDP networks) used in computer-based pattern recognition. Using the response of hepatocytes to glucagon as an example, a procedure is described by which a PDP network could simulate a cell signalling pathway. This procedure involves the following steps: (a) a bounded set of molecules is defined that carry the signals of interest; (b) each of these molecules is represented by a PDP-type of unit, with input and output functions and connection weights corresponding to specific biochemical parameters; (c) a "learning algorithm" is applied in which small random changes are made in the parameters of the cell signalling units and the new network is then tested by a selection procedure in favour of a specific input-output relationship. The analogy with PDP networks shows how living cells can recognize combinations of environmental influences, how cell responses can be stabilized and made resistant to damage, and how novel cell signalling pathways might appear during evolution.