Parallel Distributed Processing Approach Research Articles

A Feedforward Interactive Network to Explain Complex Information Processing Phenomena Including the Halo Effect and Its Generalization

Abstract Parallel interactive processing (PIP) is a parallel distributed processing approach in which a representation of an object involves interaction of its components in a way that influences information processing of the components. Prior research is summarized in the areas of categorization and probability judgment. It was found that an explanation of the halo effect and its generalization as well as phenomena in other information processing situations could be explained by PIP. The explanation of the halo effect is largely in terms of reduction of variance in judgments of components of an object produced by a holistic impression of the object. There was a focus on processes common to various subject areas.

Neural Population Dynamics and Cognitive Function.

Representations in the brain are encoded as patterns of activity of large populations of neurons. The science of population encoded representations, also known as parallel distributed processing (PDP), achieves neurological verisimilitude and has been able to account for a large number of cognitive phenomena in normal people, including reaction times (and reading latencies), stimulus recognition, the effect of stimulus salience on attention, perceptual invariance, simultaneous egocentric and allocentric visual processing, top-down/bottom-up processing, language errors, the effect of statistical regularities of experience, frequency, and age of acquisition, instantiation of rules and symbols, content addressable memory and the capacity for pattern completion, preservation of function in the face of noisy or distorted input, inference, parallel constraint satisfaction, the binding problem and gamma coherence, principles of hippocampal function, the location of knowledge in the brain, limitations in the scope and depth of knowledge acquired through experience, and Piagetian stages of cognitive development. PDP studies have been able to provide a coherent account for impairment in a variety of language functions resulting from stroke or dementia in a large number of languages and the phenomenon of graceful degradation observed in such studies. They have also made important contributions to our understanding of attention (including hemispatial neglect), emotional function, executive function, motor planning, visual processing, decision making, and neuroeconomics. The relationship of neural network population dynamics to electroencephalographic rhythms is starting to emerge. Nevertheless, PDP approaches have scarcely penetrated major areas of study of cognition, including neuropsychology and cognitive neuropsychology, as well as much of cognitive psychology. This article attempts to provide an overview of PDP principles and applications that addresses a broader audience.

The effect of semantic transparency on the processing of morphologically derived words: Evidence from decision latencies and event-related potentials.

Decomposition theories of morphological processing in visual word recognition posit an early morpho-orthographic parser that is blind to semantic information, whereas parallel distributed processing (PDP) theories assume that the transparency of orthographic-semantic relationships influences processing from the beginning. To test these alternatives, the performance of participants on transparent (foolish), quasi-transparent (bookish), opaque (vanish), and orthographic control words (bucket) was examined in a series of 5 experiments. In Experiments 1-3 variants of a masked priming lexical-decision task were used; Experiment 4 used a masked priming semantic decision task, and Experiment 5 used a single-word (nonpriming) semantic decision task with a color-boundary manipulation. In addition to the behavioral data, event-related potential (ERP) data were collected in Experiments 1, 2, 4, and 5. Across all experiments, we observed a graded effect of semantic transparency in behavioral and ERP data, with the largest effect for semantically transparent words, the next largest for quasi-transparent words, and the smallest for opaque words. The results are discussed in terms of decomposition versus PDP approaches to morphological processing. (PsycINFO Database Record

Do arbitrary input–output mappings in parallel distributed processing networks require localist coding?

ABSTRACTThe Parallel Distributed Processing (PDP) approach to cognitive modelling assumes that knowledge is distributed across multiple processing units. This view is typically justified on the basis of the computational advantages and biological plausibility of distributed representations. However, both these assumptions have been challenged. First, there is growing evidence that some neurons respond to information in a highly selective manner. Second, it has been demonstrated that localist representations are better suited for certain computational tasks. In this paper, we continue this line of research by investigating whether localist representations are learned in tasks involving arbitrary input–output mappings. The results imply that the pressure to learn local codes in such tasks is weak, but still there are conditions under which feed-forward PDP networks learn localist representation. Our findings further challenge the assumption that PDP modelling always goes hand in hand with distributed representations and provide directions for future research.

Connecting functional brain imaging and Parallel Distributed Processing

Functional neuroimaging and Parallel Distributed Processing (PDP) theory, both introduced to cognitive science in the 1980s, led to influential research programmes that have proceeded in parallel with little mutual influence. The PDP approach advanced specific claims about the nature of neural representations that, perhaps surprisingly, have gone largely untested in functional brain imaging. One reason may be the widespread use of univariate statistical methods for analysing brain imaging data, which typically rely on assumptions that render them unable to detect distributed representations of the kind that PDP predicts. More recent multivariate methods for image analysis may be better suited to detecting such representations. In the current article, we consider why univariate methods have been insufficient to test PDP's representational claims, articulate some of the properties that neural representations ought to have if the PDP view is valid and then survey the recent neuroimaging literature for evidence that neural representations do or do not have these properties. The survey establishes that the PDP view of distributed representations has considerable evidential support. This analysis underscores the importance of understanding how the assumptions underlying methods for analysing functional imaging data constrain the kinds of questions that can be addressed. We then consider the implications for our developing understanding of the neural bases of cognition and for the design of future brain imaging studies.

Computational modeling of reading in semantic dementia: Comment on Woollams, Lambon Ralph, Plaut, and Patterson (2007).

Woollams, Lambon Ralph, Plaut, and Patterson reported detailed data on reading in 51 cases of semantic dementia. They simulated some aspects of these data using a connectionist parallel distributed processing (PDP) triangle model of reading. We argue here that a different model of reading, the dual route cascaded (DRC) model of Coltheart, Rastle, Perry, Langdon, and Ziegler (2001), not only provides a more accurate simulation of these aspects of reading in semantic dementia than does the PDP model but also provides highly accurate simulations of other aspects of reading in this disorder that the PDP approach has not simulated. We conclude that our findings add to evidence both from simulations of normal skilled reading and from simulations of other kinds of acquired dyslexia that the nonconnectionist DRC model of reading offers a better account of normal and disordered reading than the connectionist PDP models of reading.

On the biological plausibility of grandmother cells: Implications for neural network theories in psychology and neuroscience.

A fundamental claim associated with parallel distributed processing (PDP) theories of cognition is that knowledge is coded in a distributed manner in mind and brain. This approach rejects the claim that knowledge is coded in a localist fashion, with words, objects, and simple concepts (e.g. "dog"), that is, coded with their own dedicated representations. One of the putative advantages of this approach is that the theories are biologically plausible. Indeed, advocates of the PDP approach often highlight the close parallels between distributed representations learned in connectionist models and neural coding in brain and often dismiss localist (grandmother cell) theories as biologically implausible. The author reviews a range a data that strongly challenge this claim and shows that localist models provide a better account of single-cell recording studies. The author also contrast local and alternative distributed coding schemes (sparse and coarse coding) and argues that common rejection of grandmother cell theories in neuroscience is due to a misunderstanding about how localist models behave. The author concludes that the localist representations embedded in theories of perception and cognition are consistent with neuroscience; biology only calls into question the distributed representations often learned in PDP models.

A fundamental limitation of the conjunctive codes learned in PDP models of cognition: Comment on Botvinick and Plaut (2006).

A central claim shared by most recent models of short-term memory (STM) is that item knowledge is coded independently from order in long-term memory (LTM; e.g., the letter A is coded by the same representational unit whether it occurs at the start or end of a sequence). Serial order is computed by dynamically binding these item codes to a separate representation of order. By contrast, Botvinick and Plaut developed a parallel distributed processing (PDP) model of STM that codes for item-order information conjunctively, such that the same letter in different positions is coded differently in LTM. Their model supports a wide range of memory phenomena, and critically, STM is better for lists that include high, as opposed to low, sequential dependencies (e.g., bigram effects). Models with context-independent item representations do not currently account for sequential effects. However, we show that their PDP model is too sensitive to these effects. A modified version of the model does better but still fails in important respects. The successes and failures can be attributed to a fundamental constraint associated with context-dependent representations. We question the viability of conjunctive coding schemes to support STM and take these findings as problematic for the PDP approach to cognition more generally.

Précis ofSemantic Cognition: A Parallel Distributed Processing Approach

AbstractIn this précis of our recent book,Semantic Cognition: A Parallel Distributed Processing Approach(Rogers & McClelland 2004), we present a parallel distributed processing theory of the acquisition, representation, and use of human semantic knowledge. The theory proposes that semantic abilities arise from the flow of activation among simple, neuron-like processing units, as governed by the strengths of interconnecting weights; and that acquisition of new semantic information involves the gradual adjustment of weights in the system in response to experience. These simple ideas explain a wide range of empirical phenomena from studies of categorization, lexical acquisition, and disordered semantic cognition. In this précis we focus on phenomena central to the reaction against similarity-based theories that arose in the 1980s and that subsequently motivated the “theory-theory” approach to semantic knowledge. Specifically, we consider (1) how concepts differentiate in early development, (2) why some groupings of items seem to form “good” or coherent categories while others do not, (3) why different properties seem central or important to different concepts, (4) why children and adults sometimes attest to beliefs that seem to contradict their direct experience, (5) how concepts reorganize between the ages of 4 and 10, and (6) the relationship between causal knowledge and semantic knowledge. The explanations our theory offers for these phenomena are illustrated with reference to a simple feed-forward connectionist model. The relationships between this simple model, the broader theory, and more general issues in cognitive science are discussed.

Structured models of semantic cognition

AbstractRogers & McClelland (R&M) criticize models that rely on structured representations such as categories, taxonomic hierarchies, and schemata, but we suggest that structured models can account for many of the phenomena that they describe. Structured approaches and parallel distributed processing (PDP) approaches operate at different levels of analysis, and may ultimately be compatible, but structured models seem more likely to offer immediate insight into many of the issues that R&M discuss.

A simple model from a powerful framework that spans levels of analysis

AbstractThe commentaries reflect three core themes that pertain not just to our theory, but to the enterprise of connectionist modeling more generally. The first concerns the relationship between a cognitive theory and an implemented computer model. Specifically, how does one determine, when a model departs from the theory it exemplifies, whether the departure is a useful simplification or a critical flaw? We argue that the answer to this question depends partially upon the model's intended function, and we suggest that connectionist models have important functions beyond the commonly accepted goals of fitting data and making predictions. The second theme concerns perceived in-principle limitations of the connectionist approach to cognition, and the specific concerns these perceived limitations raise for our theory. We argue that the approach is not in fact limited in the ways our critics suggest. One common misconception, that connectionist models cannot address abstract or relational structure, is corrected through new simulations showing directly that such structure can be captured. The third theme concerns the relationship between parallel distributed processing (PDP) models and structured probabilistic approaches. In this case we argue that there the difference between the approaches is not merely one of levels. Our PDP approach differs from structured statistical approaches at all of Marr's levels, including the characterization of the goals of cognitive computations, and of the representations and algorithms used.

Classic identity negative priming involves accessing semantic representations in the left anterior temporal cortex

Classic identity negative priming (NP) refers to the finding that when an object is ignored, subsequent naming responses to it are slower than when it has not been previously ignored (Tipper, S.P., 1985. The negative priming effect: inhibitory priming by ignored objects. Q. J. Exp. Psychol. 37A, 571–590). It is unclear whether this phenomenon arises due to the involvement of abstract semantic representations that the ignored object accesses automatically. Contemporary connectionist models propose a key role for the anterior temporal cortex in the representation of abstract semantic knowledge (e.g., McClelland, J.L., Rogers, T.T., 2003. The parallel distributed processing approach to semantic cognition. Nat. Rev. Neurosci. 4, 310–322), suggesting that this region should be involved during performance of the classic identity NP task if it involves semantic access. Using high-field (4 T) event-related functional magnetic resonance imaging, we observed increased BOLD responses in the left anterolateral temporal cortex including the temporal pole that was directly related to the magnitude of each individual’s NP effect, supporting a semantic locus. Additional signal increases were observed in the supplementary eye fields (SEF) and left inferior parietal lobule (IPL).

From Acquisition to Deterioration of Semantic Knowledge: A Mechanistic Theory of Semantic Cognition

Semantic Cognition: A Parallel Distributed Processing Approach, by Timothy T. Rogers and James L. McClelland. (2004). Cambridge, MA: MIT Press, 425 pp., $50.00/£32.95.This book is aimed at all scientists interested in semantic cognition, even those with limited experience of computational modeling. It benefits from concise and logically presented text and the effective use of figures and illustrations, with details of simulations tucked away in the Appendix. The length of the book defines it not only as a useful addition to a reference collection, but also an excellent primer for young researchers, postgraduates, and even advanced undergraduates.

The parallel distributed processing approach to semantic cognition.

How do we know what properties something has, and which of its properties should be generalized to other objects? How is the knowledge underlying these abilities acquired, and how is it affected by brain disorders? Our approach to these issues is based on the idea that cognitive processes arise from the interactions of neurons through synaptic connections. The knowledge in such interactive and distributed processing systems is stored in the strengths of the connections and is acquired gradually through experience. Degradation of semantic knowledge occurs through degradation of the patterns of neural activity that probe the knowledge stored in the connections. Simulation models based on these ideas capture semantic cognitive processes and their development and disintegration, encompassing domain-specific patterns of generalization in young children, and the restructuring of conceptual knowledge as a function of experience.

Measuring sensitivity moment-by-moment: A microanalytic look at the transmission of attachment

Meta-analyses have indicated that the mediating power of sensitivity in the transmission of attachment across generations is not nearly as great as would be predicted by attachment theory (van IJzendoorn, 1995; De Wolff & van IJzendoorn, 1997). To make sense of these findings, the author suggests that current measures of sensitivity should be expanded to include the assessment of mother-infant behavior on a micro-behavioral level. This paper reviews evidence for two central propositions: (1) The internal working model of mothers is expressed in subtle, fine-grained interactive behaviors with their infants, and (2) infants perceive and remember these behaviors in the form of interactional expectancies. Following these propositions, hypothesized representational micromodels for each major attachment organization are then offered, based on the Parallel Distributed Processing (PDP) approach to information processing. In closing, this paper presents some of the most pressing unresolved issues for attachment researchers as they seek to empirically identify the mechanisms by which attachment is transmitted across generations.

Challenging the widespread assumption that connectionism and distributed representations go hand-in-hand

One of the central claims associated with the parallel distributed processing approach popularized by D.E. Rumelhart, J.L. McClelland and the PDP Research Group is that knowledge is coded in a distributed fashion. Localist representations within this perspective are widely rejected. It is important to note, however, that connectionist networks can learn localist representations and many connectionist models depend on localist coding for their functioning. Accordingly, a commitment to distributed representations should be considered a specific theoretical claim regarding the structure of knowledge rather than a core principle, as often assumed. In this paper, it is argued that there are fundamental computational and empirical challenges that have not yet been addressed by distributed connectionist theories that are readily accommodated within localist approaches. This is highlighted in the context of modeling word and nonword naming, the domain in which some of the strongest claims have been made. It is shown that current PDP models provide a poor account of naming monosyllable items, and that distributed representations make it difficult for these models to scale up to more complex language phenomena. At the same time, models that learn localist representations are shown to hold promise in supporting many of the core reading and language functions on which PDP models fail. It is concluded that the common rejection of localist coding schemes within connectionist architectures is premature.

Activation and coherence in memory processes: Revisiting the Parallel Distributed Processing approach to retrieval

Connectionist models based on activation spreading and attractor dynamics are functionally limited by representational and processing flexibility constraints, the 'feature binding problem' and the need to balance accurately activation and inhibition. We suggest an alternative approach, in which network units are characterized by two variables: activation and phase. Whereas activation evolves according to a 'classical' connectionist rule, the phase variable is characterized by a chaotic evolution. We present a model of memory retrieval with reference to the paradigmatic McClelland's 1981 'Jets and Sharks' model. The model solves the 'multiple reinstantiation problem', i.e. the problem of retrieval of multiple items with overlapping features, implied by its classical predecessor. In our network, multiple pattern reinstantiation in terms of activation spreading is disambiguate through selective and differential coherence patterns. The system flexibly represents pattern similarity and feature relationships by means of graded and intermittent synchrony. The domain-general implications of this approach for connectionist 'interactive activation models' and its neurophysiological plausibility are discussed.

Deficits in irregular past-tense verb morphology associated with degraded semantic knowledge

Two distinct mechanisms are often considered necessary to account for generation of the past-tense of English verbs: a lexical associative process for irregular forms like speak → spoke, and a rule-governed process (‘add - ed’) for regular and novel forms like talk → talked and wug → wugged. An alternative account based on a parallel-distributed processing approach proposes that one complex procedure processes all past-tense types. In this alternative view, neuropsychological dissociations are explained by reduced input from word meaning that plays a greater role in successful generation of the past-tense for lower frequency irregular verbs, and by phonological deficits that disproportionately affect regular and novel forms. Only limited evidence has been available concerning the relationship between knowledge of word meaning and verb-tense processing. The study reported here evaluated the past-tense verb abilities of 11 patients with semantic dementia, a neurodegenerative condition characterised by degraded semantic knowledge. We predicted and confirmed that the patients would have essentially normal ability to generate and recognise regular (and novel) past-tense forms, but a marked and frequency-modulated deficit on irregular verbs. Across the set of 11 patients, the degree of impairment for the irregular past-tense was significantly correlated with the degree of comprehension impairment as measured by verb synonym judgements. These results, plus other features of the data such as the nature of the errors to irregular verbs, are discussed in relation to currently developing theories of the language system.

Neural networks for language identification: a comparative study

Since the advent of Jordan's recurrent network [Jordan, M. I. (1986) Serial Order: A Parallel Distributed Processing Approach. Tech. Rep. No. 8604. Institute for Cognitive Science, University of California, San Diego.] which allows the processing of data with a temporal component, neural networks have been used routinely for sequence processing. This type of network is analysed in this paper for its ability to discriminate between different languages based on its processing of a small sample of text. The motivation for developing this model was for its potential use in the on-line version of a Trinity College 1872 Printed Catalogue, a library catalogue which has entries in 14 different languages spanning over 5 centuries. It was thought that neural networks would perform well where entries to be analysed comprised only a few words. The neural network's performance was compared with that of trigrams and a suffix/morphology analysis. The trigrams proved to be superior, classifying over 92% of the entries correctly compared to 88% for the neural network and 85% for the morphology/suffix analysis. Trigrams were also far superior in the speed at which statistics were compiled and the rate at which text was processed.

Modelling anomia by the discrete two-stage word production architecture

Recent advancements in computational modelling have opened new ways to understand the underlying mechanisms of cognitive deficits resulting from brain damage. These implementations have followed interactive activation and parallel distributed processing approaches and the computational potential of more transparent and serial cognitive architectures has not received much attention. We tested the ability of the discrete two-stage word production model to account for naming response patterns in aphasia. Our implementation is a simple local connectionist model with two serially ordered networks representing lexical–semantic and phoneme levels. Anomic deficits were simulated by manipulating three parameters: noise within the two networks leading to semantically and phonologically based errors, and between-level threshold leading to no responses. Comparisons to actual naming data from 10 aphasics showed that this simple model provides a rather good fit to a variety of clinically observed naming response patterns. However, the existence of output-type semantic naming errors in some aphasics call for a modified discrete two-stage model which would possess monitoring mechanisms for lexical output.