Connectionist Architecture Research Articles

Research on calibration of a binocular stereo-vision imaging system based on the artificial neural network.

Camera calibration is a key problem for 3D reconstruction in computer vision. Existing calibration methods, such as traditional, active, and self-calibration, all need to solve the internal and external parameters of the imaging system to clarify the image-object mapping relationship. The artificial neural network, which is based on connectionist architecture, provides a novel idea for the calibration of nonlinear mapping vision systems. It can learn the image-object mapping relationship from some sample points without considering too many uncertain factors in the middle. This paper discusses the learning ability. A binocular stereo-vision mapping model is used as the learning model to explore the ability of image-object mapping for artificial neural networks. This paper constructs sample libraries by pixel and world coordinates of checkerboard corners, builds the artificial neural network, and, through the training samples and test samples prediction, verifies the learning performance of the network. Furthermore, by the laser scanning binocular vision device constructed in the authors' laboratory and trained-well network, the 3D point cloud reconstruction of a physical target is performed. The experimental results show that the artificial neural network can learn the image-object mapping relationship well and more effectively avoid the impact of lens distortion and achieve more accurate nonlinear mapping at the edge of the image. When the X and Y coordinates are in the range of 100mm and the Z coordinates are in the range of a 1000mm, the absolute error rarely exceeds 2.5mm, and the relative error is in the level of 10-3; for 1000mm distance measurement, the standard deviation does not exceed 1.5mm. Network parameter selection experiments show that, for image-object mapping, a three-layer network and increasing the number of hidden layer's nodes can improve the training time more significantly.

Catastrophic Interference in Predictive Neural Network Models of Distributional Semantics

The semantic memory literature has recently seen the emergence of predictive neural network models that use principles of reinforcement learning to create a “neural embedding” of word meaning when trained on a language corpus. These models have taken the field by storm, partially due to the resurgence of connectionist architectures, but also due to their remarkable success at fitting human data. However, predictive embedding models also inherit the weaknesses of their ancestors. In this paper, we explore the effect of catastrophic interference (CI), long known to be a flaw with neural network models, on a modern neural embedding model of semantic representation (word2vec). We use homonyms as an index of bias as a function of the order in which a corpus is learned. If the corpus is learned in random order, the final representation will tend towards the dominant sense of the word (bank ➔ money) as opposed to the subordinate sense (bank ➔ river). However, if the subordinate sense is presented to the network after learning the dominant sense, CI almost completely erases the dominant sense and the final representation strongly tends towards the more recent subordinate sense. We demonstrate the impact of CI and sequence of learning on the final neural embeddings learned by word2vec in both an artificial language and in an English corpus. Further, we evaluate the effectiveness of a recently proposed solution to CI from neuroscience, elastic weight consolidation, as well as a Hebbian learning architecture from the memory literature that does not produce an error signal on mitigating the effects of CI.

Evolutionary acquisition of complex traits in artificial epigenetic networks

How complex traits arise within organisms over evolutionary time is an important question that has relevance both to the understanding of biological systems and to the design of bio-inspired computing systems. This paper investigates the process of acquiring complex traits within epiNet, a recurrent connectionist architecture capable of adapting its topology during execution. Inspired by the biological processes of gene regulation and epigenetics, epiNet captures biological organisms’ ability to alter their regulatory topologies according to environmental stimulus. By applying epiNet to a series of computational tasks, each requiring a range of complex behaviours to solve, and capturing the evolutionary process in detail, we can show not only how the physical structure of epiNet changed when acquiring complex traits, but also how these changes in physical structure affected its dynamic behaviour. This is facilitated by using a lightweight optimisation method which makes minor iterative changes to the network structure so that when complex traits emerge for the first time, a direct lineage can be observed detailing exactly how they evolved. From this we can build an understanding of how complex traits evolve and which regulatory environments best allow for the emergence of these complex traits, pointing us towards computational models that allow more swift and robust acquisition of complex traits when optimised in an evolutionary computing setting.

Symbolic-Connectionist Representational Model for Optimizing Decision Making Behavior in Intelligent Systems

Modeling higher order cognitive processes like human decision making come in three representational approaches namely symbolic, connectionist and symbolic-connectionist. Many connectionist neural network models are evolved over the decades for optimizing decision making behaviors and their agents are also in place. There had been attempts to implement symbolic structures within connectionist architectures with distributed representations. Our work was aimed at proposing an enhanced connectionist approach of optimizing the decisions within the framework of a symbolic cognitive model. The action selection module of this framework is forefront in evolving intelligent agents through a variety of soft computing models. As a continous effort, a Connectionist Cognitive Model (CCN) had been evolved by bringing a traditional symbolic cognitive process model proposed by LIDA as an inspiration to a feed forward neural network model for optimizing decion making behaviours in intelligent agents. Significanct progress was observed while comparing its performance with other varients.

Artificial Epigenetic Networks: Automatic Decomposition of Dynamical Control Tasks Using Topological Self-Modification.

This paper describes the artificial epigenetic network, a recurrent connectionist architecture that is able to dynamically modify its topology in order to automatically decompose and solve dynamical problems. The approach is motivated by the behavior of gene regulatory networks, particularly the epigenetic process of chromatin remodeling that leads to topological change and which underlies the differentiation of cells within complex biological organisms. We expected this approach to be useful in situations where there is a need to switch between different dynamical behaviors, and do so in a sensitive and robust manner in the absence of a priori information about problem structure. This hypothesis was tested using a series of dynamical control tasks, each requiring solutions that could express different dynamical behaviors at different stages within the task. In each case, the addition of topological self-modification was shown to improve the performance and robustness of controllers. We believe this is due to the ability of topological changes to stabilize attractors, promoting stability within a dynamical regime while allowing rapid switching between different regimes. Post hoc analysis of the controllers also demonstrated how the partitioning of the networks could provide new insights into problem structure.

Pattern Turnover within Synaptically Perturbed Neural Systems

A critical level of synaptic perturbation within a trained, artificial neural system induces the nucleation of novel activation patterns, many of which could qualify as viable ideas or action plans. In building massively parallel connectionist architectures requiring myriad, coupled neural modules driven to ideate in this manner, the need has arisen to shift the attention of computational critics to only those portions of the neural “real estate” generating sufficiently novel activation patterns. The search for a suitable affordance to guide such attention has revealed that the rhythm of pattern generation by synaptically perturbed neural nets is a quantitative indicator of the novelty of their conceptual output, that cadence in turn characterized by a frequency and a corresponding temporal clustering that is discernible through fractal dimension. Anticipating that synaptic fluctuations are tantamount in effect to volume neurotransmitter release within cortex, a novel theory of both cognition and consciousness arises that is reliant upon the rate of transitions within cortical activation topologies.

Bayesian supervised learning and state estimation in a model of the cerebellum

The last decade has seen an explosion of research on the topic of Bayesian inference in neural systems. The cerebral cortex is now widely believed to form predictive internals models of the environment. But what does this mean for the cerebellum, which has historically been attributed just this function? And what might Bayesian inference mean in the cerebellar context? I present a cerebellar model unifying the Bayesian framework with classical cerebellar theories of supervised learning, state estimation, and control theory. This model builds on the dominant supervised learning theory of cerebellar function, which traces to the pioneering work of Marr. This model is extended to account for Bayesian optimal updating of the uncertainties inherent in probabilistic inference. Further, the model is capable of learning about states with little to no supervised input. It also provides theoretical explanation for a wide range of phenomenology observed of the circuit -- most importantly unsupervised synaptic plasticity upstream of cerebellar Purkinje cells. The connectionist architecture of this model is closely based on that of the cerebellum. In the input stage, unsupervised learning in the cerebellar granular layer is seen as essential for subsequent supervised learning in the Purkinje layer. Based on Golgi cell feedback, the granular layer learns to encode supervised learning covariances in a manner akin to the kernel learning of Gaussian processes. This circuit then predicts and filters mossy fiber input, conveying only select signals downstream for supervised learning. This granular representation is further influenced by a global serotonergic signal conveyed by Lugaro cells. In the cerebellar molecular layer, inhibitory interneurons then provide higher-order corrections to these estimates. Finally, learning in the cerebellar nuclei is seen as a parallel, simpler, regression model, allowing for faster, courser adjustments, corrected in time by the cerebellar cortical circuit.

Lexical is as lexical does: computational approaches to lexical representation

In much of neuroimaging and neuropsychology, regions of the brain have been associated with ‘lexical representation’, with little consideration as to what this cognitive construct actually denotes. Within current computational models of word recognition, there are a number of different approaches to the representation of lexical knowledge. Structural lexical representations, found in original theories of word recognition, have been instantiated in modern localist models. However, such a representational scheme lacks neural plausibility in terms of economy and flexibility. Connectionist models have therefore adopted distributed representations of form and meaning. Semantic representations in connectionist models necessarily encode lexical knowledge. Yet when equipped with recurrent connections, connectionist models can also develop attractors for familiar forms that function as lexical representations. Current behavioural, neuropsychological and neuroimaging evidence shows a clear role for semantic information, but also suggests some modality- and task-specific lexical representations. A variety of connectionist architectures could implement these distributed functional representations, and further experimental and simulation work is required to discriminate between these alternatives. Future conceptualisations of lexical representations will therefore emerge from a synergy between modelling and neuroscience.

What can we learn from learning models about sensitivity to letter-order in visual word recognition?

Recent research on the effects of letter transposition in Indo-European Languages has shown that readers are surprisingly tolerant of these manipulations in a range of tasks. This evidence has motivated the development of new computational models of reading that regard flexibility in positional coding to be a core and universal principle of the reading process. Here we argue that such approach does not capture cross-linguistic differences in transposed-letter effects, nor do they explain them. To address this issue, we investigated how a simple domain-general connectionist architecture performs in tasks such as letter-transposition and letter substitution when it had learned to process words in the context of different linguistic environments. The results show that in spite of of the neurobiological noise involved in registering letter-position in all languages, flexibility and inflexibility in coding letter order is also shaped by the statistical orthographic properties of words in a language, such as the relative prevalence of anagrams. Our learning model also generated novel predictions for targeted empirical research, demonstrating a clear advantage of learning models for studying visual word recognition.

Biochemical connectionism

In this paper, we discuss computational architectures that are motivated by connectionist patterns that occur in biochemical networks, and speculate about how this biochemical approach to connectionism might complement conventional neural approaches. In particular, we focus on three features of biochemical networks that make them distinct from neural networks: their diverse, complex nodal processes, their emergent organisation, and the dynamical behaviours that result from higher-order, self-modifying processes. We also consider the growing use of evolutionary algorithms in the design of connectionist systems, noting how this enables us to explore a wider range of connectionist architectures, and how the close relationship between biochemical networks and biological evolution can guide us in this endeavour.

Representing, binding, retrieving and unifying relational knowledge using pools of neural binders

Abstract For a long time, connectionist architectures have been criticized for having propositional fixation, lack of compositionality and, in general, for their weakness in representing sophisticated symbolic information and processing it. This work offers a novel approach that allows full integration of symbolic AI with the connectionist paradigm. We show how to encode and process relational knowledge using Artificial Neural Networks (ANNs), such as Boltzmann Machines. The neural architecture uses a working memory (WM), consisting of pools of “binders”, and a long-term synaptic-memory that can store a large relational knowledge-base (KB). A compact variable binding mechanism is proposed which dynamically allocates ensembles of neurons when a query is clamped; retrieving KB items till a solution emerges in the WM. We illustrate the proposal through non-trivial predicate unification problems: knowledge items are only retrieved into the WM upon need, and unified, graph-like structures emerge at equilibrium as an activation pattern of the neural network. Our architecture is based on the fact that some attractor-based ANNs may be viewed as performing constraint satisfaction, where, at equilibrium, fixed-points maximally satisfy a set of weighted constraints. We show how to encode relational graphs as neural activation in WM and how to use constraints that are encoded in synapses, in order to retrieve and process such complex structures. Both procedural (the unification algorithm) and declarative knowledge (logic formulae) are first expressed as constraints and then used to generate (or learn) weighted synaptic connections. The architecture has no central control and is inherently robust to unit failures. Contrary to previous connectionist suggestions, this approach is expressive, compact, accurate, and goal directed. The mechanism is universal and has a simple underlying computational principle. As such, it may be further adapted for applications that combine the advantages of both connectionist and traditional symbolic AI and may be used in modeling aspects of human reasoning.

A Multimodal Connectionist Architecture for Unsupervised Grounding of Spatial Language

We propose a bio-inspired unsupervised connectionist architecture and apply it to grounding the spatial phrases. The two-layer architecture combines by concatenation the information from the visual and the phonological inputs. In the first layer, the visual pathway employs separate ‘what’ and ‘where’ subsystems that represent the identity and spatial relations of two objects in 2D space, respectively. The bitmap images are presented to an artificial retina and the phonologically encoded five-word sentences describing the image serve as the phonological input. The visual scene is hence represented by several self-organizing maps (SOMs) and the phonological description is processed by the Recursive SOM that learns to topographically represent the spatial phrases, represented as five-word sentences (e.g., ‘blue ball above red cup’). Primary representations from the first-layer modules are unambiguously integrated in a multimodal second-layer module, implemented by the SOM or the ‘neural gas’ algorithms. The system learns to bind proper lexical and visual features without any prior knowledge. The simulations reveal that separate processing and representation of the spatial location and the object shape significantly improve the performance of the model. We provide quantitative experimental results comparing three models in terms of their accuracy.

Optimizing the operating conditions in a high precision industrial process using soft computing techniques

AbstractThis interdisciplinary research is based on the application of unsupervized connectionist architectures in conjunction with modelling systems and on the determining of the optimal operating conditions of a new high precision industrial process known as laser milling. Laser milling is a relatively new micro‐manufacturing technique in the production of high‐value industrial components. The industrial problem is defined by a data set relayed through standard sensors situated on a laser‐milling centre, which is a machine tool for manufacturing high‐value micro‐moulds, micro‐dies and micro‐tools. The new three‐phase industrial system presented in this study is capable of identifying a model for the laser‐milling process based on low‐order models. The first two steps are based on the use of unsupervized connectionist models. The first step involves the analysis of the data sets that define each case study to identify if they are informative enough or if the experiments have to be performed again. In the second step, a feature selection phase is performed to determine the main variables to be processed in the third step. In this last step, the results of the study provide a model for a laser‐milling procedure based on low‐order models, such as black‐box, in order to approximate the optimal form of the laser‐milling process. The three‐step model has been tested with real data obtained for three different materials: aluminium, cooper and hardened steel. These three materials are used in the manufacture of micro‐moulds, micro‐coolers and micro‐dies, high‐value tools for the medical and automotive industries among others. As the model inputs are standard data provided by the laser‐milling centre, the industrial implementation of the model is immediate. Thus, this study demonstrates how a high precision industrial process can be improved using a combination of artificial intelligence and identification techniques.

A probabilistic fuzzy approach to modeling nonlinear systems

Recently, the study of incorporating probability theory and fuzzy logic has received much interest. To endow the traditional fuzzy rule-based systems (FRBs) with probabilistic features to handle randomness, this paper presents a probabilistic fuzzy neural network (ProFNN) by introducing the probability of input linguistic terms and providing linguistic meaning into the connectionist architecture. ProFNN integrates the probabilistic information of fuzzy rules into the antecedent parts and quantifies the impacts of the rules on the consequent parts using mutual subsethood, which work in conjunction with volume defuzzification in a gradient descent learning frame work. Despite the increase in the number of parameters, ProFNN provides a promising solution to deal with randomness and fuzziness in a single frame. To evaluate the performance and applicability of the proposed approach, ProFNN is carried out on various benchmarking problems and compared with other existing models with a performance better than most of them.

The Cortexionist architecture: behavioural intelligence of artificial creatures

Traditionally, producing intelligent behaviours for artificial creatures involves modelling their cognitive abilities. This approach raises two problems. On the one hand, defining manually the agent’s knowledge is a heavy and error-prone task that implies the intervention of the animator. On the other hand, the relationship between cognition and intelligence has not been theoretically nor experimentally proven so far. The ecological approaches provide a solution for these problems, by exploring the links between the creature, its body and its environment. Using an artificial life approach, we propose an original model of memory based on the synthesis of several neuroscience theories. The Cortexionist controller integrates cortex-like structure into a connectionist architecture in order to enhance the agent’s adaptation in a dynamic environment, ultimately leading to the emergence of intelligent behaviour. Initial experiments presented in this paper prove the validity of the model.

A connectionist approach to embodied conceptual metaphor.

A growing body of data has been gathered in support of the view that the mind is embodied and that cognition is grounded in sensory-motor processes. Some researchers have gone so far as to claim that this paradigm poses a serious challenge to central tenets of cognitive science, including the widely held view that the mind can be analyzed in terms of abstract computational principles. On the other hand, computational approaches to the study of mind have led to the development of specific models that help researchers understand complex cognitive processes at a level of detail that theories of embodied cognition (EC) have sometimes lacked. Here we make the case that connectionist architectures in particular can illuminate many surprising results from the EC literature. These models can learn the statistical structure in their environments, providing an ideal framework for understanding how simple sensory-motor mechanisms could give rise to higher-level cognitive behavior over the course of learning. Crucially, they form overlapping, distributed representations, which have exactly the properties required by many embodied accounts of cognition. We illustrate this idea by extending an existing connectionist model of semantic cognition in order to simulate findings from the embodied conceptual metaphor literature. Specifically, we explore how the abstract domain of time may be structured by concrete experience with space (including experience with culturally specific spatial and linguistic cues). We suggest that both EC researchers and connectionist modelers can benefit from an integrated approach to understanding these models and the empirical findings they seek to explain.

The application of a two-step AI model to an automated pneumatic drilling process

Real-world processes may be improved through a combination of artificial intelligence and identification techniques. This work presents a multidisciplinary study that identifies and applies unsupervised connectionist models in conjunction with modelling systems. This particular industrial problem is defined by a data set relayed through sensors situated on a robotic drill used in the construction of industrial storage centres. The first step entails determination of the most relevant structures in the data set with the application of the connectionist architectures. The second step combines the results of the first one to identify a model for the optimal working conditions of the drilling robot that is based on low-order models such as black box that approximate the optimal form of the model. Finally, it is shown that the most appropriate model to control these industrial tasks is the Box–Jenkins algorithm, which calculates the function of a linear system from its input and output samples.

Prediction of arm trajectory from the neural activities of the primary motor cortex with modular connectionist architecture

In our previous study [Koike, Y., Hirose, H., Sakurai, Y., Iijima T., (2006). Prediction of arm trajectory from a small number of neuron activities in the primary motor cortex. Neuroscience Research, 55, 146-153], we succeeded in reconstructing muscle activities from the offline combination of single neuron activities recorded in a serial manner in the primary motor cortex of a monkey and in reconstructing the joint angles from the reconstructed muscle activities during a movement condition using an artificial neural network. However, the joint angles during a static condition were not reconstructed. The difficulties of reconstruction under both static and movement conditions mainly arise due to muscle properties such as the velocity-tension relationship and the length-tension relationship. In this study, in order to overcome the limitations due to these muscle properties, we divided an artificial neural network into two networks: one for movement control and the other for posture control. We also trained the gating network to switch between the two neural networks. As a result, the gating network switched the modules properly, and the accuracy of the estimated angles improved compared to the case of using only one artificial neural network.

Connectionist Approach to Generic Text Summarization

As the enormous amount of on-line text grows on the World-Wide Web, the development of methods for automatically summarizing this text becomes more important. The primary goal of this research is to create an efficient tool that is able to summarize large documents automatically. We propose an Evolving connectionist System that is adaptive, incremental learning and knowledge representation system that evolves its structure and functionality. In this paper, we propose a novel approach for Part of Speech disambiguation using a recurrent neural network, a paradigm capable of dealing with sequential data. We observed that connectionist approach to text summarization has a natural way of learning grammatical structures through experience. Experimental results show that our approach achieves acceptable performance. Keywords—Artificial Neural Networks (ANN); Computational Intelligence (CI); Connectionist Text Summarizer ECTS (ECTS); Evolving Connectionist systems; Evolving systems; Fuzzy systems (FS); Part of Speech (POS) disambiguation. I. INTRODUCTION VER the past half a century, the problem of text summarization has been addressed from many different perspective, in various domains and using various paradigms. This paper intends to investigate Connectionist architecture for the Text Summarization system, taking into account of existing new developments in adaptive evolving systems. Evolving processes, through both individual development and evolution, inexorably led the human race to our supreme intelligence and our superior position in the animal kingdom (2) (3). In this paper, we consider the system of an Automatic Text Summarization as Evolving system which learns incrementally through experience in the environment. This paper highlights the practical experiences, Connectionist learning environment and new ideas to promote further validations.

A face emotion tree structure representation with probabilistic recursive neural network modeling

This paper describes a novel structural approach to recognize the human facial features for emotion recognition. Conventionally, features extracted from facial images are represented by relatively poor representations, such as arrays or sequences, with a static data structure. In this study, we propose to extract facial expression features vectors as Localized Gabor Features (LGF) and then transform these feature vectors into FacE Emotion Tree Structures (FEETS) representation. It is an extension of the Human Face Tree Structures (HFTS) representation presented in (Cho and Wong in Lecture notes in computer science, pp 1245–1254, 2005). This facial representation is able to simulate as human perceiving the real human face and both the entities and relationship could contribute to the facial expression features. Moreover, a new structural connectionist architecture based on a probabilistic approach to adaptive processing of data structures is presented. The so-called probabilistic based recursive neural network (PRNN) model extended from Frasconi et al. (IEEE Trans Neural Netw 9:768–785, 1998) is developed to train and recognize human emotions by generalizing the FEETS representation. For empirical studies, we benchmarked our emotion recognition approach against other well known classifiers. Using the public domain databases, such as Japanese Female Facial Expression (JAFFE) (Lyons et al. in IEEE Trans Pattern Anal Mach Intell 21(12):1357–1362, 1999; Lyons et al. in third IEEE international conference on automatic face and gesture recognition, 1998) database and Cohn–Kanade AU-Coded Facial Expression (CMU) Database (Cohn et al. in 7th European conference on facial expression measurement and meaning, 1997), our proposed system might obtain an accuracy of about 85–95% for subject-dependent and subject-independent conditions. Moreover, by testing images having artifacts, the proposed model significantly supports the robust capability to perform facial emotion recognition.