Sigma-pi Units Research Articles

A self-organizing map of sigma–pi units

By frame of reference transformations, an input variable in one coordinate system is transformed into an output variable in a different coordinate system depending on another input variable. If the variables are represented as neural population codes, then a sigma–pi network is a natural way of coding this transformation. By multiplying two inputs it detects coactivations of input units, and by summing over the multiplied inputs, one output unit can respond invariantly to different combinations of coactivated input units. Here, we present a sigma–pi network and a learning algorithm by which the output representation self-organizes to form a topographic map. This network solves the frame of reference transformation problem by unsupervised learning.

Traitement des règles graduelles au moyen des systèmes hybrides neuro-symboliques

This work belongs to the field of hybrid systems for Artificial Intelligence (Al). It concerns the study of gradual rules, which makes it possible to represent correlations and modulation relations between variables. We propose a set of characteristics to identify these gradual rules, and a classification into direct rules and modulation rules. In neurobiology, pre-synaptic neuronal connections lead to gradual processing and modulation of cognitive information. We propose in the field of connectionism the use of Sigma-Pi connections to allow gradual processing in AI systems. In order to represent as well as possible the modulation processes between the inputs of a network, we have created a new type of connection, Asymmetric Sigma-Pi (ASP) units. These models have been implemented within a pre-existing hybrid neuro-symbolic system, the INSS system. The new hybrid system thus obtained, INSS-Gradual, allows the learning of bases of examples containing gradual modulation relations. ASP units facilitate the extraction of gradual rules from a neural network.

Elman topology with sigma–pi units: An application to the modeling of verbal hallucinations in schizophrenia

The development of neural network models has greatly enhanced the comprehension of cognitive phenomena. Here, we show that models using multiplicative processing of inputs are both powerful and simple to train and understand. We believe they are valuable tools for cognitive explorations. Our model can be viewed as a subclass of networks built on sigma–pi units and we show how to derive the Kronecker product representation from the classical sigma–pi unit. We also show how the connectivity requirements of the Kronecker product can be relaxed considering statistical arguments. We use the multiplicative network to implement what we call an Elman topology, that is, a simple recurrent network (SRN) that supports aspects of language processing. As an application, we model the appearance of hallucinated voices after network damage, and show that we can reproduce results previously obtained with SRNs concerning the pathology of schizophrenia.

Note on interpolation on the hypercube by means of sigma–pi neural networks

In this note, we sketch the construction of a quite easy interpolation process on the hypercube, which can be put into the framework of a three-layer feed-forward neural network based on sigma–pi units. Detailed learning and recall algorithms and concrete examples are also included.

Transformations of sigma–pi nets: obtaining reflected functions by reflecting weight matrices

This paper presents a methodology that reflected functions by reflecting the weight matrices of an artificial neural network. One of the major problems with the connectionist approach is that trained neural networks can only associate fixed sets of input–output mappings. We provide a methodology which allows the post-trained net to associate different input–output mappings. The different mappings are reflected in a horizontal axis, reflected in a vertical axis and scaling of the initial mapping. The methodology does not train the net on the different mappings but it transforms the weight matrix of the neural network. This paper describes a novel way of utilising sigma–pi neural networks. Our new methodology manipulates sigma–pi unit's weight matrices which transform the unit's output. The weights are cast in a matrix formulation, and then transformations can be performed on the weight matrix of the sigma–pi net. To test the new methodology, the following three steps were carried out on a neural network: (1) the network was trained to perform a mapping function, f; (2) the weights of the network were transformed; and (3) the network was tested to evaluate whether it performs the reflection in the vertical axis, f ref−vert( x)= a− f( x). This reflects the function in one dimension. A reflection transformation was used to manipulate the network's weight matrices to obtain a reflection in the vertical axis. Note that the network was not trained to perform the reflection in the vertical axis. The transformation of the weight matrix transformed the function the output performs. This article explains the theory which enables us to perform transformations of sigma–pi networks and obtain reflections of the output by reflecting the weight matrices. These transforms empower the network to perform related mapping tasks once one mapping task has been learnt. This article explains how each transformation is performed and it considers whether a set of ‘standard’ transformations can indeed be derived.

On the complexity of computing and learning with multiplicative neural networks.

In a great variety of neuron models, neural inputs are combined using the summing operation. We introduce the concept of multiplicative neural networks that contain units that multiply their inputs instead of summing them and thus allow inputs to interact nonlinearly. The class of multiplicative neural networks comprises such widely known and well-studied network types as higher-order networks and product unit networks. We investigate the complexity of computing and learning for multiplicative neural networks. In particular, we derive upper and lower bounds on the Vapnik-Chervonenkis (VC) dimension and the pseudo-dimension for various types of networks with multiplicative units. As the most general case, we consider feedforward networks consisting of product and sigmoidal units, showing that their pseudo-dimension is bounded from above by a polynomial with the same order of magnitude as the currently best-known bound for purely sigmoidal networks. Moreover, we show that this bound holds even when the unit type, product or sigmoidal, may be learned. Crucial for these results are calculations of solution set components bounds for new network classes. As to lower bounds, we construct product unit networks of fixed depth with super-linear VC dimension. For sigmoidal networks of higher order, we establish polynomial bounds that, in contrast to previous results, do not involve any restriction of the network order. We further consider various classes of higher-order units, also known as sigma-pi units, that are characterized by connectivity constraints. In terms of these, we derive some asymptotically tight bounds. Multiplication plays an important role in both neural modeling of biological behavior and computing and learning with artificial neural networks. We briefly survey research in biology and in applications where multiplication is considered an essential computational element. The results we present here provide new tools for assessing the impact of multiplication on the computational power and the learning capabilities of neural networks.

Neuron function: the mystery persists

There are currently three primary models of how neurons function, each with its uses and variations, according to James McClelland, a professor of psychology and computer science at Carnegie Mellon University and codirector of the Center for Neural Basis of Cognition. The first and simplest is the integrate-and-fire model, which is based on the idea that the neuron adds and subtracts excitatory and inhibitory inputs until it reaches a threshold, at which point it fires a single impulse or action potential. Another model is the sigmoid transfer function, in which the neuron adds up excitatory and inhibitory inputs (as in the integrate-and-fire model) but treats the output as a continuous quantity. Finally, in the sigma-pi unit model, a neuron's output is equal to the sum of many products, each consisting of a multiplication of several inputs.

Randomly connected sigma–pi neurons can form associator networks

A set of sigma–pi units randomly connected to two input vectors forms a type of hetero-associator related to convolution- and matrix-based associative memories. Associations are represented as patterns of activity rather than connection strengths. Decoding the associations requires another network of sigma–pi units, with connectivity dependent on the encoding network. Learning the connectivity of the decoding network involves setting n3 parameters (where n is the size of the vectors), and can be accomplished in approximately 3e n log n presentations of random patterns. This type of network encodes information in activation values rather than in weight values, which makes the information about relationships accessible to further processing. This accessibility is essential for higher-level cognitive tasks such as analogy processing. The fact that random networks can perform useful operations makes it more plausible that these types of associative network could have arisen in the nervous systems of natural organisms during the course of evolution.

MEMORIES AND MEMORY: A PHYSICIST'S APPROACH TO THE BRAIN

International Journal of Modern Physics AVol. 15, No. 26, pp. 4069-4082 (2000) ReviewNo AccessMEMORIES AND MEMORY: A PHYSICIST'S APPROACH TO THE BRAINLEON N. COOPERLEON N. COOPERBrain Science Program and The Institute for Brain and Neural Systems, Brown University, Providence, Rhode Island 02912, USAhttps://doi.org/10.1142/S0217751X0000272XCited by:10 Next AboutSectionsPDF/EPUB ToolsAdd to favoritesDownload CitationsTrack CitationsRecommend to Library ShareShare onFacebookTwitterLinked InRedditEmail This article is based in part on lectures given at the Robert Serber Memorial April 30, 1998 and the CN Yang Symposium May 21, 1999. You currently do not have access to the full text article. Recommend the journal to your library today! FiguresReferencesRelatedDetailsCited By 10A memristor-based generalization and differentiation circuit design and the application in recognitionMeijia Shang, Xiaoping Wang and Mian Li1 Jul 2020A memristor-based circuit design for generalization and differentiation on Pavlov associative memoryMeijia Shang and Xiaoping Wang1 May 2020 | Neurocomputing, Vol. 389Implementation of Memristive Neural Network With Full-Function Pavlov Associative MemoryXiaoyang Liu, Zhigang Zeng and Shiping Wen1 Sep 2016 | IEEE Transactions on Circuits and Systems I: Regular Papers, Vol. 63, No. 9Pavlov associative memory in a memristive neural network and its circuit implementationLidan Wang, Huifang Li, Shukai Duan, Tingwen Huang and Huamin Wang1 Jan 2016 | Neurocomputing, Vol. 171A modular approach to language production: Models and factsJuan C. Valle-Lisboa, Andrés Pomi, Álvaro Cabana, Brita Elvevåg and Eduardo Mizraji1 Jun 2014 | Cortex, Vol. 55A Developmental Approach to Structural Self-Organization in Reservoir ComputingJun Yin, Yan Meng and Yaochu Jin1 Dec 2012 | IEEE Transactions on Autonomous Mental Development, Vol. 4, No. 4Logic in a Dynamic BrainEduardo Mizraji and Juan Lin4 September 2010 | Bulletin of Mathematical Biology, Vol. 73, No. 2Experimental demonstration of associative memory with memristive neural networksYuriy V. Pershin and Massimiliano Di Ventra1 Sep 2010 | Neural Networks, Vol. 23, No. 7Dynamic searching in the brainEduardo Mizraji, Andrés Pomi and Juan C. Valle-Lisboa3 June 2009 | Cognitive Neurodynamics, Vol. 3, No. 4Elman topology with sigma–pi units: An application to the modeling of verbal hallucinations in schizophreniaJuan C. Valle-Lisboa, Florencia Reali, Héctor Anastasía and Eduardo Mizraji1 Sep 2005 | Neural Networks, Vol. 18, No. 7 Recommended Vol. 15, No. 26 Metrics History Received 26 June 2000 PDF download

Partially pre-calculated weights for the backpropagation learning regime and high accuracy function mapping using continuous input RAM-based sigma–pi nets

In this article we present a methodology that partially pre-calculates the weight updates of the backpropagation learning regime and obtains high accuracy function mapping. The paper shows how to implement neural units in a digital formulation which enables the weights to be quantised to 8-bits and the activations to 9-bits. A novel methodology is introduced to enable the accuracy of sigma–pi units to be increased by expanding their internal state space. We, also, introduce a novel means of implementing bit-streams in ring memories instead of utilising shift registers. The investigation utilises digital “Higher Order” sigma–pi nodes and studies continuous input RAM-based sigma–pi units. The units are trained with the backpropagation learning regime to learn functions to a high accuracy. The neural model is the sigma–pi units which can be implemented in digital microelectronic technology. The ability to perform tasks that require the input of real-valued information, is one of the central requirements of any cognitive system that utilises artificial neural network methodologies. In this article we present recent research which investigates a technique that can be used for mapping accurate real-valued functions to RAM-nets. One of our goals was to achieve accuracies of better than 1% for target output functions in the range Y∈[0,1], this is equivalent to an average Mean Square Error (MSE) over all training vectors of 0.0001 or an error modulus of 0.01. We present a development of the sigma–pi node which enables the provision of high accuracy outputs. The sigma–pi neural model was initially developed by Gurney (Learning in nets of structured hypercubes. PhD Thesis, Department of Electrical Engineering, Brunel University, Middlessex, UK, 1989; available as Technical Memo CN/R/144). Gurney's neuron models, the Time Integration Node (TIN), utilises an activation that was derived from a bit-stream. In this article we present a new methodology for storing sigma–pi node's activations as single values which are averages. In the course of the article we state what we define as a real number; how we represent real numbers and input of continuous values in our neural system. We show how to utilise the bounded quantised site-values (weights) of sigma–pi nodes to make training of these neurocomputing systems simple, using pre-calculated look-up tables to train the nets. In order to meet our accuracy goal, we introduce a means of increasing the bandwidth capability of sigma–pi units by expanding their internal state-space. In our implementation we utilise bit-streams when we calculate the real-valued outputs of the net. To simplify the hardware implementation of bit-streams we present a method of mapping them to RAM-based hardware using ‘ring memories’. Finally, we study the sigma–pi units’ ability to generalise once they are trained to map real-valued, high accuracy, continuous functions. We use sigma–pi units as they have been shown to have shorter training times than their analogue counterparts and can also overcome some of the drawbacks of semi-linear units (Gurney, 1992. Neural Networks, 5, 289–303).

Randomly connected sigma–pi neurons can form associator networks

A set of sigma–pi units randomly connected to two input vectors forms a type of hetero-associator related to convolution- and matrix-based associative memories. Associations are represented as patterns of activity rather than connection strengths. Decoding the associations requires another network of sigma–pi units, with connectivity dependent on the encoding network. Learning the connectivity of the decoding network involves setting n3 parameters (where n is the size of the vectors), and can be accomplished in approximately 3e n log n presentations of random patterns. This type of network encodes information in activation values rather than in weight values, which makes the information about relationships accessible to further processing. This accessibility is essential for higher-level cognitive tasks such as analogy processing. The fact that random networks can perform useful operations makes it more plausible that these types of associative network could have arisen in the nervous systems of natural organisms during the course of evolution.

Self-supervised learning in cooperative stereo vision correspondence.

This paper presents a neural network model of stereoscopic vision, in which a process of fusion seeks the correspondence between points of stereo inputs. Stereo fusion is obtained after a self-supervised learning phase, so called because the learning rule is a supervised-learning rule in which the supervisory information is autonomously extracted from the visual inputs by the model. This supervisory information arises from a global property of the potential matches between the points. The proposed neural network, which is of the cooperative type, and the learning procedure, are tested with random-dot stereograms (RDS) and feature points extracted from real-world images. Those feature points are extracted by a technique based on the use of sigma-pi units. The matching performance and the generalization ability of the model are quantified. The relationship between what have been learned by the network and the constraints used in previous cooperative models of stereo vision, is discussed.

Hyperbolic sigma-pi neural network operators for compactly supported continuous functions

It is the aim of this contribution to continue our investigations on a special family of hyperbolic-type linear operators (here, for compactly supported continuous functions on IRn) which immediately can be interpreted as concrete real-time realizations of three-layer feedforward neural networks with sigma-pi units in the hidden layer. To indicate how these results are connected with density results we start with some introductory theorems on this topic. Moreover, we take a detailed look at the complexity of the generated neural networks in order to achieve global e-accuracy.

Generalization in multi-layer networks of Sigma-pi units

This paper investigates a method for predicting the generalization error that a multi-layer network of Sigma-pi units will, on average, exhibit. Theoretical values are compared with those obtained by the computer simulation of small networks learning parity and contiguity functions.

Generalization in Sigma-Pi Networks

In this paper, we investigate generalization in supervised feedforward Sigma-pi nets with particular reference to means of augmentation of generalization of the network for specific tasks. The work was initiated because logical (digital) neural networks of this type do not function in the same manner as the more normal semi-linear unit, hence the general principle behind Sigma-pi networks generalization required examination, to enable one to put forward means of augmenting their generalization abilities. The paper studies four methods, two of which are novel methodologies for enhancing Sigma-pi networks generalization abilities. The networks are hardware realizable and the Sigma-pi units are logical (digital) nodes that respond to their input patterns in addressable locations, the locations (site-values) then define the probability of the output being a logical ‘1’. In this paper, we evaluate the performance of Sigma-pi nets with perceptual problems (in pattern recognition). This was carried out by compar...

On local and global sigma-pi neural networks a common link

In general, there is a great difference between usual three-layer feedforward neural networks with local basis functions in the hidden processing elements and those with standard sigmoidal transfer functions (in the following often called global basis functions). The reason for this difference in nature can be seen in the ridge-type arguments which are commonly used. It is the aim of this paper to show that the situation completely changes when instead of ridge-type arguments were so-called hyperbolic-type arguments. In detail, we show that usual sigmoidal transfer functions evaluated at hyperbolic-type arguments—usually called sigma—pi units—can be used to construct local basis functions which vanish at infinity and, moreover, are integrable and give rise to a partition of unity, both in Cauchy's principal value sense. At this point, standard strategies for approximation with local basis functions can be used without giving up the concept of non-local sigmoidal transfer functions.

How to make sigma-pi neural networks perform perfectly on regular training sets

In this paper, we show how to design three-layer feedforward neural networks with sigma-pi units in the hidden layer in order to perform perfectly on regular training sets. We obtain real-time design schemes based on massively parallel sampling and induced by so-called hyperbolic cardinal translation-type interpolation operators. The real-time nature of our strategy is due to the fact that in the neural network language our approach is nothing else but a very general and efficient one-shot learning scheme. Moreover, because of the very special hyperbolic structure of our sigma-pi units we do not have the usual dramatic increase of parameters and weights that in general happens in case of higher order networks. The final networks are of manageable complexity and may be applied to multigroup discriminant problems, pattern recognition, and image processing. In detail, the XOR-problem and a special multigroup discriminant problem are discussed at the end of the paper.

Artificial neural networks with quasipolynomial synapses and product synaptic contacts

A neural network model with quasipolynomial synapses and product contacts is investigated. The model further generalizes the sigma-pi and product unit models. What and how many quasipolynomial terms, both for individual variables and for cross-product terms, are learned, not predetermined, subject to hardware constraint. Three possible cases are considered. In case 1, the number of learnable parameters needed is determined in learning. It can be considered another method of "growing" a network for a given task, although the graph of the network is fixed. Mechanisms preventing the network from growing too many parameters are designed. In cases 2 and 3, the number of parameters allowed or available is fixed. Cases 2 and 3 may offer both some control on the generalizability of learning and flexibility in functional representation, and may provide a compromise between the complexity of loading and generalizability of learning. Gradient-descent algorithms for training feedforward networks with polynomial synapses and product contacts are developed. Hardware issues are considered, and experimental results are presented.

Training nets of hardware realizable sigma-pi units

Learning convergence is demonstrated for networks of nodes which are defined by a population of values at the vertices of the n-dimensional hypercube. These are functionally equivalent to higher order nodes or sigma-pi units but have the potential to be implemented in readily available memory components. The cube based structure also offers insight into the process of generalization. Three algorithms are discussed: reward-penalty, back propagation, and a new one based on system identification in control theory. It is shown that reward-penalty style techniques emerge as a special case of system identification.

TRAINING RECURRENT NETS OF HARDWARE REALISABLE SIGMA-PI UNITS

The dynamics and training of recurrent nets which make use of probabilistic nodes based on Boolean functions are explored. These are shown to be equivalent to sigma-pi units when using binary inputs and have the advantage of an immediate implementation in readily available digital hardware. The underlying hypercube structure of the Boolean address space is used to describe the process of generalisation and also sheds light on the role of hidden units. This is made more precise with the aid of the concept of hypercube order statistics. Training algorithms are described which make use of a node time-integration apparatus to promote near optimal cube structures in a natural way.