Self-consistent Signal-to-noise Analysis Research Articles

Replica Analysis of Multiuser Detection for Code Division Multiple Access withM-Ary Phase-Shift Keying

We analyze the performance of the maximizer of the posterior marginals (MPM) detector for code division multiple access (CDMA) multiuser detection with \(M\)-ary phase shift keying (\(M\)-ary PSK) in the large system limit by the replica method. The obtained theory agrees with computer simulation reasonably well. We also derive the theory in the case of the large \(M\) limit and discuss the dependence of the properties of \(M\)-ary PSK CDMA communication on \(M\). We show that the waterfall phenomenon occurs for both the finite and infinite values of \(M\). We also show that a value of \(M\) for which the decoded phase information on the original user symbol becomes minimum exists. Furthermore, we discuss the relationship between the theory based on the replica method and that based on self-consistent signal-to-noise analysis (SCSNA).

Self-Consistent Signal-to-Noise Analysis of CDMA Multiuser Detection withM-Ary Phase-Shift Keying

We present a theory of the performance of parallel interference cancellation (PIC) for code division multiple access (CDMA) multiuser detection with \(M\)-ary phase-shift keying (\(M\)-ary PSK) in the large-system limit. The behavior of PIC is essentially the same as that of the associative memory model. Therefore, we analyze the PIC for CDMA using self-consistent signal-to-noise analysis (SCSNA), which was developed to describe the behavior of the associative memory model. We obtain a quantitative description of the performance of PIC.

Influence of Synaptic Depression on Memory Storage Capacity

Synaptic efficacy between neurons is known to change within a short time scale dynamically. Neurophysiological experiments show that high-frequency presynaptic inputs decrease synaptic efficacy between neurons. This phenomenon is called synaptic depression, a short term synaptic plasticity. Many researchers have investigated how the synaptic depression affects the memory storage capacity. However, the noise has not been taken into consideration in their analysis. By introducing "temperature", which controls the level of the noise, into an update rule of neurons, we investigate the effects of synaptic depression on the memory storage capacity in the presence of the noise. We analytically compute the storage capacity by using a statistical mechanics technique called Self Consistent Signal to Noise Analysis (SCSNA). We find that the synaptic depression decreases the storage capacity in the case of finite temperature in contrast to the case of the low temperature limit, where the storage capacity does not change.

The Thouless–Anderson–Palmer equation for an analogue neural network with temporally fluctuating white synaptic noise

Effects of synaptic noise on the retrieval process of associative memory neural networks are studied from the viewpoint of neurobiological and biophysical understanding of information processing in the brain. We investigate the statistical mechanical properties of stochastic analogue neural networks with temporally fluctuating synaptic noise, which is assumed to be white noise. Such networks, in general, defy the use of the replica method, since they have no energy concept. The self-consistent signal-to-noise analysis (SCSNA), which is an alternative to the replica method for deriving a set of order parameter equations, requires no energy concept and thus becomes available in studying networks without energy functions. Applying the SCSNA to stochastic networks requires the knowledge of the Thouless–Anderson–Palmer (TAP) equation which defines the deterministic networks equivalent to the original stochastic ones. The study of the TAP equation which is of particular interest for the case without energy concept is very less, while it is closely related to the SCSNA in the case with energy concept. This paper aims to derive the TAP equation for networks with synaptic noise together with a set of order parameter equations by a hybrid use of the cavity method and the SCSNA.

Thouless–Anderson–Palmer equation for associative memory neural network with synaptic noise

We study the effects of temporal fluctuations in synaptic couplings on the properties of analog neural networks. Since no energy concept exists in networks with such couplings, the use of the replica method does not make sense. On the other hand, the self-consistent signal-to-noise analysis (SCSNA), which is an alternative to the replica method for deriving a set of order parameter equations, requires no energy concept and thus plays an important role for studying such networks. To apply the SCSNA to stochastic networks, it is necessary to define the deterministic networks equivalent to the original stochastic ones, which are given by the Thouless–Anderson–Palmer (TAP) equations. Therefore the TAP equation is of interest for studying the statistical properties of the networks with synaptic noise, while such study is very few. In this paper, we show the TAP equation together with a set of order parameter equations for such networks by using both the cavity method and the SCSNA.

Thouless-Anderson-Palmer equation and self-consistent signal-to-noise analysis for the Hopfield model with three-body interaction

The self-consistent signal-to-noise analysis (SCSNA) is an alternative to the replica method for deriving the set of order parameter equations for associative memory neural network models and is closely related with the Thouless-Anderson-Palmer equation (TAP) approach. In the recent paper by Shiino and Yamana the Onsager reaction term of the TAP equation has been found to be obtained from the SCSNA for Hopfield neural networks with two-body interaction. We study the TAP equation for an associative memory stochastic analog neural network with three-body interaction to investigate the structure of the Onsager reaction term, in connection with the term proportional to the output characteristic to the SCSNA. We report on the SCSNA framework for analog networks with three-body interactions as well as provide a recipe based on the cavity concept that involves two cavities and the hybrid use of the SCSNA to obtain the TAP equation.

Analysis of Bidirectional Associative Memory using Self-consistent Signal to Noise Analysis and Statistical Neurodynamics

Bidirectional associative memory (BAM) is a kind of an artificial neural network used to memorize and retrieve heterogeneous pattern pairs. Many efforts have been made to improve BAM from the the viewpoint of computer application, and few theoretical studies have been done. We investigated the theoretical characteristics of BAM using a framework of statistical-mechanical analysis. To investigate the equilibrium state of BAM, we applied self-consistent signal to noise analysis (SCSNA) and obtained a macroscopic parameter equations and relative capacity. Moreover, to investigate not only the equilibrium state but also the retrieval process of reaching the equilibrium state, we applied statistical neurodynamics to the update rule of BAM and obtained evolution equations for the macroscopic parameters. These evolution equations are consistent with the results of SCSNA in the equilibrium state.

Statistical mechanics of stochastic neural networks: Relationship between the self-consistent signal-to-noise analysis, Thouless-Anderson-Palmer equation, and replica symmetric calculation approaches.

We study the statistical mechanical aspects of stochastic analog neural network models for associative memory with correlation type learning. We take three approaches to derive the set of the order parameter equations for investigating statistical properties of retrieval states: the self-consistent signal-to-noise analysis (SCSNA), the Thouless-Anderson-Palmer (TAP) equation, and the replica symmetric calculation. On the basis of the cavity method the SCSNA can be generalized to deal with stochastic networks. We establish the close connection between the TAP equation and the SCSNA to elucidate the relationship between the Onsager reaction term of the TAP equation and the output proportional term of the SCSNA that appear in the expressions for the local fields.

Stability of oscillatory retrieval solutions in the oscillator neural network without Lyapunov functions

Constructing a Ginzburg-Landau map neural network, we analyze its storage capacity with an equilibrium theory of the self-consistent signal-to-noise analysis (SCSNA); however, the prediction does not consist with the simulation results just in the parameter region where the characteristic of the non-Lyapunov-function system gets enhanced [J. Phys. A 32, 4623 (1999)]. It is expected that this inconsistency comes from the fact that the dynamics of retrieval and nonretrieval states governs the phase transition. Alternatively, we investigate its storage capacity with the help of the Amari-Maginu-Okada theory, a dynamical theory, for the stability analysis of dynamical states. We consequently found that the theory predicts dynamical states quite well especially in the region where the SCSNA breaks down, and that the phase diagram coincides quantitatively well with the simulation results.

Application of two-parameter dynamical replica theory to retrieval dynamics of associative memory with non-monotonic neurons

The two-parameter dynamical replica theory (2-DRT) is applied to investigate retrieval properties of non-monotonic associative memory, a model which lacks thermodynamic potential functions. 2-DRT reproduces dynamical properties of the model quite well, including the capacity and basin of attraction. Superretrieval state is also discussed in the framework of 2-DRT. The local stability condition of the superretrieval state is given, which provides a better estimate of the region in which superretrieval is observed experimentally than the self-consistent signal-to-noise analysis (SCSNA) does.

Phase transitions of an oscillator neural network with a standard Hebb learning rule

Studies have been made on the phase transition phenomena of an oscillator network model based on a standard Hebb learning rule like the Hopfield model. The relative phase informations---the in-phase and anti-phase, can be embedded in the network. By self-consistent signal-to-noise analysis (SCSNA), it was found that the storage capacity is given by $\alpha_c = 0.042$, which is better than that of Cook's model. However, the retrieval quality is worse. In addition, an investigation was made into an acceleration effect caused by asymmetry of the phase dynamics. Finally, it was numerically shown that the storage capacity can be improved by modifying the shape of the coupling function.

Associative Memory of Analog Networks without the Energy Concept Using Biased Patterns and Positive-Valued Transfer Functions

Associative memory of analog neural networks for which the concept of energy functions is lost is studied by means of the SCSNA (Self-Consistent Signal-to-Noise Analysis). Asymmetric synaptic couplings with biased random patterns are assumed together with positive-valued transfer functions which allow nonmonotonicity resulting from an appropriate cut off of output activity. Phase diagrams are given in terms of several parameters of the networks, showing the occurrence of enhancement of the storage capacity due to nonmonotonic transfer functions. Super retrieval states ensuring errorless memory retrieval under the loading of extensively many patterns are allowed to remain in existence in the presence of asymmetric synaptic couplings with biased patterns. A sample-dependent component in the local fields of neurons, which arises from the assumed asymmetric couplings, is discussed and shown to become of no effect in the super retrieval phase.

Retrieval properties of analog neural networks and the nonmonotonicity of transfer functions

Characteristic properties of associative memory networks with continuous-time dynamics are extensively studied for a certain class of nonmonotonic transfer functions by means of the self-consistent signal-to-noise analysis (SCSNA) and numerical simulations. The conventional Hebb-type symmetric synaptic connections with unbiased random patterns are assumed. Although the occurrence of instability, including an oscillatory one, makes the storage capacity fall below the upper bound for storage ratio obtained by the SCSNA, the storage capacity remains as large as 0.4 in the optimal cases. It is also noted that noise in the local fields (i.e., the inputs to neurons) can vanish for certain cases of nonmonotonic transfer functions even with an extensive number of stored patterns. Implication of the present results is the possibility of improving the network performances by the achievement of errorless retrieval and enhancement of storage capacity with the use of nonmonotonic transfer functions.

The S/N Analysis of Associative Memory

There are some models of associative memory that have improved learning algorithms or dynamics. We obtained the memory capacity of an associative memory using the sign alternating memorization method and the pairing memorization method by means of self-consistent signal-to-noise analysis(SCSNA). In the case using the grouping memorization method, as the number of patterns of a group increases, the memory capacity increases. If more than about 10 patterns are grouped, the memory capacity is about 0.27 as well as the cross-correlation type.

Self-consistent signal-to-noise analysis of the statistical behavior of analog neural networks and enhancement of the storage capacity.

Based on the self-consistent signal-to-noise analysis (SCSNA) capable of dealing with analog neural networks with a wide class of transfer functions, enhancement of the storage capacity of associative memory and the related statistical properties of neural networks are studied for random memory patterns. Two types of transfer functions with the threshold parameter \ensuremath{\theta} are considered, which are derived from the sigmoidal one to represent the output of three-state neurons. Neural networks having a monotonically increasing transfer function ${\mathit{F}}^{\mathrm{M}}$, ${\mathit{F}}^{\mathrm{M}}$(u)=sgnu (\ensuremath{\Vert}u\ensuremath{\Vert}g\ensuremath{\theta}), ${\mathit{F}}^{\mathrm{M}}$(u)=0 (\ensuremath{\Vert}u\ensuremath{\Vert}\ensuremath{\le}\ensuremath{\theta}), are shown to make it impossible for the spin-glass state to coexist with retrieval states in a certain parameter region of \ensuremath{\theta} and \ensuremath{\alpha} (loading rate of memory patterns), implying the reduction of the number of spurious states. The behavior of the storage capacity with changing \ensuremath{\theta} is qualitatively the same as that of the Ising spin neural networks with varying temperature. On the other hand, the nonmonotonic transfer function ${\mathit{F}}^{\mathrm{NM}}$, ${\mathit{F}}^{\mathrm{NM}}$(u)=sgnu (\ensuremath{\Vert}u\ensuremath{\Vert}), ${\mathit{F}}^{\mathrm{NM}}$(u)=0 (\ensuremath{\Vert}u\ensuremath{\Vert}\ensuremath{\ge}\ensuremath{\theta}) gives rise to remarkable features in several respects.First, it yields a large enhancement of the storage capacity compared with the Amit-Gutfreund-Sompolinsky (AGS) value: with decreasing \ensuremath{\theta} from \ensuremath{\theta}=\ensuremath{\infty}, the storage capacity ${\mathrm{\ensuremath{\alpha}}}_{\mathit{c}}$ of such a network is increased from the AGS value (\ensuremath{\approxeq}0.14) to attain its maximum value of \ensuremath{\approxeq}0.42 at \ensuremath{\theta}\ensuremath{\simeq}0.7 and afterwards is decreased to vanish at \ensuremath{\theta}=0. Whereas for \ensuremath{\theta}\ensuremath{\gtrsim}1 the storage capacity ${\mathrm{\ensuremath{\alpha}}}_{\mathit{c}}$ coincides with the value ${\mathrm{\ensuremath{\alpha}}}_{\mathit{c}}$\ifmmode \tilde{}\else \~{}\fi{} determined by the SCSNA as the upper bound of \ensuremath{\alpha} ensuring the existence of retrieval solutions, for \ensuremath{\theta}\ensuremath{\lesssim}1 the ${\mathrm{\ensuremath{\alpha}}}_{\mathit{c}}$ is shown to differ from the ${\mathrm{\ensuremath{\alpha}}}_{\mathit{c}}$\ifmmode \tilde{}\else \~{}\fi{} with the result that the retrieval solutions claimed by the SCSNA are unstable for ${\mathrm{\ensuremath{\alpha}}}_{\mathit{c}}$${\mathrm{\ensuremath{\alpha}}}_{\mathit{c}}$\ifmmode \tilde{}\else \~{}\fi{}. Second, in the case of \ensuremath{\theta}1 the network can exhibit a new type of phase which appears as a result of a phase transition with respect to the non-Gaussian distribution of the local fields of neurons: the standard type of retrieval state with r\ensuremath{\ne}0 (i.e., finite width of the local field distribution), which is implied by the order-parameter equations of the SCSNA, disappears at a certain critical loading rate ${\mathrm{\ensuremath{\alpha}}}_{0}$, and for \ensuremath{\alpha}\ensuremath{\le}${\mathrm{\ensuremath{\alpha}}}_{0}$ a qualitatively different type of retrieval state comes into existence in which the width of the local field distribution vanishes (i.e., r=${0}^{+}$). As a consequence, memory retrieval without errors becomes possible even in the saturation limit \ensuremath{\alpha}\ensuremath{\ne}0. Results of the computer simulations on the statistical properties of the novel phase with \ensuremath{\alpha}\ensuremath{\le}${\mathrm{\ensuremath{\alpha}}}_{0}$ are shown to be in satisfactory agreement with the theoretical results. The effect of introducing self-couplings on the storage capacity is also analyzed for the two types of networks. It is conspicuous for the networks with ${\mathit{F}}^{\mathrm{NM}}$, where the self-couplings increase the stability of the retrieval solutions of the SCSNA with small values of \ensuremath{\theta}, leading to a remarkable enhancement of the storage capacity.