Hopfield Neural Network Research Articles

Wav2vec2 Without Attention: Do You Need Hopfield Networks for Self-Supervised Learning of Speech Representations?

Wav2vec2 Without Attention: Do You Need Hopfield Networks for Self-Supervised Learning of Speech Representations?

Effect of electromagnetic radiations on the dynamics of a five-chain coupled inertial Hopfield neural network and control of multi-stability with the selection of a desired attractor.

Abstract In this paper, we investigate the effect of electromagnetic radiation on the dynamics of a network consisting of five chain-coupled inertial Hopfield neurons. The study revealed that the neural system designed such a way involves several complex phenomena, namely Hopf bifurcation, chaos, hyperchaos and the coexistence of up to thirty-two attractors in phase space. The complexity specific to our system is due to the higher number of equilibrium points, namely two hundred and forty-three. Under the conditions of safe functioning of our neural system (chaos or hyperchaos), we have been able to observe that electromagnetic radiation has a harmful character for the system, because we found for a range of variation in the intensity of the electromagnetic feedback induction current a coexistence of chaotic and periodic states (epileptic state). We then controlled the multi-stability using a desired attractor selection scheme, with the aim of suppressing the pathological state. All the work carried out in this contribution is done with the help of dynamical system analysis tools such as the bifurcation diagram, the spectrum and the maximum exponent of Lyapunov, phase portraits and basins of attraction. The scheme used is the Runge-Kutta-4. In order to validate the numerical results, we use analog calculation and some results were derived from the Pspice software. These results are in good accordance in amplitude and location on the plane to those of numerical simulations.

Hebbian learning from first principles

Recently, the original storage prescription for the Hopfield model of neural networks – as well as for its dense generalizations – has been turned into a genuine Hebbian learning rule by postulating the expression of its Hamiltonian for both the supervised and unsupervised protocols. In these notes, first, we obtain these explicit expressions by relying upon maximum entropy extremization à la Jaynes. Beyond providing a formal derivation of these recipes for Hebbian learning, this construction also highlights how Lagrangian constraints within entropy extremization force network’s outcomes on neural correlations: these try to mimic the empirical counterparts hidden in the datasets provided to the network for its training and, the denser the network, the longer the correlations that it is able to capture. Next, we prove that, in the big data limit, whatever the presence of a teacher (or its lacking), not only these Hebbian learning rules converge to the original storage prescription of the Hopfield model but also their related free energies (and, thus, the statistical mechanical picture provided by Amit, Gutfreund and Sompolinsky is fully recovered). As a sideline, we show mathematical equivalence among standard Cost functions (Hamiltonian), preferred in Statistical Mechanical jargon, and quadratic Loss Functions, preferred in Machine Learning terminology. Remarks on the exponential Hopfield model (as the limit of dense networks with diverging density) and semi-supervised protocols are also provided.

Jaynes machine: The universal microstructure of deep neural networks

Despite the recent stunning progress in large-scale deep neural network applications, our understanding of their microstructure, ‘energy’ functions, and optimal design remains incomplete. Here, we present a new game-theoretic framework, called statistical teleodynamics, that reveals important insights into these key properties. The optimally robust design of such networks inherently involves computational benefit-cost trade-offs that physics-inspired models do not adequately capture. These trade-offs occur as neurons and connections compete to increase their effective utilities under resource constraints during training. In a fully trained network, this results in a state of arbitrage equilibrium, where all neurons in a given layer have the same effective utility, and all connections to a given layer have the same effective utility. The equilibrium is characterized by the emergence of two lognormal distributions of connection weights and neuronal output as the universal microstructure of large deep neural networks. We call such a network the Jaynes Machine. Our theoretical predictions are shown to be supported by empirical data from seven large-scale deep neural networks. We also show that the Hopfield network and the Boltzmann Machine are the same special case of the Jaynes Machine.

High-order rotor Hopfield neural networks for associative memory

High-order rotor Hopfield neural networks for associative memory

Prediction of potential energy on player scoring trend in tennis match based on Hopfield neural network model

In tennis, it is very important to catch the direction and trend of the match when the score happens. This paper aims to establish a mathematical model to predict the dynamic change of match progress and the appearance of match points, and explain the role of "momentum" in the match. By mining the real-time match data of 2023 Wimbledon Tennis Open, based on Hopfield neural network and entropy weight method, the technical level, psychological pressure and fatigue level of players in the match are selected as the influencing factors, and the mathematical model between the "momentum" of players and the change of the match flow is established, and the change of the match schedule can be predicted more accurately. The change of momentum may affect the motivation of players, and the choice of a reasonable strategy may be a magic weapon to win the match. It is hoped that this can help the pre-competition training of tennis players and promote the vigorous development of tennis events.

Dynamic Analysis and FPGA Implementation of Fractional-Order Hopfield Networks with Memristive Synapse

Memristors have become important components in artificial synapses due to their ability to emulate the information transmission and memory functions of biological synapses. Unlike their biological counterparts, which adjust synaptic weights, memristor-based artificial synapses operate by altering conductance or resistance, making them useful for enhancing the processing capacity and storage capabilities of neural networks. When integrated into systems like Hopfield neural networks, memristors enable the study of complex dynamic behaviors, such as chaos and multistability. Moreover, fractional calculus is significant for their ability to model memory effects, enabling more accurate simulations of complex systems. Fractional-order Hopfield networks, in particular, exhibit chaotic and multistable behaviors not found in integer-order models. By combining memristors with fractional-order Hopfield neural networks, these systems offer the possibility of investigating different dynamic phenomena in artificial neural networks. This study investigates the dynamical behavior of a fractional-order Hopfield neural network (HNN) incorporating a memristor with a piecewise segment function in one of its synapses, highlighting the impact of fractional-order derivatives and memristive synapses on the stability, robustness, and dynamic complexity of the system. Using a network of four neurons as a case study, it is demonstrated that the memristive fractional-order HNN exhibits multistability, coexisting chaotic attractors, and coexisting limit cycles. Through spectral entropy analysis, the regions in the initial condition space that display varying degrees of complexity are mapped, highlighting those areas where the chaotic series approach a pseudo-random sequence of numbers. Finally, the proposed fractional-order memristive HNN is implemented on a Field-Programmable Gate Array (FPGA), demonstrating the feasibility of real-time hardware realization.

Three-Dimensional Space Chaotic Scrolls Generated by Memristive Hopfield Neural Network and Its Application in Robot Path Planning

Abstract In this paper, a novel three-dimensional space-scroll chaotic attractor is constructed by introducing piece-wise linear memristors into a three-neurons-based Hopfield neural network to simulate the electromagnetic influence on neural dynamics. The dynamical behaviors of the novel memristive Hopfield neural network are discussed by means of numerical methods including the Lyapunov exponent spectrum, bifurcation diagrams, phase plots and dynamic maps. Furthermore, we also investigated different neurons subject to the memristive electromagnetic influence, where the two-dimensional grid-scroll chaotic attractors can be generated by one neuron subjects to the electromagnetic influence and three-dimensional space-scroll chaotic attractors can be generated by all of the neurons subject to the memristive electromagnetic influence. Moreover, we present a three-dimensional space-scroll scroll attractor-based chaotic mobile robot system. Superiority of the proposed space-scroll-based chaotic mobile robot system compared to existing methods has been demonstrated through numerical simulations.

Long sequence Hopfield memory*

Abstract Sequence memory is an essential attribute of natural and artificial intelligence that enables agents to encode, store, and retrieve complex sequences of stimuli and actions. Computational models of sequence memory have been proposed where recurrent Hopfield-like neural networks are trained with temporally asymmetric Hebbian rules. However, these networks suffer from limited sequence capacity (maximal length of the stored sequence) due to interference between the memories. Inspired by recent work on Dense Associative Memories, we expand the sequence capacity of these models by introducing a nonlinear interaction term, enhancing separation between the patterns. We derive novel scaling laws for sequence capacity with respect to network size, significantly outperforming existing scaling laws for models based on traditional Hopfield networks, and verify these theoretical results with numerical simulation. Moreover, we introduce a generalized pseudoinverse rule to recall sequences of highly correlated patterns. Finally, we extend this model to store sequences with variable timing between states’ transitions and describe a biologically-plausible implementation, with connections to motor neuroscience.

Regulation and analysis of multi-scroll attractors in a class of memristive Hopfield neural networks with high Lyapunov exponents

Multi-scroll chaotic systems have gained significant attention in recent years. However, there is little research on the chaotic systems of memristive neural networks with multiple complex structural attractors. A class of multi-scroll chaotic systems are designed in this paper, which are based on Hopfield neural network by using memristors as synapses. Implementing synaptic control with memristors at various coupling positions allows for the creation of multiple chaotic attractors with distinct topological structures. The proposed systems exhibit special and irregular shaped attractors, and can generate a fixed number of scrolls. Through analysis we find that the systems have high Lyapunov exponents, indicating strong sensitivity and randomness. Meanwhile, the systems exhibit extremely rich dynamic behaviors, such as symmetric coexistence phenomena. Especially, we observe signal amplitude control and offset boosting phenomena in the systems. In addition, simulation circuits are designed based on such chaotic systems to verify the physical feasibility of the systems.

Prototype Analysis in Hopfield Networks With Hebbian Learning.

We discuss prototype formation in the Hopfield network. Typically, Hebbian learning with highly correlated states leads to degraded memory performance. We show that this type of learning can lead to prototype formation, where unlearned states emerge as representatives of large correlated subsets of states, alleviating capacity woes. This process has similarities to prototype learning in human cognition. We provide a substantial literature review of prototype learning in associative memories, covering contributions from psychology, statistical physics, and computer science. We analyze prototype formation from a theoretical perspective and derive a stability condition for these states based on the number of examples of the prototype presented for learning, the noise in those examples, and the number of nonexample states presented. The stability condition is used to construct a probability of stability for a prototype state as the factors of stability change. We also note similarities to traditional network analysis, allowing us to find a prototype capacity. We corroborate these expectations of prototype formation with experiments using a simple Hopfield network with standard Hebbian learning. We extend our experiments to a Hopfield network trained on data with multiple prototypes and find the network is capable of stabilizing multiple prototypes concurrently. We measure the basins of attraction of the multiple prototype states, finding attractor strength grows with the number of examples and the agreement of examples. We link the stability and dominance of prototype states to the energy profile of these states, particularly when comparing the profile shape to target states or other spurious states.

Enhancing Hopfield network performance for pattern retrieval using sparse recovery algorithm and Parzen estimator

An improved pattern recovery approach that integrates the Hopfield neural network (HNN) with the iterative signal reconstruction and Parzen window-based classification is proposed. The HNN is observed as a form of associative memory network, used for various pattern recognition and optimization tasks. However, when the input pattern is highly damaged with a very limited set of available samples, the Hopfield network fails to perform the retrieval. The convex optimization-based gradient descent algorithm is considered for pattern recovery of damaged inputs in order to provide an improved pattern approximation for further processing within the HNN, enabling successful network performance. Additionally, in the case of grayscale images, the Parzen window approach is used to classify the probability density functions (pdfs) of the training set and to choose those being comparable to the pdf of the input pattern, therefore refining the selection of patterns and providing better convergence to the exact retrieval. The theoretical considerations are verified experimentally, showing the high performance of the proposed approach when only 10 % of the pixels are available for binary patterns and 40 % of pixels for grayscale patterns.

Exact solutions for analog Hawking effect in dielectric media

In the framework of the analog Hawking radiation for dielectric media, we analyze a toy model and also the 2D reduction of the Hopfield model for a specific monotone and realistic profile for the refractive index. We are able to provide exact solutions, which do not require any weak dispersion approximation. The theory of Fuchsian ordinary differential equations is the basic tool for recovering exact solutions, which are rigorously identified and involve the so-called generalized hypergeometric functions F34(α1,α2,α3,α4;β1,β2,β3;z). A complete set of connection formulas are available, both for the subcritical case and for the transcritical one, and also the Stokes phenomenon occurring in the problem is fully discussed. From the physical point of view, we focus on the problem of thermality. Under suitable conditions, the Hawking temperature is deduced, and we show that it is in full agreement with the expression deduced in other frameworks under various approximations. Published by the American Physical Society 2024

Dissipative quantum Hopfield network: a numerical analysis

Abstract We present extensive simulations of a quantum version of the Hopfield Neural Network to explore its emergent behavior. The system is a network of $N$ qubits oscillating at a given $\Omega$ frequency and which are coupled via Lindblad jump operators built with local fields $h_i$ depending on some given stored patterns. Our simulations show the emergence of pattern-antipattern oscillations of the overlaps with the stored patterns similar (for large $\Omega$ and small temperature) to those reported within a recent mean-field description of such a system, and which are originated deterministically by the quantum term including $s_x^i$ qubit operators. However, in simulations we observe that such oscillations are stochastic due to the interplay between noise and the inherent metastability of the pattern attractors induced by quantum oscillations, and then are damped in finite systems when one averages over many quantum trajectories. In addition, we report the system behavior for large number of stored patterns at the lowest temperature we can reach in simulations (namely $T=0.005\, T_C$). Our study reveals that for small-size systems the quantum term of the Hamiltonian has a negative effect on storage capacity, decreasing the overlap with the starting memory pattern for increased values of $\Omega$ and number of stored patterns. However, it also impedes the system to be trapped for long time in mixtures and spin-glass states. Interestingly, the system also presents a range of $\Omega$ values for which, although the initial pattern is destabilized due to quantum oscillations, other patterns can be retrieved and remain stable even for many stored patterns, implying a quantum-dependent nonlinear relationship between the recall process and the number of stored patterns.

Coexisting and multiple scroll attractors in a Hopfield neural network with a controlled memristor

Abstract In this paper, a method of generating multi-double scroll attractors is proposed based on the memristor Hopfield neural network (HNN) under pulse control. First, the original hyperbolic-type memristor is added to the neural network mathematical model, and the influence of this memristor on the dynamic behavior of the new HNN is analyzed. The numerical results show that after adding the memristor, the abundant dynamic behaviors such as chaos coexistence, period coexistence and chaos period coexistence can be observed when the initial value of the system is changed. Then the logic pulse is added to the external memristor. It is found that the equilibrium point of the HNN can multiply and generate multi-double scroll attractors after the pulse stimulation. When the number of logical pulses is changed, the number of multi-double scroll attractors will also change, so that the pulse can control the generation of multi-double scroll attractors. Finally, the HNN circuit under pulsed stimulation was realized by circuit simulation, and the results verified the correctness of the numerical results.

НЕЙРОМЕРЕЖЕВІ МАТЕМАТИЧНІ МОДЕЛІ ЛЮДИНОМАШИННОГО СПІЛКУВАННЯ ТА РОЗПІЗНАВАННЯ

The study is devoted to the neural network interpretation of the task of human-machine communication and recognition by multiple criteria, considered as a task of assignment. The main goal is to reduce this problem to a standard form, where the number of criteria groups is equal to the number of ranked documents. The study defined the architecture of a neural network and proposed the use of a network of binary neurons, which is a matrix of a certain dimension. The proposed ranking model is based on a neural network that contains arbitrary feedback. This allows the excitation to be transmitted back to the neuron, which contributes to the repeated performance of its function. However, in dynamic neural networks instability occurs, which is manifested in a random change in the states of neurons without reaching stationary states. The question of stability of the dynamics of such systems remains open. The considered discrete Hopfield neural network has the following characteristics: one layer of elements, each element is connected to all others, but not to itself; only one element is updated per stage; elements are updated in random order, but each is updated with the same frequency; the output function is binary (value "0" or "1"). A Hopfield neural network is recurrent: the output of the network is reused as input until a steady state is reached. After starting, the neural network changes its state, gradually moving to a stable mode, which allows identifying a plan for evaluating the process of human-machine communication according to a set of criteria. Random search procedures are used to refine the results. The proposed energy function is minimized to ensure that the constraints are met and the problem is solved. The constructed function reaches a minimum only in the states corresponding to the assignment plans. The definition of the network parameters is carried out by comparing the obtained functions with the energy function in general form. The practical implementation of the model demonstrated that Hopfield's neural network can be successfully applied to document ranking in human-machine communication and recognition systems, providing high accuracy and efficiency in solving ranking problems.

Study on the construction and application of a community emergency capacity evaluation model based on a combined weighting-discrete Hopfield neural network

To accurately evaluate community emergency capability and solve the problems with the existing index system, which has been declared unreasonable with respect to components, unclear in meaning and unreliable in terms of empowerment results caused by a single algorithm, a triangle model of community safety and resilience is introduced to construct an emergency capability index system suitable for communities with frequent geological disasters in Yunnan Province. Additionally, a combined empowerment model is constructed to improve the accuracy of the empowerment results. First, according to the relevant information and the results of expert consultations and field investigations, the index system is determined. Second, three weighting methods, the G1-entropy weight, G1-CRITIC and G1-coefficient of variation, are used to calculate the index weight and perform a comparative analysis, after which the best weighting method is selected and combined with the discrete Hopfield neural network model to construct a weighting-evaluation model. Finally, the model is applied to evaluate the emergency capability of a community with frequent geological disasters in Yunnan Province. The results show that the index of monitoring and early warning ability of geological disasters is added, which makes the index more targeted, and thus, the result obtained using the G1-CRITIC combination weighting model is more accurate. The evaluation model constructed in this paper accurately evaluates community emergency capacity and is further popularized and applied in communities with frequent geological disasters in Yunnan Province.

Modification of the hopfield neural network model for solving the task of optimal task allocation in a group of mobile robots

In the context of group interaction among mobile robots, there arises the challenge of task distribution within the group, considering the robots' characteristics and the working environment. This study aims to modify the Hopfield neural network and develop methodologies for its application in solving the task allocation problem for an arbitrary number of tasks within a group of mobile robots. To achieve this, the Hopfield neural network is represented as a graph. An algorithm is presented, demonstrating the transition from the initial problem to the Traveling Salesman Problem (TSP). The application of the Hopfield model to the task distribution problem in a group of robots is described, along with the development of an optimization function calculation algorithm. An assessment is conducted to evaluate the impact of neural network parameters on the quality and speed of solving the optimization problem. By comparing it with other heuristic methods (genetic and ant colony algorithms), the domains of application for the modified algorithm are determined.

Опыт внедрения информационной системы с технологиями искусственного интеллекта в работу поликлиники для оптимизации организации периодических медицинских осмотров.

Organizing medical examinations for patients with minimal time losses is the most difficult task for an outpatient clinic when creating a New model of a medical organization providing primary health care. Since during medical examinations it is necessary to perform a whole range of works associated with certain sets of actions, the authors developed an information system with technologies of artificial intelligence, which was tested during periodic medical examinations. The purpose of the study is to develop and test an information system with technologies of artificial intelligence that allows balancing the capabilities of a medical organization and the needs of the patient when undergoing periodic medical examinations. Materials and methods. The tool for solving this problem was an artificial intelligence platform based on the principle of decision making by controlling the positions of attractors in the state space of the Hopfield neural network, which was built into the medical information system using HL7 FHIR. Results. The result of the work was the “Smart Clinic” project, which provides for the creation of an information system with technologies of artificial intelligence for routing patients during periodic medical examinations. The information system provided dynamic routing of patients in real time with the construction of the shortest trajectory and minimal movement across the floors of the building, based on current information about the workload of specialists and the duration of each stage of the examination. Conclusions. The use of an artificial intelligence information system in the outpatient clinic made it possible to reduce the average duration of a patient’s trajectory during a periodic medical examination by 45%, the average waiting time for an appointment – from 7 minutes to 3 minutes, and the total time for undergoing a preventive examination – from 200 to 150 minutes. This made it possible to increase the number of periodic medical examinations per year by 1.5 times, and the level of patient satisfaction reached 95%.

Multi-ringlike volumes and offset of Hopfield neural networks based on a discrete memristive self-synapse

Multi-ringlike volumes and offset of Hopfield neural networks based on a discrete memristive self-synapse