Neuron Activation Function Research Articles

FORMATION OF THE DATASET AND SELECTION OF PARAMETERS FOR THE BUILDING OF THE ARTIFICIAL NEURAL NETWORKS OF THE ENTERPRISES FINANCIAL SUSTAINABILITY CLASSIFICATION

The purpose. The article is devoted to the preparatory actions preceding of construction of a fully-connected three-layer artificial neural network of direct distribution, which should classify the enterprises for their financial sustainability. Methods. To prepare for the construction of an artificial neural network are presented a range of reasoned judgments and solutions, based on the results of experiments on building a neural network of financial sustainability. The Results. The article presents ways to obtain a dataset for training an artificial neural network of financial sustainability of Ukrainian enterprises. There are presented the optimal architectural parameters and recommendations about the neural network teaching. The attention was paid to such parameters of the artificial neural network as the type of neural network, the error function, the activation function on the hidden and on the initial layers, the learning algorithm, the parameters of initial weight initialization. The application of the recommendations from the article allows constructing an artificial neural network, which classifies enterprises for their financial sustainability with sufficiently high accuracy. Particular attention is paid to selecting neuron activation functions and to the distribution of a general dataset into training, testing and validation subsets. Scientific novelty. The use of 17 financial indicators as factors for modeling financial sustainability is proposed, what should describe this complex concept and allow to obtain maximum accuracy of classification. The article substantiates the use of the construction parameters and training of the neural network, which are best suited for the use of these factors. The practical significance. Using the article`s material will help with building of the artificial neural network of financial sustainability, which can be used to classify enterprises as "financially sustainable" or "potential bankrupt" and can be used by financial and credit institutions, investment funds and state authorities when assessing the risk of bankruptcy of an enterprise. Such a neural network is designed to automate the work of an experienced financial analyst when checking the company for financial sustainability.

Synthesizing Images From Spatio-Temporal Representations Using Spike-Based Backpropagation.

Spiking neural networks (SNNs) offer a promising alternative to current artificial neural networks to enable low-power event-driven neuromorphic hardware. Spike-based neuromorphic applications require processing and extracting meaningful information from spatio-temporal data, represented as series of spike trains over time. In this paper, we propose a method to synthesize images from multiple modalities in a spike-based environment. We use spiking auto-encoders to convert image and audio inputs into compact spatio-temporal representations that is then decoded for image synthesis. For this, we use a direct training algorithm that computes loss on the membrane potential of the output layer and back-propagates it by using a sigmoid approximation of the neuron's activation function to enable differentiability. The spiking autoencoders are benchmarked on MNIST and Fashion-MNIST and achieve very low reconstruction loss, comparable to ANNs. Then, spiking autoencoders are trained to learn meaningful spatio-temporal representations of the data, across the two modalities—audio and visual. We synthesize images from audio in a spike-based environment by first generating, and then utilizing such shared multi-modal spatio-temporal representations. Our audio to image synthesis model is tested on the task of converting TI-46 digits audio samples to MNIST images. We are able to synthesize images with high fidelity and the model achieves competitive performance against ANNs.

Visualization of Intra-neuronal Motor Protein Transport through Upconversion Microscopy.

Cargo transport along axons, a physiological process mediated by motor proteins, is essential for neuronal function and survival. A current limitation in the study of axonal transport is the lack of a robust imaging technique with a high spatiotemporal resolution to visualize and quantify the movement of motor proteins in real-time and in different depth planes. Herein, we present a dynamic imaging technique that fully exploits the characteristics of upconversion nanoparticles. This technique can be used as a microscopic probe for the quantitative in situ tracking of retrograde transport neurons with single-particle resolution in multilayered cultures. This study may provide a powerful tool to reveal dynamic neuronal activity and intra-axonal transport function as well as any associated neurodegenerative diseases resulting from mutation or impairment in the axonal transport machinery.

Gonadoliberin – Synthesis, Secretion, Molecular Mechanisms and Targets of Action

Decapeptide gonadoliberin (GnRH) is the most important regulator of the hypothalamic-pituitary-gonadal (HPG) axis that controls the synthesis and secretion of the luteinizing and follicle-stimulating hormones by gonadotrophs in the adenohypophysis. GnRH is produced by the specialized hypothalamic neurons using the site-specific proteolysis of the precursor protein and is secreted into the portal pituitary system, where it binds to the specific receptors. These receptors belong to the family of G protein-coupled receptors, and they are located on the surface of gonadotrophs and mediate the regulatory effects of GnRH on the gonadotropins production. The result of GnRH binding to them is the activation of phospholipase C and the calcium-dependent pathways, the stimulation of different forms of mitogen-activated protein kinases, as well as the activation of the enzyme adenylyl cyclase and the triggering of cAMP-dependent signaling pathways in the gonadotrophs. The gonadotropins, kisspeptin, sex steroid hormones, insulin, melatonin and a number of transcription factors have an important role in the regulation of GnRH1 gene expression, which encodes the GnRH precursor, as well as the synthesis and secretion of GnRH. The functional activity of GnRH-producing neurons depends on their migration to the hypothalamic region at the early stages of ontogenesis, which is controlled by anosmin, ephrins, and lactosamine-rich surface glycoconjugate. Dysregulation of the migration of GnRH-producing neurons and the impaired production and secretion of GnRH, lead to hypogonadotropic hypogonadism and other dysfunctions of the reproductive system. This review is devoted to the current state of the problem of regulating the synthesis and secretion of GnRH, the mechanisms of migration of hypothalamic GnRH-producing neurons at the early stages of brain development, the functional activity of the GnRH-producing neurons in the adult hypothalamus and the molecular mechanisms of GnRH action on the pituitary gonadotrophs. New experimental data are analyzed, which significantly change the current understanding of the functioning of GnRH-producing neurons and the secretion of GnRH, which is very important for the development of effective approaches for correcting the functions of the HPG axis.

Prospects of intranasal insulin for correction of cognitive impairments, in particular those associated with diabetes mellitus.

Despite the well-studied effect of insulin in peripheral tissues, its role in functioning of the central nervous system is much less understood. The effects of insulin in the brain are extremely diverse: insulin plays an important role in neuron growth and differentiation, affects higher cognitive functions (in particular, the formation of long-term memory), and also has a neuroprotective effect. Both peripheral and central insulin resistance as well as absolute insulin deficiency impairs the functional activity of neurons and neurogenesis. Several studies have investigated intranasal administration of insulin as a potential way for correction of these disorders. The review presents data on abnormalities of the insulin signaling system in the brain in diabetes mellitus, which is accompanied by cognitive dysfunction of varying severity and is associated with the development of neurodegenerative disorders, including Alzheimer's disease. We analyzed the results of studies on the use of intranasal insulin in animal models with diabetes mellitus, healthy volunteers, and patients with cognitive impairments.

Stability and pinning synchronization analysis of fractional order delayed Cohen–Grossberg neural networks with discontinuous activations

This article, we explore the asymptotic stability and asymptotic synchronization analysis of fractional order delayed Cohen–Grossberg neural networks with discontinuous neuron activation functions (FCGNNDDs). First, under the framework of Filippov theory and differential inclusion theoretical analysis, the global existence of Filippov solution for FCGNNDDs is studied by means of the given growth condition. Second, by virtue of suitable Lyapunov functional, Young inequality and comparison theorem for fractional order delayed linear system, some global asymptotic stability conditions for such system is derived by limiting discontinuous neuron activations. Third, the global asymptotic synchronization condition for FCGNNDDs is obtained based on the pinning control. At last, two numerical simulations are given to verify the theoretical findings.

Phospholipids of Synaptic Membranes in the Pathogenesis of Encephalopathy During Hemorrhagic Shock (Review)

Correction of brain cell damages caused by massive blood loss is one of the urgent problems of hemorrhagic shock, which ensures the need in clarification of mechanisms of such damages with the prospect of developing strategies to restore the functional activity of neurons. Analysis of the data presented in the review suggests that the dysregulation of phospholipid metabolism underlies both structural damage of synaptic membranes and their functions, including receptor signaling, the disturbances of which lead to encephalopathy in hemorrhagic shock. Correction of synaptic membranes phospholipid composition seems to possess a potential for increasing the effectiveness of treatment of shock-induced brain function disorders.

Stochastic reliable synchronization for coupled Markovian reaction–diffusion neural networks with actuator failures and generalized switching policies

This paper is concerned with the synchronization of coupled Markovian reaction–diffusion neural networks (RDNNs) with actuator failures and generalized switching policies (GSPs) via mode-dependent reliable control. Different from some existing results with all known transition rates, GSPs are considered for coupled Markovian RDNNs, where each transition rate can be completely unknown or only its estimation is known, or each transition rate of some modes is completely unknown. To reflect more realistic behaviors, actuator failures are considered for coupled Markovian RDNNs, and a mode-dependent reliable control scheme is proposed. Then, a new Lyapunov–Krasovskii functional (LKF) is introduced, which fully utilizes the information on the slope of neuron activation functions. Based on the LKF, a synchronization criterion is established in the form of linear matrix inequalities (LMIs). Moreover, the mode-dependent reliable control gains are obtained. Finally, a numerical example is given to verify the effectiveness of the proposed results.

Dynamical Effects of Neuron Activation Gradient on Hopfield Neural Network: Numerical Analyses and Hardware Experiments

Hyperbolic tangent function, a bounded monotone differentiable function, is usually taken as a neuron activation function, whose activation gradient, i.e. gain scaling parameter, can reflect the response speed in the neuronal electrical activities. However, the previously published literatures have not yet paid attention to the dynamical effects of the neuron activation gradient on Hopfield neural network (HNN). Taking the neuron activation gradient as an adjustable control parameter, dynamical behaviors with the variation of the control parameter are investigated through stability analyses of the equilibrium states, numerical analyses of the mathematical model, and experimental measurements on a hardware level. The results demonstrate that complex dynamical behaviors associated with the neuron activation gradient emerge in the HNN model, including coexisting limit cycle oscillations, coexisting chaotic spiral attractors, chaotic double scrolls, forward and reverse period-doubling cascades, and crisis scenarios, which are effectively confirmed by neuron activation gradient-dependent local attraction basins and parameter-space plots as well. Additionally, the experimentally measured results have nice consistency to numerical simulations.

Multi-rotor unmanned aerial vehicle control system based on hybrid neuroregulator

Compact multirotor unmanned aerial vehicles are used to solve a variety of problems, from aerial photography to reconnaissance and goods delivery, but at present, their widespread use is constrained by the imperfection of stabilization and navigation systems. The problem can be solved by methods of neurocontrol, which are known to be an essential tool when nonlinear behavior of the craft should be taken into account. A new approach for creation of a multirotor aircraft control system is considered based on a hybrid parallel neurocontroller that uses an artificial neural network of radial basis functions. Parallel neurocontrol provides for the simultaneous use of both the conventional proportional-integral-derivative controller and the neurocontroller in the control loop, their outputs being combined. In order to optimize the calculations performed during the second layer of the neurocontroller network training, a piecewise linear function of N segments, symmetric about zero, was chosen as the function for activating the neurons of the second layer. A low-cost method is proposed for parameter optimization of the activation functions of the hidden layer neurons. It allows for network training in real time. The proposed hybrid parallel neurocontroller is implemented and studied based on an artificial neural network of radial basis functions. Bench tests have shown that the stabilization of the three axes of the hybrid neurocontroller provides quality control better than a conventional manually tuned proportional-integral-derivative controller, namely, reducing the transition time, lower amplitude oscillations in the stabilization of the unmanned aerial vehicle and increase the stability of the craft to external influences. The use of hybrid parallel neurocontroller will help to solve the problems of maneuvering and stabilization of the craft for the transition to the local navigation problem solution.

Glutamate diffusion in the rat brain in vivo under light and deep anesthesia conditions.

Glutamate (Glu) is the most abundant neurotransmitter in the human central nervous system and glutamatergic neurotransmission has been implicated in many common and severe neuropsychiatric disorders. In vivo MRS techniques have been developed to measure brain Glu concentration to investigate the pathophysiology of various brain disorders. However, it is difficult to interpret Glu signal changes because Glu plays multiple roles in the brain and is found in multiple microenvironments including cytosolic, vesicular, and extracellular. In vivo diffusion-weighted MRS (DW-MRS) with low to very high b-values was performed on the rat prefrontal cortex at 9.4T under both light and deep anesthetic conditions to examine Glu diffusion properties. Significant alterations in Glu diffusion as well as reduced Glu concentration were observed under deep anesthesia compared with superficial anesthesia in the absence of similar changes in NAA or creatine. The modifications in Glu diffusion under deep anesthesia might reflect changes in Glu microenvironment. The present work shows that Glu DW-MRS could be an important tool to explore Glu physiology with changing levels of neuronal activity and synaptic function.

Various Threshold Switching Devices for Integrate and Fire Neuron Applications

AbstractThis study demonstrates an integrate and fire (I&F) neuron using threshold switching (TS) devices to implement spike‐based neuromorphic system. An I&F neuron can be realized using the hysteric voltage switch characteristics of a TS device. To investigate the effects of various TS devices on neuron behavior, neurons are compared using three different types of TS device: NbO2‐based insulator‐to‐metal transition (IMT) device, B–Te‐based ovonic threshold switching device, and Ag/HfO2‐based atomic‐switching TS device. The results show that the off‐state resistance and switching time of the TS devices determine the leaky/nonleaky characteristics and types of activation function of neuron, respectively. In addition, it is confirmed that the threshold voltage and on‐state resistance of the TS device determine the total power consumption of neuron. Furthermore, the feasibility of the TS devices in a spiking neural network is verified by simulation program with integrated circuit emphasis (SPICE) simulation. This work indicates applicability of TS‐based I&F neuron in neuromorphic hardware application.

Dopamine neuronal protection in the mouse Substantia nigra by GHSR is independent of electric activity

ObjectiveDopamine neurons in the Substantia nigra (SN) play crucial roles in control of voluntary movement. Extensive degeneration of this neuronal population is the cause of Parkinson's disease (PD). Many factors have been linked to SN DA neuronal survival, including neuronal pacemaker activity (responsible for maintaining basal firing and DA tone) and mitochondrial function. Dln-101, a naturally occurring splice variant of the human ghrelin gene, targets the ghrelin receptor (GHSR) present in the SN DA cells. Ghrelin activation of GHSR has been shown to protect SN DA neurons against 1-methyl-4-phenyl-1,2,5,6 tetrahydropyridine (MPTP) treatment. We decided to compare the actions of Dln-101 with ghrelin and identify the mechanisms associated with neuronal survival. MethodsHistologial, biochemical, and behavioral parameters were used to evaluate neuroprotection. Inflammation and redox balance of SN DA cells were evaluated using histologial and real-time PCR analysis. Designer Receptors Exclusively Activated by Designer Drugs (DREADD) technology was used to modulate SN DA neuron electrical activity and associated survival. Mitochondrial dynamics in SN DA cells was evaluated using electron microscopy data. ResultsHere, we report that the human isoform displays an equivalent neuroprotective factor. However, while exogenous administration of mouse ghrelin electrically activates SN DA neurons increasing dopamine output, as well as locomotion, the human isoform significantly suppressed dopamine output, with an associated decrease in animal motor behavior. Investigating the mechanisms by which GHSR mediates neuroprotection, we found that dopamine cell-selective control of electrical activity is neither sufficient nor necessary to promote SN DA neuron survival, including that associated with GHSR activation. We found that Dln101 pre-treatment diminished MPTP-induced mitochondrial aberrations in SN DA neurons and that the effect of Dln101 to protect dopamine cells was dependent on mitofusin 2, a protein involved in the process of mitochondrial fusion and tethering of the mitochondria to the endoplasmic reticulum. ConclusionsTaken together, these observations unmasked a complex role of GHSR in dopamine neuronal protection independent on electric activity of these cells and revealed a crucial role for mitochondrial dynamics in some aspects of this process.

Neuromorphic photonics with electro-absorption modulators.

Photonic neural networks benefit from both the high-channel capacity and the wave nature of light acting as an effective weighting mechanism through linear optics. Incorporating a nonlinear activation function by using active integrated photonic components allows neural networks with multiple layers to be built monolithically, eliminating the need for energy and latency costs due to external conversion. Interferometer-based modulators, while popular in communications, have been shown to require more area than absorption-based modulators, resulting in a reduced neural network density. Here, we develop a model for absorption modulators in an electro-optic fully connected neural network, including noise, and compare the network's performance with the activation functions produced intrinsically by five types of absorption modulators. Our results show the quantum well absorption modulator-based electro-optic neuron has the best performance allowing for 96% prediction accuracy with 1.7×10-12 J/MAC excluding laser power when performing MNIST classification in a 2 hidden layer feed-forward photonic neural network.

Neural network for prediction of hardness profiles for steel alloys after plasma nitriding

Plasma nitriding is a state-of-the-art thermochemical treatment, which is used by the processing industry for hardening of the surface-near material volumes of tools and components. An artificial neural network, particularly backpropagation neural network, has been developed to investigate the hardness behavior of different steel types according to various process parameters. Therefor eleven steels with different chemical compositions were plasma nitrated with different concentrations of nitrogen at various temperatures. The hardness profiles were studied with indentation measurement. The experimental data were used to train the neural network with the Levenberg-Marquardt algorithm. For an optimal training, weight distribution, hidden neurons and gradients of neuron activation functions were varied, to allow simulations of hardness profiles for the different types of steel with deviant nitriding parameters.

Learning-induced sensory plasticity of mouse olfactory epithelium

Traditionally, studies of the neurobiology of learning and memory focus on the circuitry that interfaces between sensory inputs and behavioral outputs, such as the amygdala and cerebellum. However, evidence is accumulating that some forms of learning can in fact drive stimulus­specifc changes very early in sensory systems, including not only primary sensory cortices but also precortical structures and even the peripheral sensory organs themselves. In this study, we investigated the effect of olfactory associative training on the functional activity of olfactory epithelium neurons in response to an indifferent stimulus (orange oil). It was found that such a peripheral structure of the olfactory system of adult mice as the olfactory epithelium (OE) demonstrates experience­dependent plasticity. In our experiment, associative learning led to changes in the patterns of OE cell activation in response to orange oil in comparison with the control group and animals that were given odor without reinforcement. To interpret the results obtained, we compared the distribution of MRI contrast across the zones of OE in response to a conditioned odor in trained animals and in control animals that were given orange oil at three concentrations: original (used for conditioning), 4­fold higher and 4­fold lower. Since the OE activation patterns obtained coincided in the group of trained animals and controls, which were stimulated with orange oil at the 4­fold higher concentration, it can be concluded that associative conditioning increased the sensitivity of the OE to the conditioned stimulus. The observed increase in OE response to orange oil may be the result of neurogenesis, i. e. the maturation of new olfactory neurons responsive to this stimulus, or the consequence of an increase in individual sensitivity of each OE neuron. Based on data of MRI contrast accumulation in mouse OE, the sensory plasticity way in learning­induced increase in sensitivity of OE to conditioned stimulus is more possible. Thus, the sensory plasticity of the OE plays a signifcant role in the formation of the neuronal response to the provision of an initially indifferent odor and is part of the adaptive responses to the environmental changing.

Epigenetic Mechanisms of Peptide-Driven Regulation and Neuroprotective Protein FKBP1b

Cortexin is a clinically approved cerebral cortex polypeptide complex in calves. The mechanism of cortexin action is not understood well. Two cortexin derivatives, short peptides EDR and DS with neuroprotective activity, were synthesized. According to the data of molecular modeling, these peptides are able to bind to the histone H1.3 protein. This can affect the conformation of histone H1.3, which leads to a change in the chromatin structure in the loci of some genes, in particular Fkbp1b encoding the FK506-binding protein. Electrophysiological processes associated with the Ca^(2+) exchange are disturbed in the pyramidal neurons of the hippocampus during aging of the brain. The Fkbp1b gene encodes peptidyl-prolyl cis-trans isomerase, regulating the release of calcium ions from the sarcoplasmic and endoplasmic reticulum of neurons. The activation of the Fkbp1b gene transcription under treatment with short peptides can promote the synthesis of its protein product and the activation of the Ca^(2+) release from organelles of the sarcoplasmic and endoplasmic reticulum of neurons, which, in turn, can lead to an increase in the functional activity of neurons.

Further Stability Analysis for Time-Delayed Neural Networks Based on an Augmented Lyapunov Functional

In this paper, the stability of time-delayed neural networks (DNN) is further analyzed. First, an augmented N-dependent Lyapunov-Krasovskii functional (LKF) is designed, where the non-integral terms are augmented with delay-dependent items and some additional state variables, and the integrated vector in the single-integral terms is also augmented by adding some integral interval-dependent items. The novel LKF complements some coupling information between the neuron activation function and other state variables. Second, a new delay-dependent stability criterion is proposed via the above LKF application. Third, in order to further demonstrate the advantages of the new LKF, two corollaries are also given under other simplistic LKFs. Finally, some common numerical examples are presented to show the effectiveness of the proposed approach.

Synaptic Localisation of Tau.

The microtubule-associated protein tau has been identified in several intraneuronal compartments, including in association with synapses. In Alzheimer's disease, frontotemporal dementia and related tauopathies, highly phosphorylated tau accumulates as intraneuronal protein aggregates that are likely responsible for the demise of neurons and the subsequent progressive cognitive decline. However, the molecular mechanisms underlying such tau-mediated damage in the tauopathies is not fully understood. Tauopathy induces loss of synapses, which is one of the earliest structural correlates of cognitive dysfunction and disease progression. Notably, altered post-translational modifications of tau, including increased phosphorylation and acetylation, augment the mislocalisation of tau to synapses, impair synaptic vesicle release and might influence the activity-dependent release of tau from neurons. Thus, disease-associated accumulation of modified tau at the synapse adversely affects critical neuronal processes that are linked to neuronal activity and synaptic function. These findings emphasise the importance of gaining a comprehensive understanding of the diverse roles of tau at distinct intraneuronal locations. An improved knowledge of the impact of synaptic tau under physiological and pathological conditions and how tau localisation impacts on neuronal function will provide valuable insights that may lead to the development of new therapies for the tauopathies.

Comparison of Neural Network–Based Adaptive Controllers Using Hypercomplex Numbers for Controlling Robot Manipulator

This study investigates an adaptive controller by applying a neural network, in which all the network parameters, states, signals and functions are expressed using hypercomplex numbers and algebras; its application to dynamics control of a robot manipulator. To design hypercomplex–valued neural networks where each neural network is a multilayer feedforward network with a split–type activation function of neurons using a tapped–delay–line input, we consider the following four types of hypercomplex numbers: complex, hyperbolic, bicomplex and quaternion numbers. In the control system, we utilise a feedback error–learning scheme to conduct the training of the network through a back–propagation algorithm. In the computational experiments, we explore a hypercomplex–valued neural network–based controller as a trajectory control problem of a three–link robot manipulator, in which the position of the end–effector follows to the desired trajectory in a 3–dimensional space. The simulation results validate the feasibility and effectiveness of the quaternion neural network–based controller for this task.