Neural Activation Functions Research Articles

Improving signal prediction performance of neural networks through multiresolution learning approach

In this paper, we extend the original work on multiresolution learning for neural networks, and present new developments on the multiresolution learning paradigm. The contributions of this paper include: (1) proposing a new concept and method of adjustable neural activation functions in multiresolution learning to improve neural network learning efficacy and generalization performance for signal predictions; (2) providing new insightful explanations for the multiresolution learning paradigm from a multiresolution optimization perspective; (3) exploring underlying ideas why the multiresolution learning scheme associated with adjustable activation functions would be more appropriate for the multiresolution learning paradigm; and (4) providing rigorous validations to evaluate the multiresolution learning paradigm with adjustable activation functions and comparing it with the schemes of multiresolution learning with fixed activation functions and traditional learning. This paper presents systematically new analytical and experimental results on the multiresolution learning approach for training an individual neural network model, demonstrates our integral solution on neural network learning efficacy, and illustrates the significant improvements on neural networks' generalization performance and robustness for nonlinear signal predictions.

Optimal implementation of neural activation functions in programmable logic using fuzzy logic

Abstract This work presents a methodology for implementing neural activation function in programmable logic using tools from fuzzy logic. This methodology will allow implementing these intrinsic non-linear functions using comparators and simple linear modellers, easily implemented in programmable logic. This work is particularized to the case of a hyperbolic tangent, the most common function in neural models, showing the excellent results yielded with the proposed approximation.

Programmable analogue VLSI implementation for asymmetric sigmoid neural activation function and its derivative

A new CMOS VLSI implementation of an asymmetric programmable sigmoid neural activation function, as well as of its derivative, is presented. It consists of two coupled PMOS and NMOS differential pairs with different programmable bias currents that set the upper and lower limits of the sigmoid. The circuit works in the weak inversion region, for low power consumption and exponential envelope, or in strong inversion to achieve higher speeds. The results obtained from the theoretical transfer function, and from the simulations of the circuit implemented in AMI's 0.35 µm technology, show a very good match.

Fault Detection and Diagnosis in Deep-trough Hydroponics using Intelligent Computational Tools

The intelligent computational tools of feedforward neural networks and genetic algorithms are used to develop a real-time detection and diagnosis system of specific mechanical, sensor and plant (biological) failures in a deep-trough hydroponic system. The capabilities of the system are explored and validated. In the process of designing the fault detection neural network model, a new technique for neural network designing and training parameterisation is developed, based on the heuristic optimisation method of genetic algorithms. Sensor and actuator faults are detected and diagnosed in sufficient time that the fault detection model can be applied on-line as a reliable supervisor of the operation of an unattended deep-trough hydroponic system. Biological faults were not detected in general. It seems that the interaction between plants and their root-zone microenvironment is not equally balanced, as the condition of the plants is highly influenced by the conditions in their root zone microenvironment, while these microenvironment conditions (as they are represented by the measurable variables) are not influenced in the same degree by the conditions of the plants. Finally, the genetic algorithm system developed here can be successfully applied to a combinatorial problem such as deciding the best neural network architecture, activation functions and training algorithm for a specific model.

Enhancement of speed and efficiency of an internet-based gear design optimisation

An internet-based gear design optimisation program has been developed for geographically dispersed teams to collaborate over the internet. The optimisation program implements genetic algorithm. A novel methodology is presented that improves the speed of execution of the optimisation program by integrating artificial neural networks into the system. The paper also proposes a method that allows an improvement to the performance of the back propagation-learning algorithm. This is done by rescaling the output data patterns to lie slightly below and above the two extreme values of the full range neural activation function. Experimental tests show the reduction of execution time by approximately 50%, as well as an improvement in the training and generalisation errors and the rate of learning of the network.

On the exact solution of the Parity- N problem using ordered neural networks

In this paper, an exact solution of the Parity- N function representation problem is provided using a fully connected feedforward (ordered) neural network. The solution requires neither training nor adaptation, and employs ( N−1) neurons. It is shown that any function σ: R→ R with the property: σ(−1)= σ(1)=1 and σ(0)=0, can be used as the neural network activation function.

A neural network model of general olfactory coding in the insect antennal lobe.

A central problem in olfaction is understanding how the quality of olfactory stimuli is encoded in the insect antennal lobe (or in the analogously structured vertebrate olfactory bulb) for perceptual processing in the mushroom bodies of the insect protocerebrum (or in the vertebrate olfactory cortex). In the study reported here, a relatively simple neural network model, inspired by our current knowledge of the insect antennal lobes, is used to investigate how each of several features and elements of the network, such as synapse strengths, feedback circuits and the steepness of neural activation functions, influences the formation of an olfactory code in neurons that project from the antennal lobes to the mushroom bodies (or from mitral cells to olfactory cortex). An optimal code in these projection neurons (PNs) should minimize potential errors by the mushroom bodies in misidentifying the quality of an odor across a range of concentrations while maximizing the ability of the mushroom bodies to resolve odors of different quality. Simulation studies demonstrate that the network is able to produce codes independent or virtually independent of concentration over a given range. The extent of this range is moderately dependent on a parameter that characterizes how long it takes for the voltage in an activated neuron to decay back to its resting potential, strongly dependent on the strength of excitatory feedback by the PNs onto antennal lobe intrinsic neurons (INs), and overwhelmingly dependent on the slope of the activation function that transforms the voltage of depolarized neurons into the rate at which spikes are produced. Although the code in the PNs is degraded by large variations in the concentration of odor stimuli, good performance levels are maintained when the complexity of stimuli, as measured by the number of component odorants, is doubled. When excitatory feedback from the PNs to the INs is strong, the activity in the PNs undergoes transitions from initial states to stimulus-specific equilibrium states that are maintained once the stimulus is removed. When this PN-IN feedback is weak the PNs are more likely to relax back to a stimulus-independent equilibrium state, in which case the code is not maintained beyond the application of the stimulus. Thus, for the architecture simulated here, strong feedback from the PNs onto the INs, together with step-like neuronal activation functions, could well be important in producing easily discriminable odor quality codes that are invariant over several orders of magnitude in stimulus concentration.

Discriminant component pruning. Regularization and interpretation of multi-layered back-propagation networks.

Neural networks are often employed as tools in classification tasks. The use of large networks increases the likelihood of the task's being learned, although it may also lead to increased complexity. Pruning is an effective way of reducing the complexity of large networks. We present discriminant components pruning (DCP), a method of pruning matrices of summed contributions between layers of a neural network. Attempting to interpret the underlying functions learned by the network can be aided by pruning the network. Generalization performance should be maintained at its optimal level following pruning. We demonstrate DCP's effectiveness at maintaining generalization performance, applicability to a wider range of problems, and the usefulness of such pruning for network interpretation. Possible enhancements are discussed for the identification of the optimal reduced rank and inclusion of nonlinear neural activation functions in the pruning algorithm.

Harmonic Analysis of Neural Networks

It is known that superpositions of ridge functions (single hidden-layer feedforward neural networks) may give good approximations to certain kinds of multivariate functions. It remains unclear, however, how to effectively obtain such approximations. In this paper, we use ideas from harmonic analysis to attack this question. We introduce a special admissibility condition for neural activation functions. The new condition is not satisfied by the sigmoid activation in current use by the neural networks community; instead, our condition requires that the neural activation function be oscillatory. Using an admissible neuron we construct linear transforms which represent quite general functionsfas a superposition of ridge functions. We develop••a continuous transform which satisfies a Parseval-like relation;••a discrete transform which satisfies frame bounds.Both transforms representfin a stable and effective way. The discrete transform is more challenging to construct and involves an interesting new discretization of time–frequency–direction space in order to obtain frame bounds for functions inL2(A) whereAis a compact set of Rn. Ideas underlying these representations are related to Littlewood–Paley theory, wavelet analysis, and group representation theory.

Global properties for a class of dynamical neural circuits

In this paper, we study the global properties of a class of asymmetrical Hopfield-type neural circuits. We first present a result for the existence and uniqueness of an equilibrium point; this result does not assume smoothness of the neural activation functions. Then we give some testable sufficient conditions for the global stability of such neural circuits. These results generalize a few previous known results and remove some restrictions on the neural circuits.