Nonlinear Activation Function Research Articles

Universality Laws for High-Dimensional Learning With Random Features

We prove a universality theorem for learning with random features. Our result shows that, in terms of training and generalization errors, a random feature model with a nonlinear activation function is asymptotically equivalent to a surrogate linear Gaussian model with a matching covariance matrix. This settles a so-called Gaussian equivalence conjecture based on which several recent papers develop their results. Our method for proving the universality theorem builds on the classical Lindeberg approach. Major ingredients of the proof include a leave-one-out analysis for the optimization problem associated with the training process and a central limit theorem, obtained via Stein’s method, for weakly correlated random variables.

Delay-Variation-Dependent Criteria on Extended Dissipativity for Discrete-Time Neural Networks With Time-Varying Delay.

This article is concerned with the extended dissipativity of discrete-time neural networks (NNs) with time-varying delay. First, the necessary and sufficient condition on matrix-valued polynomial inequalities reported recently is extended to a general case, where the variable of the polynomial does not need to start from zero. Second, a novel Lyapunov functional with a delay-dependent Lyapunov matrix is constructed by taking into consideration more information on nonlinear activation functions. By employing the Lyapunov functional method, a novel delay and its variation-dependent criterion are obtained to investigate the effects of the time-varying delay and its variation rate on several performances, such as performance, passivity, and performance, of a delayed discrete-time NN in a unified framework. Finally, a numerical example is given to show that the proposed criterion outperforms some existing ones.

All-Optical Nonlinear Activation Function Based on Germanium Silicon Hybrid Asymmetric Coupler

Nonlinear activation functions are crucial for optical neural networks (ONNs) to achieve more various functions. However, the current nonlinear functions suffer from some dilemma, including high power consumption, high loss, and limited bandwidth. Here, we propose and demonstrate an all-optical implementation of a nonlinear activation function based on germanium silicon hybrid integration. The principle lies in the intrinsic absorption and the carrier-induced refractive index change of germanium in C -band. It has a large operating bandwidth and a response frequency of 70 MHz, with a loss of 4.28 dB and a threshold power of 5.1 mW. Adopting it to the MNIST handwriting data set classification, it shows an improvement in accuracy from 91.6% to 96.8%. This proves that our scheme has great potential for advanced ONN applications.

Bias-Scalable Near-Memory CMOS Analog Processor for Machine Learning

Bias-scalable analog computing is attractive for implementing machine learning (ML) processors with distinct power-performance specifications. For instance, ML implementations for server workloads are focused on higher computational throughput for faster training, whereas ML implementations for edge devices are focused on energy-efficient inference. In this paper, we demonstrate the implementation of bias-scalable approximate analog computing circuits using the generalization of the margin-propagation principle called shape-based analog computing (S-AC). The resulting S-AC core integrates several near-memory compute elements, which include: (a) non-linear activation functions; (b) inner-product compute circuits; and (c) a mixed-signal compressive memory, all of which can be scaled for performance or power while preserving its functionality. Using measured results from prototypes fabricated in a 180nm CMOS process, we demonstrate that the performance of computing modules remains robust to transistor biasing and variations in temperature. In this paper, we also demonstrate the effect of bias-scalability and computational accuracy on a simple ML regression task.

Stationary states of a dissipative two-qubit quantum channel and their applications for quantum machine learning

Entangled-state preparation and preservation are the cornerstones of any quantum information platform. However, the strongest adversaries in quantum information science are unwanted environmental effects such as decoherence and dissipation. Here, we address how to control and harness these unwanted effects that arise from the coupling of a system with its environment, to provide stationary entangled states for quantum machine learning (QML). To do so, we design a dissipative quantum channel, i.e., a two-qubit system interacting with a squeezed vacuum field reservoir, and study the output state of the channel by solving the corresponding master equation, especially, in the small squeezing regime. We show that the time-dependent output state of the channel is the so-called two-qubit X-states that generalize many families of entangled two-qubit states. Also, by considering a general Bell diagonal state as the initial state of the system, we reveal that this dissipative channel generates two well-known classes of entangled mixed state and Werner-like states in the steady-state regime. Moreover, this channel provides an efficient way to determine whether a given initial state results in a stationary entangled state or not. Finally, we examine the potential application of the designed two-qubit channel for QML. In this line, we propose a general theoretical scheme for quantum neural networks (QNNs) implemented with the variational quantum circuits, which encode data in continuous variables (CVs) of the two-qubit states. The linear (also referred to as affine) and nonlinear (activation function) transformations are enacted in the QNN using the stationary states of the two-qubit channel and measurement process, respectively. Finally, we proceed to test our proposed model by solving some supervised binary classification tasks. Integrating the non-unitary transformation and parallel-processed neural computing on such a two-qubit channel establishes the requirements for a meaningful QNN. Such a CV-QNN model with sufficient layers may execute any algorithm implementable on a universal CV quantum computer.

Mixed [formula omitted] and passive exponential synchronization for singular Markov jump neural networks based on hybrid event trigger scheme

This article considers the driving-response-based exponential synchronization issue for delayed singular Markov jump neural networks by virtue of hybrid event triggering strategy. By employing singular value decomposition method, the regularity and impulse-freeness of synchronization error system are obtained. Under the framework of hybrid event-triggered scheme, the novel exponential admissibility conditions, mixed H∞ and passive performance are acquired by establishing improved Lyapunov–Krasovskill functional and dealing with nonlinear neuron activation function. Furthermore, the desired synchronization state feedback controller is designed by virtue of linear matrix inequalities. Finally, a numerical example and an analog resistance-capacitance network circuit are used to verify the effectiveness and practicability of the method.

Growing axons: greedy learning of neural networks with application to function approximation

Abstract We propose a new method for learning deep neural network models, which is based on a greedy learning approach: we add one basis function at a time, and a new basis function is generated as a non-linear activation function applied to a linear combination of the previous basis functions. Such a method (growing deep neural network by one neuron at a time) allows us to compute much more accurate approximants for several model problems in function approximation.

Robustness quantification method for network intrusion detection models

Deep learning techniques have been successfully applied to network intrusion detection tasks, but as in the case of autonomous driving and face recognition, the reliability of the system itself has become a pressing issue. Robustness is a key attribute to determine whether a deep learning system is secure and reliable, and we also choose to explore the security of intrusion detection models from a new perspective of robustness quantification. In this paper, we focus on the intrusion detection model based on long and short-term memory, and use a fine-grained linear approximation method to derive a more accurate robustness bound on the nonlinear activation function with tighter linear constraints. We can use this bound to quantitatively measure the robustness of the detection model and determine whether the model is susceptible to the influence of adversarial samples. In our experiments, we test networks with various structures on the MNIST dataset, and the results show that our proposed method can effectively deduce the robustness bounds of output elements, and has good scalability and applicability.

Intrinsic Lexical Intentionality and the Mathematics of Homomorphism

Moisl [1, 2] proposed a model of how the brain implements intrinsic intentionality with respect to lexical and sentence meaning, where 'intrinsic' is understood as 'independent of interpretation by observers external to the cognitive agent'. The discussion in both was mainly philosophical and qualitative; the present paper gives a mathematical account of the distance structure preservation that underlies the proposed mechanism of intrinsic intentionality. The three-layer autoassociative multilayer perceptron (aMLP) architecture with nonlinear hidden and linear output layers is the component in the model which generates representations homomorphic with the environment. The discussion first cites existing work which identifies the aMLP as an implementation architecture for principal component analysis (PCA), and then goes on to argue that the homomorphism characteristic of linear functions like PCA extends to aMLPs with nonlinear activation functions in the hidden layer. The discussion is in two main parts: the first part outlines the model, and the second presents the mathematical account.

A DEEP LEARNING NETWORK FOR MODELLING THE RELATIONSHIP OF THE REAL GDP, CO2 EMISSION AND RENEWABLE ENERGY CONSUMPTION FOR TURKIYE

In this study, the relationship between Türkiye’s real gross domestic product, CO2 emission and renewable energy consumption is modelled using machine learning techniques. The data between 2003-2020, the period when renewable energy production and consumption have accelerated, are included in the modelling. First of all, it was tested whether there is a causal relationship between the real gross domestic product, CO2 emission and renewable energy consumption data. Afterwards, a deep learning model was developed in Python programming language by considering the real gross domestic product as the dependent variable while CO2 emission and renewable energy consumption are considered as independent variables. The developed deep learning model has two input nodes, three hidden layers consisting of 100 neurons each, and an output node. In addition, the rectified unit functions are used as nonlinear activation functions in the deep learning network. Based on the standard usage, seventy percent of the data was used as the training data and the remaining thirty percent were employed as the test data. The results of the developed deep learning network and actual gross domestic product data were compared, and it is shown that the developed deep learning network successfully models the relationship between the real gross domestic product, CO2 emission and the renewable energy consumption. The coefficient of determination of the developed model was calculated in Python environment as R2=0.986. This parameter value also indicates that the developed deep learning network model has a good performance for the modelling of the real gross domestic product dependent on the CO2 emission and renewable energy consumption.

MXene-Nanoflakes-Enabled All-Optical Nonlinear Activation Function for On-Chip Photonic Deep Neural Networks.

2D metal carbides and nitrides (MXene) are promising material platforms for on-chip neural networks owing to their nonlinear saturable absorption effect. The localized surface plasmon resonances in metallic MXene nanoflakes may play an important role in enhancing the electromagnetic absorption; however, their contribution is not determined due to the lack of a precise understanding of its localized surface plasmon behavior. Here, a saturable absorber made of MXene thin film and a silicon waveguide with MXene flakes overlayer are developed to perform neuromorphic tasks. The proposed configurations are reconfigurable and can therefore be adjusted for various applications without the need to modify the physical structure of the proposed MXene-based activator configurations via tuning the wavelength of operation. The capability and feasibility of the obtained results of machine-learning applications are confirmed via handwritten digit classification task, with near 99% accuracy. These findings can guide the design of advanced ultrathin saturable absorption materials on a chip for a broad range of applications.

Global Solar Radiation Modelling using an Artificial Neural Network for Kazaure, Jigawa State, Nigeria

This research presents an algorithm based on Artificial Neural Networks (ANN), for estimating monthly mean daily and hourly values of solar global radiation. To effectively investigate solar energy consumption and estimate solar renewable energy resources, the Hourly Global Solar Radiation measurements are necessary. In order to predict monthly average daily global sun irradiance on a horizontal area of Kazaure- Nigeria, this study creates a model utilizing ANN to solve the problem of solar energy distribution. Five empirical correlations are developed using the data from 42 months to aid in the prediction of the solar energy distribution pattern. The software is constructed around the Multilayer Perceptron under categorized tabs, with Multilayer perception in neural network Toolbox in MATLAB 9.7 version as a feed forward ANN that maps sets of input data into a set of suitable output. It differs from conventional linear perception by employing three or more layers of neurons (nodes) with nonlinear activation functions. It is also more effective than perceptrons in identifying input that is not linearly separable by a linear hyper-plane. Results obtained utilizing the suggested structure reveals good agreement between the calculated and measured levels of global solar irradiation. The ANN model is shown to be superior when compared to empirical models, due to negligible noise margin.

A Method for Calculating the Derivative of Activation Functions Based on Piecewise Linear Approximation

Nonlinear functions are widely used as activation functions in artificial neural networks, which have a great impact on the fitting ability of artificial neural networks. Due to the complexity of the activation function, the computation of the activation function and its derivative requires a lot of computing resources and time during training. In order to improve the computational efficiency of the derivatives of the activation function in the back-propagation of artificial neural networks, this paper proposes a method based on piecewise linear approximation method to calculate the derivative of the activation function. This method is hardware-friendly and universal, it can efficiently compute various nonlinear activation functions in the field of neural network hardware accelerators. In this paper, we use least squares to improve a piecewise linear approximation calculation method that can control the absolute error and get less number of segments or smaller average error, which means fewer hardware resources are required. We use this method to perform a segmented linear approximation to the original or derivative function of the activation function. Both types of activation functions are substituted into a multilayer perceptron for binary classification experiments to verify the effectiveness of the proposed method. Experimental results show that the same or even slightly higher classification accuracy can be achieved by using this method, and the computation time of the back-propagation is reduced by 4–6% compared to the direct calculation of the derivative directly from the function expression using the operator encapsulated in PyTorch. This shows that the proposed method provides an efficient solution of nonlinear activation functions for hardware acceleration of neural networks.

Event-Triggered Neural Network Control for LTI Systems

This paper addresses the problem of event-triggered control (ETC) of discrete-time linear time-invariant (LTI) systems stabilized by neural network controllers. The event-triggering mechanisms (ETMs) is proposed to update only a portion of the layers required to maintain stability and satisfactory performance of the feedback system, thus reducing the computational cost associated with the evaluation of the neural network. Sufficient convex conditions in the form of linear matrix inequalities (LMIs) are provided to compute the triggering parameters and characterize an estimate of the domain of attraction for the feedback system. The formulation is based on the use of Lyapunov theory and a set of generalized sector constraints that deal with the nonlinear activation functions, represented in this case by the saturation function. Optimization procedures are also formulated to effectively reduce the amount of computation in the neural network. An example borrowed from the literature is used to illustrate the effectiveness of the proposal.

VGG-COVIDNet: A Novel model for COVID detection from X-Ray and CT Scan images

In this research work, a new deep learning model named VGG-COVIDNet has been proposed which can classify COVID-19 cases from normal cases over X-Rays and CT scan images of lungs. Medical practitioners use the X-Rays and CT scan images of lungs to identify whether a person is infected from COVID or not. In present times, it is very important to give real time COVID prediction with high reliability of results. Deep learning models equipped with machine learning support have been found very influential in accurate prediction of COVID or Non-COVID cases in real time. However, there are some limitations associated with the performance of these model which are model size, achieving good balance of model size and accuracy, and making a single model fitting well for both X-Ray and CT Scan image datasets. Keeping in mind these performance constraints, this new model (VGG-COVIDNet) has been proposed for real time prediction of COVID cases with good balance of model size and accuracy working well for both type of datasets (CT Scan and X-Ray). In order to control model size, an improved version of VGG-16 architecture has been proposed which contains only 13 convolutional layers and 5 fully connected layers. Multiple dropout layers have been added in the proposed architecture which can drop some percentage of features and applies random transformations to decrease the model over-fitting issue. Keeping in mind the primary goal to increase the model accuracy the proposed model has been trained on different datasets with ReLU activation function which is one of the best non-linear activation functions. Four different capacity datasets with CT scan and X-Ray images have been used to validate the performance of proposed model. The proposed model gives an overall accuracy of more than 90% on both types of input datasets i.e. X-Ray and CT Scan.

Noise influence on recurrent neural network with nonlinear neurons

The purpose of this study is to establish the features of noise propagation and accumulation in a recurrent neural network using a simplified echo network as an example. In this work, we studied the influence of activation function of artificial neurons and the connection matrices between them. Methods. We have considered white Gaussian noise sources. We used additive, multiplicative and mixed noise depending on how the noise is introduced into artificial neurons. The noise impact was estimated using the dispersion (variance) of the output signal. Results. It is shown that the activation function plays a significant role in noise accumulation. Two nonlinear activation functions have been considered: the hyperbolic tangent and the sigmoid function with range form 0 to 1. It is shown that some types of noise are suppressed in the case of the second function. As a result of considering the influence of coupling matrices, it was found that diagonal coupling matrices with a large blurring coefficient lead to less noise accumulation in the echo network reservoir with an increase in the reservoir memory influence. Conclusion. It is shown that activation functions of the form of sigmoid with range from 0 to 1 are suitable for suppressing multiplicative and mixed noise. The accumulation of noise in the reservoir was considered for three types of coupling matrices inside the reservoir: a uniform matrix, a band matrix with a small blurring coefficient, and a band matrix with a large blurring coefficient. It has been found that the band matrix echo networks with a high blurring coefficient accumulates the least noise. This holds for both additive and multiplicative noise.

HcLSH: A Novel Non-Linear Monotonic Activation Function for Deep Learning Methods

HcLSH: A Novel Non-Linear Monotonic Activation Function for Deep Learning Methods

Real-Valued Optical Matrix Computing with Simplified MZI Mesh

The Mach–Zehnder interferometer (MZI) mesh, a mainstream structure for optical matrix-vector multiplication (MVM), has been widely employed in recently developed optical neural networks (ONNs) and combination optimization problem solvers. The conventional MZI mesh was designed specifically for complex-valued optical MVM. The network includes 2 N 2 phase shifters, and coherent detection is indispensable for retrieving the output complex-valued vectors. Nonetheless, the majority of applications, including ONNs, merely require real-valued optical matrices with N 2 degrees of freedom (DOFs). The DOF gap between the 2 types of matrices results in a severe redundancy in the number of phase shifters when the conventional MZI mesh is applied to implement real-valued optical MVM. In this study, we propose a simplified MZI mesh for performing real-valued incoherent optical MVM. It has N 2 phase shifters and an optical depth of N + 1, and it outperforms the conventional MZI mesh. Furthermore, we constructed an ONN with the proposed MZI mesh and successfully performed the iris classification task via in situ training of particle swarm optimization. More importantly, we introduced a matched on-chip nonlinear activation function, so the proposed MZI mesh can be cascaded onto a single chip. Overall, the proposed real-valued MZI mesh and in situ training method are space efficient, energy efficient, scalable, and robust to fabrication errors. Therefore, they are suitable for large-scale ONNs.

Research on Optimization of 3D Printing Manufacturing Technology Based on Segmentation Algorithm

3D printing technology is a relatively mature processing technology. In actual production, external support needs to be added to ensure the smooth progress of the processing process. Tree support is a new type of external support method. Combined with instance segmentation technology in artificial intelligence, it can better solve the problems of collapse, deformation, and lack of precision in the 3D printing process and improve production efficiency. For the feature loss problem in the printing model, a new instance segmentation algorithm is adopted, including deepening the feature extraction architecture and using the nonlinear activation function Mish. For the feature reconstruction after instance segmentation, this paper optimizes and improves the classical concealment method, including the spatial range concealment method Z-buffer algorithm and the area scan line algorithm of the plane range concealment method to realize the projection mapping transformation of the 3D object model by combining coordinate refinement and Z-buffer algorithm.

Principles and Applications for Optical Nonlinear Activation Function Devices

Principles and Applications for Optical Nonlinear Activation Function Devices