Nonlinear Activation Function Research Articles

An on-chip photonic deep neural network for image classification.

Deep neural networks with applications from computer vision to medical diagnosis1-5 are commonly implemented using clock-based processors6-14, in which computation speed is mainly limited by the clock frequency and the memory access time. In the optical domain, despite advances in photonic computation15-17, the lack of scalable on-chip optical non-linearity and the loss of photonic devices limit the scalability of optical deep networks. Here we reportan integrated end-to-end photonic deep neural network (PDNN) that performs sub-nanosecond image classification through direct processing of the optical waves impinging on the on-chip pixel array as they propagate through layers of neurons. In each neuron, linear computation is performed optically and the non-linear activation function is realized opto-electronically, allowing a classification time of under 570 ps, which is comparable with a single clock cycle of state-of-the-art digital platforms. A uniformly distributed supply light provides the same per-neuron optical output range, allowing scalability to large-scale PDNNs. Two-class and four-class classification of handwritten letters with accuracies higher than 93.8% and 89.8%, respectively, is demonstrated. Direct, clock-less processing of optical data eliminates analogue-to-digital conversion and the requirement for a large memory module, allowing faster and more energyefficient neural networks for the next generations of deep learning systems.

Programmable chalcogenide-based all-optical deep neural networks

Abstract We demonstrate a passive all-chalcogenide all-optical perceptron scheme. The network’s nonlinear activation function (NLAF) relies on the nonlinear response of Ge2Sb2Te5 to femtosecond laser pulses. We measured the sub-picosecond time-resolved optical constants of Ge2Sb2Te5 at a wavelength of 1500 nm and used them to design a high-speed Ge2Sb2Te5-tuned microring resonator all-optical NLAF. The NLAF had a sigmoidal response when subjected to different laser fluence excitation and had a dynamic range of −9.7 dB. The perceptron’s waveguide material was AlN because it allowed efficient heat dissipation during laser switching. A two-temperature analysis revealed that the operating speed of the NLAF is ≤ 1 $\le 1$ ns. The percepton’s nonvolatile weights were set using low-loss Sb2S3-tuned Mach Zehnder interferometers (MZIs). A three-layer deep neural network model was used to test the feasibility of the network scheme and a maximum training accuracy of 94.5% was obtained. We conclude that combining Sb2S3-programmed MZI weights with the nonlinear response of Ge2Sb2Te5 to femtosecond pulses is sufficient to perform energy-efficient all-optical neural classifications at rates greater than 1 GHz.

Prescribed-Time Convergent Adaptive ZNN for Time-Varying Matrix Inversion under Harmonic Noise

Harmonic noises widely exist in industrial fields and always affect the computational accuracy of neural network models. The existing original adaptive zeroing neural network (OAZNN) model can effectively suppress harmonic noises. Nevertheless, the OAZNN model’s convergence rate only stays at the exponential convergence, that is, its convergence speed is usually greatly affected by the initial state. Consequently, to tackle the above issue, this work combines the dynamic characteristics of harmonic signals with prescribed-time convergence activation function, and proposes a prescribed-time convergent adaptive ZNN (PTCAZNN) for solving time-varying matrix inverse problem (TVMIP) under harmonic noises. Owing to the nonlinear activation function used having the ability to reject noises itself and the adaptive term also being able to compensate the influence of noises, the PTCAZNN model can realize double noise suppression. More importantly, the theoretical analysis of PTCAZNN model with prescribed-time convergence and robustness performance is provided. Finally, by varying a series of conditions such as the frequency of single harmonic noise, the frequency of multi-harmonic noise, and the initial value and the dimension of the matrix, the comparative simulation results further confirm the effectiveness and superiority of the PTCAZNN model.

An Efficient Asymmetric Nonlinear Activation Function for Deep Neural Networks

As a key step to endow the neural network with nonlinear factors, the activation function is crucial to the performance of the network. This paper proposes an Efficient Asymmetric Nonlinear Activation Function (EANAF) for deep neural networks. Compared with existing activation functions, the proposed EANAF requires less computational effort, and it is self-regularized, asymmetric and non-monotonic. These desired characteristics facilitate the outstanding performance of the proposed EANAF. To demonstrate the effectiveness of this function in the field of object detection, the proposed activation function is compared with several state-of-the-art activation functions on the typical backbone networks such as ResNet and DSPDarkNet. The experimental results demonstrate the superior performance of the proposed EANAF.

Impact of activation functions and number of layers on detection of exudates using circular Hough transform and convolutional neural networks

Convolutional neural networks (CNN), which are used for object detection, are widely used in image recognition and segmentation fields due to their good performance. In this study, a method is proposed for the detection of exudate from DR lesions based on image processing and CNN model with different activation functions and layer numbers. Activation functions are one of the main reasons why the CNN model is complicated in hierarchical structure. CNN models have real artificial intelligence capabilities, especially thanks to non-linear activation functions. Although ReLU is the most commonly used activation function in CNN models, it has some shortcomings in practice. The most important shortcoming is that if the input value is negative, the learning process will slow down due to the inability to take the derivative. In order to solve this problem, the effect of the activation function in the CNN model on a real world problem and its image was investigated in this study. By applying circular Hough transform, one of the image processing methods, the OD region, which has a similar structure to the exuding regions, was determined and removed from the image, and then the detection of the exuding regions was carried out using the CNN model. Exudate detection was performed using ReLU, ELU, Leaky ReLU, Softplus and Swish activation functions in the CNN model and the results were compared. Experimental results in DiaretDB0 and DiaretDB1 databases show that the Swish activation function has a better performance than the others.

All-optical ultrafast ReLU function for energy-efficient nanophotonic deep learning

Abstract In recent years, the computational demands of deep learning applications have necessitated the introduction of energy-efficient hardware accelerators. Optical neural networks are a promising option; however, thus far they have been largely limited by the lack of energy-efficient nonlinear optical functions. Here, we experimentally demonstrate an all-optical Rectified Linear Unit (ReLU), which is the most widely used nonlinear activation function for deep learning, using a periodically-poled thin-film lithium niobate nanophotonic waveguide and achieve ultra-low energies in the regime of femtojoules per activation with near-instantaneous operation. Our results provide a clear and practical path towards truly all-optical, energy-efficient nanophotonic deep learning.

Convolutional Neural Network-Aided DP-64 QAM Coherent Optical Communication Systems

Optical nonlinearity impairments have been a major obstacle for high-speed, long-haul and large-capacity optical transmission. In this paper, we propose a novel convolutional neural network (CNN)-based perturbative nonlinearity compensation approach in which we reconstruct a feature map with two channels that rely on first-order perturbation theory and build a classifier and a regressor as a nonlinear equalizer. We experimentally demonstrate the CNN equalizer in 375 km 120-Gbit/s dual-polarization 64-quadrature-amplitude modulation (64-QAM) coherent optical communication systems. We studied the influence of the dropout value and nonlinear activation function on the convergence of the CNN equalizer. We measured the bit-error-ratio (BER) performance with different launched optical powers. When the channel size is 11, the optimum BER for the CNN classifier is 0.0012 with 1 dBm, and for the CNN regressor, it is 0.0020 with 0 dBm; the BER can be lower than the 7 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\%$</tex-math></inline-formula> hard decision-forward threshold of 0.0038 from −3 dBm to 3 dBm. When the channel size is 15, the BERs at −4 dBm, 4 dBm and 5 dBm can be lower than 0.0020. The network complexity is also analyzed in this paper. Compared with perturbative nonlinearity compensation using a fully connected neural network (2392-64-64), we can verify that the time complexity is reduced by about 25 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\%$</tex-math></inline-formula> , while the space complexity is reduced by about 50 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\%$</tex-math></inline-formula> .

Fast nonlinear Hoyergram for bearings fault diagnosis under random impact interference

Random impact interference has always been an important subject in fault diagnosis. In the fast kurtogram (FK), kurtosis is a sparsity index used to locate optimal resonance frequency bands. However, the shortcomings of the kurtosis measure are a tendency to select the random impact with a large amplitude and a relatively weak sparse measure ability, which reduces the accuracy and anti-interference of the FK method. To overcome the effects of random impact interference, this paper proposes a novel bearing fault diagnosis method named the fast nonlinear Hoyergram (FNH), in which the nonlinear Hoyer index is employed to replace kurtosis to improve the fault representation capability under random impulse interference. Firstly, Z-score normalization and a generalized nonlinear sigmoid activation function are used for signal preprocessing, and the scale distribution of the signal is changed to weaken the impact of random impact interference. Secondly, the Hoyer index, which can be viewed as the normalized form of the norm, is used to replace kurtosis to improve the sparsity measure capability. Thirdly, in the Hoyergram, the frequency band with the largest Hoyer value can be chosen as the best resonance frequency band for the squared envelope analysis. Finally, the proposed FNH is compared with the FK method through simulation and experimental signals, and the effectiveness of the proposed method is verified.

Handling information loss of graph convolutional networks in collaborative filtering

Collaborative filtering (CF) methods based on graph convolutional network (GCN) and autoencoder (AE) achieve outstanding performance. But the GCN-based CF methods suffer from information loss problems, which are caused by information lossy initialization and using low-order Chebyshev Polynomial to fit the graph convolution kernel. And the AE-based CF methods obtain the prediction results by reconstructing the user-item interaction matrix, which does not conduct deep excavation of the behavior patterns, resulting in the limited-expression ability.To solve the above problems, we propose Variational AutoEncoder-Enhanced Graph Convolutional Network (VE-GCN) for CF. Specifically, we use a variational autoencoder (VAE) to compress the interactive behavior patterns as the prior information of GCN to achieve sufficient learning, thus alleviating the information lossy initialization problem. And then the generalized graph Laplacian convolution kernel is proposed in GCN to handle the high-frequency information loss problem caused by Chebyshev Polynomial fitting in the GCN-based CF. To the best of our knowledge, VE-GCN is a feasible method to handle the information loss problems mentioned above in GCN-based CF for the first time. Meanwhile, the structure of GCN is optimized by removing redundant feature transformation and nonlinear activation function, and using DenseGCN to complete multi-level information interaction. Experiments on four real-world datasets show that the VE-GCN achieves state-of-the-art performance.

Multistability analysis of delayed recurrent neural networks with a class of piecewise nonlinear activation functions

This paper studies the multistability of delayed recurrent neural networks (DRNNs) with a class of piecewise nonlinear activation functions. The coexistence as well as the stability of multiple equilibrium points (EPs) of DRNNs are proved. With the Brouwer’s fixed point theorem as well as the Lagrange mean value theorem, it is obtained that under some conditions, the n-neuron DRNNs with the proposed activation function can have at least 5n EPs and 3n of them are locally stable. Compared with the DRNNs with sigmoidal activation functions, DRNNs with this kind of activation function can have more total EPs and more locally stable EPs. It implies that when designing DRNNs with the proposed activation function to apply in associative memory, it can have an even larger storage capacity. Furthermore, it is obtained that there exists a relationship between the number of the total EPs/stable EPs and the frequency of the sinusoidal function in the proposed activation function. Last, the above obtained results are extended to a more general case. It is shown that, DRNNs with the extended activation function can have (2k+1)n EPs, (k+1)n of which are locally stable, therein k is closely related to the frequency of the sinusoidal function in the extended activation function. Two simulation examples are given to verify the correctness of the theoretical results.

Hybrid Method for Detecting Anomalies in Cosmic ray Variations Using Neural Networks Autoencoder

Cosmic rays were discovered by the Austrian physicist Victor Hess in 1912 in a series of balloon experiments performed between 1911 and 1912. Cosmic rays are an integral part of fundamental and applied research in the field of solar–terrestrial physics and space weather. Cosmic ray data are applied in different fields from the discovery of high-energy particles coming to Earth from space, and new fundamental symmetries in the laws of nature, to the knowledge of residual matter and magnetic fields in interstellar space. The properties of interplanetary space are determined from intensity variations, angular distribution, and other characteristics of galactic cosmic rays. The measure of cosmic ray flux intensity variability is used as one of the significant space weather factors. The negative impact of cosmic rays is also known. The negative impact can significantly increase the level of radiation hazard and pose a threat to astronauts, crews, and passengers of high-altitude aircraft on polar routes and to modern space equipment. Therefore, methods aimed at timely detection and identification of anomalous manifestations in cosmic rays are of particular practical relevance. The article proposes a method for analyzing cosmic ray variations and detecting anomalous changes in the rate of galactic cosmic ray arrival to the Earth. The method is based on a combination of the Autoencoder neural network with wavelet transform. The use of non-linear activation functions and the ability to flexibly change the structure of the network provide the ability of the Autoencoder to approximate complex dependencies in the recorded variations of cosmic rays. The article describes the numerical operations of the method implementation. Verification of the adequacy of the neural network model is based on the use of Box–Ljung Q-statistics. On the basis of the wavelet transform constructions, data-adaptive operations for detecting complex singular structures are constructed. The parameters of the applied threshold functions are estimated with a given confidence probability based on the α-quantiles of Student’s distribution. Using data from high-latitude neutron monitor stations, it is shown that the proposed method provides efficient detection of anomalies in cosmic rays during increased solar activity and magnetic storms. Using the example of a moderate magnetic storm on 10–11 May 2019, the necessity of applying different methods and approaches to the study of cosmic ray variations is confirmed, and the importance of taking them into account when making space weather forecast is shown.

A novel rolling bearing fault diagnosis method based on generalized nonlinear spectral sparsity

In the fast kurtogram (FK), kurtosis is used as an indicator to locate the fault frequency band, and is widely aplied to fault diagnosis. However, kurtosis has been proven to favor a single large impulse rather than the required small fault characteristics, especially in the strong interference environment. To eliminate the impact of large-amplitude impact and further improve the accuracy of fault extraction, a method based on generalized nonlinear spectral sparsity (GNSS) is proposed for fault diagnosis of bearings. First, Z-score normalization and generalized nonlinear sigmoid activation function are used for signal preprocessing, and the scale distribution of the signal will be changed to eliminate the effects of large amplitude shocks under noisy environment. Then, to improve the sparsity measure capability, an improved L3/2 norm is used to replace kurtosis as the basis for selecting the best resonance frequency band. Finally, the effectiveness of the GNSS is verified by simulation data and experimental data. Compared with FK method, the performance of fault extraction of the proposed method is significantly improved, especially for the interference of abnormal impact.

Neuromorphic photonic circuit modeling in Verilog-A

One of the significant challenges in neuromorphic photonic architectures is the lack of good tools to simulate large-scale photonic integrated circuits. It is crucial to perform simulations on a single platform to capture the circuit’s behavior in the presence of both optical and electrical components. Here, we adopted a Verilog-A based approach to model neuromorphic photonic circuits by considering both the electrical and optical properties. Verilog-A models for the primary optical devices, such as lasers, couplers, waveguides, phase shifters, and photodetectors, are discussed, along with studying the composite devices such as microring resonators. Model parameters for different optical devices are extracted and tuned by analyzing the measured data. The simulated and experimental results are also compared for validation of Verilog-A models. Finally, a single photonic neuron circuit is simulated by implementing input, weight, and non-linear activation function by using lasers, microring resonators, and modulator, respectively. Electro-optical rapid co-simulation would significantly improve the efficiency of optimizing the devices and provide an accurate simulation of the circuit performance.

Stability Analysis Using Quadratic Constraints for Systems With Neural Network Controllers

A method is presented to analyze the stability of feedback systems with neural network controllers. Two stability theorems are given to prove asymptotic stability and to compute an ellipsoidal innerapproximation to the region of attraction (ROA). The first theorem addresses linear time-invariant systems, and merges Lyapunov theory with local (sector) quadratic constraints to bound the nonlinear activation functions in the neural network. The second theorem allows the system to include perturbations such as unmodeled dynamics, slope-restricted nonlinearities, and time delay, using integral quadratic constraint (IQCs) to capture their input/output behavior. This in turn allows for off-by-one IQCs to refine the description of activation functions by capturing their slope restrictions. Both results rely on semidefinite programming to approximate the ROA. The method is illustrated on systems with neural networks trained to stabilize a nonlinear inverted pendulum as well as vehicle lateral dynamics with actuator uncertainty.

Programmable low-threshold optical nonlinear activation functions for photonic neural networks.

We experimentally demonstrate two types of programmable, low-threshold, optically controlled nonlinear activation functions, which are challenging to realize in photonic neural networks (PNNs). These devices rely on on-chip integrated Ge-Si photoelectric detectors and silicon electro-optical switches, and they generate rectified linear unit (ReLU) or sigmoid functions with arbitrary slopes without additional electrical processing. Both devices function at an extremely low threshold of 0.2 mW. The embedding of these nonlinear activation functions into convolutional neural networks facilitates the attainment of high inference accuracies of up to 95% when applied to Modified National Institute of Standards and Technology (MNIST) handwritten digit-classification tasks. The devices are suitable for low-power PNNs with an arbitrary number of propagation layers in photonic-computing chips.

Suboptimal Local Minima Exist for Wide Neural Networks with Smooth Activations

Does a large width eliminate all suboptimal local minima for neural nets? An affirmative answer was given by a classic result published in 1995 for one-hidden-layer wide neural nets with a sigmoid activation function, but this result has not been extended to the multilayer case. Recently, it was shown that, with piecewise linear activations, suboptimal local minima exist even for wide nets. Given the classic positive result on smooth activation and the negative result on nonsmooth activations, an interesting open question is: Does a large width eliminate all suboptimal local minima for deep neural nets with smooth activation? In this paper, we give a largely negative answer to this question. Specifically, we prove that, for neural networks with generic input data and smooth nonlinear activation functions, suboptimal local minima can exist no matter how wide the network is (as long as the last hidden layer has at least two neurons). Therefore, the classic result of no suboptimal local minimum for a one-hidden-layer network does not hold. Whereas this classic result assumes sigmoid activation, our counterexample covers a large set of activation functions (dense in the set of continuous functions), indicating that the limitation is not a result of the specific activation. Together with recent progress on piecewise linear activations, our result indicates that suboptimal local minima are common for wide neural nets.

Optical neural network quantum state tomography

Quantum state tomography (QST) is a crucial ingredient for almost all aspects of experimental quantum information processing. As an analog of the “imaging” technique in quantum settings, QST is born to be a data science problem, where machine learning techniques, noticeably neural networks, have been applied extensively. We build and demonstrate an optical neural network (ONN) for photonic polarization qubit QST. The ONN is equipped with built-in optical nonlinear activation functions based on electromagnetically induced transparency. The experimental results show that our ONN can determine the phase parameter of the qubit state accurately. As optics are highly desired for quantum interconnections, our ONN-QST may contribute to the realization of optical quantum networks and inspire the ideas combining artificial optical intelligence with quantum information studies.

Identifying key differences between linear stochastic estimation and neural networks for fluid flow regressions

Neural networks (NNs) and linear stochastic estimation (LSE) have widely been utilized as powerful tools for fluid-flow regressions. We investigate fundamental differences between them considering two canonical fluid-flow problems: (1) the estimation of high-order proper orthogonal decomposition coefficients from low-order their counterparts for a flow around a two-dimensional cylinder, and (2) the state estimation from wall characteristics in a turbulent channel flow. In the first problem, we compare the performance of LSE to that of a multi-layer perceptron (MLP). With the channel flow example, we capitalize on a convolutional neural network (CNN) as a nonlinear model which can handle high-dimensional fluid flows. For both cases, the nonlinear NNs outperform the linear methods thanks to nonlinear activation functions. We also perform error-curve analyses regarding the estimation error and the response of weights inside models. Our analysis visualizes the robustness against noisy perturbation on the error-curve domain while revealing the fundamental difference of the covered tools for fluid-flow regressions.

Zeroing Neural Networks for Dynamic Quaternion-Valued Matrix Inversion

This article, for the first time, extends the zeroing neural network (ZNN) method to address the problem of dynamic quaternion-valued matrix inversion. Due to the noncommutative property of quaternion multiplication, the complex representation method is first adopted to transform quaternion-valued matrices into the corresponding complex-valued matrices. Then, based on two kinds of ways to deal with nonlinear activation functions in the complex-valued domain, this article proposes two quaternion-valued ZNN (QVZNN) models for dynamic quaternion-valued matrix inversion. In addition, a novel nonlinear activation function is given to accelerate the convergence rate of the models to reach the predefined-time convergence. The detailed theoretical analysis, together with four theorems, are given to show the excellent properties of the QVZNN models. Furthermore, the upper bound of the convergence time is derived analytically with the residual error being zero theoretically. Finally, two numerical examples are provided to verify the theoretical results and the effectiveness of the QVZNN models for the dynamic quaternion-valued matrix inversion, and an application to mobile manipulator control is provided to indicate the practical application value of the QVZNN models.

Zeroing Neural Network for Time-Varying Linear Equations With Application to Dynamic Positioning

In this article, considering the effectiveness and efficiency in solving time-varying problems, a new zeroing neural network (ZNN) is proposed to solve time-varying linear equations with column full rank coefficient matrix. In addition, two novel nonlinear activation functions are developed to enhance the comprehensive performance of the ZNN model. It is demonstrated through theoretical analysis and numerical experiments that the nonlinear activated ZNN model has better noise immunity, and faster prescribed-time convergence speed. Finally, the ZNN method is successfully applied to 2-D and 3-D dynamic positioning, with lower positioning error than the traditional pseudoinverse method.