Nonlinear Activation Function Research Articles

Applications of Catalan Neural Network (CaNN) model

In this paper, we construct Catalan neural network (CaNN) model and examine the nonlinear solvers of the ordinary nonlinear differential equation using the CaNN model with linear and nonlinear activation functions. Using a single-layer functional linking neural network (FLANN) architecture, we extend input models using the first five Catalan polynomials, update network parameters that we initially randomly select, and use error backpropagation algorithms. The CaNN trial solution for solving nonlinear differential equations is in excellent agreement with the analytical solution. Also, the validity of the CaNN method is investigated by treating second-order differential equations of Lane–Emden type as boundary and initial value problems.

Field-free spin–orbit torque magnetization switching in Pt/CoTb devices grown on flexible substrates for neuromorphic computing

Flexible spintronic devices based on spin–orbit torque (SOT)-induced perpendicular magnetization switching (PMS) have attracted increasing attention due to their high storage intensity and good programming capability. However, to achieve deterministic PMS, an in-plane auxiliary magnetic field is required, which greatly limits its application. Here, we show that “robust” magnetic field-free SOT-driven PMS is realized in the oblique sputtered Pt/CoTb multilayers grown on a flexible polyimide substrate. “Robust” means the magnetic field-free SOT switching is highly repeatable and stable after 100 bending cycles under various bending conditions. Additionally, the fabricated flexible multilayers exhibit nearly linear and nonvolatile multistate plasticity as synapses and a nonlinear sigmoid activation function when acting as neurons. We construct a fully connected neural network for handwritten digit recognition, achieving an over 96.27% recognition rate. Our findings may spur further investigations on the SOT-based flexible spintronic devices for wearable artificial intelligence applications.

Cascadable optical nonlinear activation function based on Ge-Si.

To augment the capabilities of optical computing, specialized nonlinear devices as optical activation functions are crucial for enhancing the complexity of optical neural networks. However, existing optical nonlinear activation function devices often encounter challenges in preparation, compatibility, and multi-layer cascading. Here, we propose a cascadable optical nonlinear activation function architecture based on Ge-Si structured devices. Leveraging dual-source modulation, this architecture achieves cascading and wavelength switching by compensating for loss. Experimental comparisons with traditional Ge-Si devices validate the cascading capability of the new architecture. We first verified the versatility of this activation function in a MNIST task, and then in a multi-layer optical dense neural network designed for complex gesture recognition classification, the proposed architecture improves accuracy by an average of 23% compared to a linear network and 15% compared to a network with a traditional activation function architecture. With its advantages of cascadability and high compatibility, this work underscores the potential of all-optical activation functions for large-scale optical neural network scaling and complex task handling.

Ultra-broadband all-optical nonlinear activation function enabled by MoTe2/optical waveguide integrated devices

All-optical nonlinear activation functions (NAFs) are crucial for enabling rapid optical neural networks (ONNs). As linear matrix computation advances in integrated ONNs, on-chip all-optical NAFs face challenges such as limited integration, high latency, substantial power consumption, and a high activation threshold. In this work, we develop an integrated nonlinear optical activator based on the butt-coupling integration of two-dimensional (2D) MoTe2 and optical waveguides (OWGs). The activator exhibits an ultra-broadband response from visible to near-infrared wavelength, a low activation threshold of 0.94 μW, a small device size (~50 µm2), an ultra-fast response rate (2.08 THz), and high-density integration. The excellent nonlinear effects and broadband response of 2D materials have been utilized to create all-optical NAFs. These activators were applied to simulate MNIST handwritten digit recognition, achieving an accuracy of 97.6%. The results underscore the potential application of this approach in ONNs. Moreover, the classification of more intricate CIFAR-10 images demonstrated a generalizable accuracy of 94.6%. The present nonlinear activator promises a general platform for three-dimensional (3D) ultra-broadband ONNs with dense integration and low activation thresholds by integrating a variety of strong nonlinear optical (NLO) materials (e.g., 2D materials) and OWGs in glass.

Deep Information Retention Network‐Enabled Data Modeling for Key Quality Indicator Prediction in the Chemical Industry

ABSTRACTDeep learning has attracted widespread attention in data modeling and key quality indicator prediction in the chemical industry. However, traditional deep learning networks usually distort the original data distribution due to the superposition effect of multiple layers of nonlinear activation functions. In this case, multivariate statistical learning techniques present an avenue to reveal the intrinsic relationship of the data by combining the linear trends between input and predictor variables. To comprehensively capture data features from multiple perspectives, this study proposes a deep learning‐based data modeling network called the information retention unit (IRU). This network combines intrinsic attributes to partial least squares (PLS) and autoencoder (AE) modalities, thus engendering an adaptive response to the complex linear and nonlinear data features. Furthermore, multiple IRUs can be stacked to construct a deep information retention network (DIRN), which enhances the robust extraction of deep data features. Finally, the effectiveness of the proposed network is validated through its prediction application on a dataset obtained from a real‐world chemical industrial process. This method combines multivariate statistical learning techniques based on deep learning, providing an innovative and practical solution for data analysis and prediction in the chemical industry.

Fractional-order spike-timing-dependent gradient descent for multi-layer spiking neural networks

Accumulated detailed knowledge about the neuronal activities in human brains has brought more attention to bio-inspired spiking neural networks (SNNs). In contrast to non-spiking deep neural networks (DNNs), SNNs can encode and transmit spatiotemporal information more efficiently by exploiting biologically realistic and low-power event-driven neuromorphic architectures. However, the supervised learning of SNNs still remains a challenge because the spike-timing-dependent plasticity (STDP) of connected spiking neurons is difficult to implement and interpret in existing backpropagation learning schemes. This paper proposes a fractional-order spike-timing-dependent gradient descent (FO-STDGD) learning model by considering a derived nonlinear activation function that describes the relationship between the quasi-instantaneous firing rate and the temporal membrane potentials of nonleaky integrate-and-fire neurons. The training strategy can be generalized to any fractional orders between 0 and 2 since the FO-STDGD incorporates the fractional gradient descent method into the calculation of spike-timing-dependent loss gradients. The proposed FO-STDGD model is tested on the MNIST and DVS128 Gesture datasets and its accuracy under different network structure and fractional orders is analyzed. It can be found that the classification accuracy increases as the fractional order increases, and specifically, the case of fractional order 1.9 improves by 155 % relative to the case of fractional order 1 (traditional gradient descent). In addition, our scheme demonstrates the state-of-the-art computational efficacy for the same SNN structure and training epochs.

All-optical reconfigurable optical neural network chip based on wavelength division multiplexing

Optical computing has become an important way to achieve low power consumption and high computation speed. Optical neural network (ONN) is one of the key branches of optical computing due to its wide range of applications. However, the integrated ONN schemes proposed in previous works have some disadvantages, such as fixed network structure, complex matrix-vector multiplication (MVM) unit, and few all-optical nonlinear activation function (NAF) methods. Moreover, for the most compact MVM schemes based on wavelength division multiplexing (WDM), it is infeasible to employ intrinsic nonlinear effects to implement NAF, which brings frequent O-E-O conversion in ONN chips. Besides, it is also hard to realize a reconfigurable ONN with coherent MVMs, while it is much easier to implement in WDM schemes. We propose for the first time an all-optical silicon-based ONN chip based on WDM by adopting a new adjustment mechanism: optical gradient force (OGF). The proposed scheme is reconfigurable with tunable layers, variable neurons per layer, and adjustable NAF curves. In the task of classification of the MNIST dataset, our chip can realize an accuracy of 85.13% with 4 full-connected layers and only 50 neurons in total. In addition, we analyze the influence of the OGF-based NAF under fabrication errors and propose a calibration method. Compared to the previous works, our scheme has the two-fold advantages of compactness and reconfiguration, and it paves the way for the all-optical ONN based on WDM and opens the path to unblocking the bottleneck of integrated large-dimension ONNs.

DELM: Deep Ensemble Learning Model for Anomaly Detection in Malicious Network Traffic-based Adaptive Feature Aggregation and Network Optimization

With the rapid advancements in internet technology, the complexity and sophistication of network traffic attacks are increasing, making it challenging for traditional anomaly detection systems to analyze and detect malicious network attacks. The increasing advancedness of cyber threats calls for innovative approaches to identify malicious patterns within network traffic precisely. The primary issue lies in the fact that these approaches do not focus on the essential adaptive features of network traffic. We proposed an effective anomaly detection system for malicious network traffic attacks called the Deep Ensemble Learning Model (DELM). We leverage the structure of the Feedforward Deep Neural Network (FDNN), and Deep Belief Network (DBN), incorporating multiple hidden layers with non-linear activation functions. Integrating Adaptive Feature Aggregation (AFA) with the FDNN algorithm dynamically adjusts the feature aggregation process based on incoming traffic characteristics to improve adaptability. The Conditional Generative Network was employed to enhance DELM for generating data for minority classes. To improve the model’s accuracy, we applied batch normalization and data augmentation techniques for preprocessing, utilized n-gram, one-hot encoding, and feature aggregation methods for effective feature extraction. This study significantly contributes to network security by enhancing systems for detecting malicious network traffic. With its interpretability and adaptability, our proposed model shows promise in addressing the evolving cyber threat and fortifying critical network infrastructure. The experimental results demonstrate that our model performs with higher stability than the existing state-of-the-art detection approaches, as reflected by its higher accuracy, precision, recall, F1-score, and AUC-ROC.

Optical activation function using a metamaterial waveguide for an all-optical neural network.

In this study, we experimentally demonstrated that the nonlinear optical coefficient of the original Si can be enhanced by incorporating a metamaterial structure into an existing silicon waveguide. The two-photon absorption coefficient enhanced by the metamaterial structure was 424 cm/GW, which is 1.2 × 103 times higher than that of Si. Using this metamaterial waveguide-based nonlinear optical activation function, we achieved a high inference accuracy of 98.36% in the handwritten character recognition task, comparable to that obtained with the ReLU function as the activation function. Therefore, our approach can contribute to the realization of more power-efficient and compact all-optical neural networks.

Predicting Binge Eating Disorder Using Machine Learning Methods

Eating disorders are enduring conditions characterized by elevated rates of mortality and morbidity, presenting a serious threat to life. Among these disorders, binge eating disorder is the most prevalent. Therefore, it is an important health problem that often results in obesity worldwide. This study was conducted to evaluate the eating attitudes and behaviors of university students and predict binge eating disorder using machine learning methods. The study was carried out on 306 individuals (117 males, 189 females). Individuals' personal characteristics were questioned with the questionnaire form. The Bulimic Investigatory Test Edinburgh (BITE) test was used to determine whether individuals taking part in the study had binge eating disorder. In this study, in which binge eating disorder was classified, different artificial neural network models were created by changing the basic parameters, and the optimum model was assessed accordingly. Among the models created with different layers and activation functions, the optimum results were obtained using the number of fully connected layers as 2, first and second layers' sizes as 10, and ReLU, a non-linear activation function, in the Bilayered Neural Network structure. This study is the first trial in which binge eating disorder is predicted using machine learning methods, and we believe that machine learning is an important tool to help researchers and clinicians diagnose, prevent, and treat eating disorders at an early stage.

S2VSNet: Single stage V-shaped network for image deraining & dehazing

Producing high-quality, noise-free images from noisy or hazy inputs relies on essential tasks such as single image deraining and dehazing. In many advanced multi-stage networks, there is often an imbalance in contextual information, leading to increased complexity. To address these challenges, we propose a simplified method inspired by a U-Net structure, resulting in the “Single-Stage V-Shaped Network” (S2VSNet), capable of handling both deraining and dehazing tasks. A key innovation in our approach is the introduction of a Feature Fusion Module (FFM), which facilitates the sharing of information across multiple scales and hierarchical layers within the encoder-decoder structure. As the network progresses towards deeper layers, the FFM gradually integrates insights from higher levels, ensuring that spatial details are preserved while contextual feature maps are balanced. This integration enhances the image processing capability, producing noise-free, high-quality outputs. To maintain efficiency and reduce system complexity, we replaced or removed several non-essential non-linear activation functions, opting instead for simple multiplication operations. Additionally, we introduced a “Multi-Head Attention Integrated Module” (MHAIM) as an intermediary layer between encoder-decoder levels. This module addresses the limited receptive fields of traditional Convolutional Neural Networks (CNNs), allowing for the capture of more comprehensive feature-map information. Our focus on deraining and dehazing led to extensive experiments on a wide range of synthetic and real-world datasets. To further validate the robustness of our network, we implemented S2VSNet on a low-end edge device, achieving deraining in 2.46 seconds.

Asymptotic Synchronization for Caputo Fractional-Order Time-Delayed Cellar Neural Networks with Multiple Fuzzy Operators and Partial Uncertainties via Mixed Impulsive Feedback Control

To construct Caputo fractional-order time-delayed cellar neural networks (FOTDCNNs) that characterize real environments, this article introduces partial uncertainties, fuzzy operators, and nonlinear activation functions into the network models. Specifically, both the fuzzy AND operator and the fuzzy OR operator are contemplated in the master–slave systems. In response to the properties of the considered cellar neural networks (NNs), this article designs a new class of mixed control protocols that utilize both the error feedback information of systems and the sampling information of impulse moments to achieve network synchronization tasks. This approach overcomes the interference of time delays and uncertainties on network stability. By integrating the fractional-order comparison principle, fractional-order stability theory, and hybrid control schemes, readily verifiable asymptotic synchronization conditions for the studied fuzzy cellar NNs are established, and the range of system parameters is determined. Unlike previous results, the impulse gain spectrum considered in this study is no longer confined to a local interval (−2,0) and can be extended to almost the entire real number domain. This spectrum extension relaxes the synchronization conditions, ensuring a broader applicability of the proposed control schemes.

Constructed encoded data based coded distributed DNN training for edge computing scenario

Deep learning in unmanned aerial vehicle (UAV) deployment encounters two problems: 1, One UAV may fail to store and execute large Deep Neural Network (DNN) model. 2, One UAV fails to accomplish real time services. One possible solution is the group of UAVs (nodes) collectively generate swarm intelligence in the form of edge computing scenario, namely in the distributed computing (DC) mode with clever arrangement, say Coded Distributed Computing (CDC). In CDC systems, the redundant computation introduced by linear coding can compensate stragglers. However, since linear property cannot pass the nonlinear activation function in Deep Neural Network (DNN) training, coding/decoding for CDC need to be applied layer by layer, which slows down the training. To avoid layer-by-layer coding/decoding, we propose a novel DNN training scheme based on constructing encoded data. This construction process lies before the training process (can be done before training without any impact on training efficiency). Based on both the original data and the newly constructed encoded data, the training phase can take advantage of the (n,k) property (Wait for the first k returned data) and hence improve the training speed. The training process does not require encoding/decoding operations, and hence significantly improves the training speed. Experimental results show that the training scheme based on constructed encoded data can achieve prediction accuracy approximating that of the centralized one and significantly reduce the latency compared to the layer-by-layer linear encoding and decoding scheme.

A photonic integrated chip for distinguishing the optical wave packets based on a neural-network

In this work, we use an optical-neural network to recognize a series of optical triangular and square wave packets. This chip is designed based on the reservoir computing, consisting of the directional couplers, phase shifters, Mach-Zehnder modulators, spiral waveguide and grating couplers. The non-linear activation function is obtained by the directional couplers. The optimization of the optical reservoir computing system is performed by tuning the voltage on the phase shifters to obtain the largest extinction ratio of the output signals for different input wave packets. The photonic integrated chip is fabricated on silicon-on-insulator and is characterized at an operation frequency of 3 GHz. The results demonstrate that the photonic integrated chip can efficiently distinguish between optical triangular and square wave packets.

Enhanced multidimensional neo-fuzzy classification system and its learning for the video classification task

This research introduces an innovative hybrid neo-fuzzy system for video classification, integrating multidimensional neo-fuzzy components with adjustable synaptic weights and Gaussian membership functions. By combining extended neo-fuzzy neurons (ENFN) and neo-fuzzy units (NFU) with nonlinear activation functions and incorporating extended nonlinear synapses (ENS), the system leverages the neuro-fuzzy Takagi-Sugeno-Kang inference system to enhance traditional models' approximating capabilities. Video classification is complex due to high data volume and variability, especially with moving objects and varying video quality. Traditional models struggle with real-time processing and maintaining accuracy, necessitating advanced techniques for robust performance. The goal is to develop and optimize a hybrid neo-fuzzy system for real-time video stream classification, maintaining high accuracy. Computational experiments demonstrated its robustness, achieving high precision and recall. The proposed optimization algorithm, using cross-entropy learning with one-hot encoding and adaptive δ-rule adjustments, improved learning speed and accuracy. The novelty lies in developing a hybrid neo-fuzzy system with advanced components and a unique optimization algorithm, ensuring robustness and efficiency in complex video classification tasks.

An emergent attractor network in a passive resistive switching circuit

Resistive memory devices feature drastic conductance change and fast switching dynamics. Particularly, nonvolatile bipolar switching events (set and reset) can be regarded as a unique nonlinear activation function characteristic of a hysteretic loop. Upon simultaneous activation of multiple rows in a crosspoint array, state change of one device may contribute to the conditional switching of others, suggesting an interactive network existing in the circuit. Here, we prove that a passive resistive switching circuit is essentially an attractor network, where the binary memory devices are artificial neurons while the pairwise voltage differences define an anti-symmetric weight matrix. An energy function is successfully constructed for this network, showing that every switching in the circuit would decrease the energy. Due to the nonvolatile hysteretic function, the energy change for bit flip in this network is thresholded, which is different from the classic Hopfield network. It allows more stable states stored in the circuit, thus representing a highly compact and efficient solution for associative memory. Network dynamics (towards stable states) and their modulations by external voltages have been demonstrated in experiment by 3-neuron and 4-neuron circuits.

Arbitrary sensitive transitions in recurrent neural networks

An Excitable Network Attractor (ENA) is a forward-invariant set in phase space that can be used to explain input-driven behaviour of Recurrent Neural Networks (RNNs) trained on tasks involving switching between a discrete set of states. An ENA is composed of two or more attractors and excitable connections that allow transitions from one attractor to another under some input perturbation. The smallest such perturbation that makes a connection between two attractors is called the excitability threshold associated with that connection. The excitability threshold provides a measure of sensitivity of the connection to input perturbations. Errors in performance of such trained RNNs can be related to errors in transitions around the associated ENA. Previous work has demonstrated that ENAs of arbitrary sensitivity and structure can be realised in a RNN by suitable choice of connection weights and nonlinear activation function. In this paper we show that ENAs of arbitrary sensitivity and structure can be realised even using a suitable fixed nonlinear activation function, i.e. by suitable choice of weights only. We show that there is a choice of weights such that the probability of erroneous transitions is very small.

Better approximation of sigmoid function for privacy-preserving neural networks

Abstract With the development of artificial intelligence technology, privacy protection has become a prominent research direction. Homomorphic encryption is an important and promising technology in the construction of privacy-preserving neural networks. However, computing nonlinear activation functions remains a challenging task in the context of homomorphic encryption. Considering Sigmoid is one of the most commonly used activation functions, this paper proposes a polynomial approximation method for the Sigmoid function and applies it to the homomorphic encryption scheme of CKKS. We design polynomial approximation functions using Improved Taylor Expansion Segmentation (ITES), Least Squares method (LS), and Random Noise Addition method, respectively. These methods are implemented through the CKKS scheme in the homomorphic encryption library of SEAL. The experimental results show that the ITES achieves high fitting accuracy, reducing the average absolute error by 11.09% and the mean squared error by 38.06% compared to previous fitting methods. Therefore, experiments have identified methods with better fitting results in the CKKS scheme, which can meet the requirements of using homomorphic encryption in deep learning technologies. Finally, our analysis points out that different technologies have unique advantages and adapt to different scenarios, e.g., ITES has higher accuracy, LS has higher performance in scenarios with limited computing resources.

Enhanced marine fish small sample image recognition with RVFL in Faster R-CNN model

Addressing the challenge of insufficient underwater marine life images leading to diminished accuracy in existing target recognition models, this paper introduces Random Vector Functional Link (RVFL), a suitable model for small sample training, to reduce the dependency of traditional visual recognition models on data volume by replacing the Softmax classifier in the Faster Region-Based Convolutional Neural Network (R-CNN) model. The RVFL network assigns random weights to enhance the model's adaptability to diverse data distributions, thereby improving its generalization performance. Furthermore, the RVFL model incorporates multiple nonlinear activation functions, thereby enhancing the neural network's representational capacity for high-dimensional data. The application of the proposed model to the task of marine fish small sample target recognition demonstrates its effectiveness in mitigating the low accuracy issue caused by data scarcity. It significantly improved recognition accuracy and offered new insights into addressing underwater small sample image recognition challenges.

A novel green learning artificial intelligence model for regional electrical load prediction

Load prediction is crucial to unit commitment and scheduling in electricity systems; however, load prediction models consume considerable electricity. Most conventional prediction methods are based on deep learning (DL) models, such as long short-term memory and gated recurrent units. The nonlinear activation functions and backpropagation mechanisms used in these methods result in high computational complexity and correspondingly high energy consumption. This paper proposes a prediction model based on a green learning (GL) framework aimed at reducing energy consumption. The proposed GL model replaces the activation functions conventionally used for feature extraction with a hybrid scheme combining categorical and numerical features, such as seasonal climate features and multi-autoregression terms. The proposed GL model also eliminates the backpropagation methods used to optimize hyperparameters by employing seasonal centroids and the quantile auto-regression forest algorithm for classification/regression. In a case study, the proposed GL model significantly outperformed conventional DL models in terms of energy consumption without compromising accuracy. The energy savings of the proposed GL model are equivalent to the annual electricity consumption of 211 ∼ 565 households, increasing with model penetration. This paper demonstrates the importance of energy-saving models in mitigating the linkage effect of industrial carbon emissions. It also explores policy implications for the implementation of GL models.