Nonlinear Activation Function Research Articles

Making depthwise convolution SR-friendly via kernel attention injection

Despite the remarkable results achieved by deep convolutional neural networks (CNNs) in single image super-resolution (SISR), their computational cost increase exponentially as CNN models get deeper and wider. Consequently, depthwise separable convolutions (DSConvs) have emerged as the fundamental building basis for various contemporary lightweight network architectures. However, depthwise convolution is sub-optimal for restoring missing high-frequency details. In this paper, we commence with the vanilla depthwise convolution and progressively develop it to alleviate the above problem. Specifically, we introduce an effective method, kernel attention injection, to integrate intra-kernel correlation into the depthwise convolution layer by incorporating the learned kernel attention into the depthwise kernel. We find that the determinant is capable of capturing the correlation between diagonal elements of the depthwise kernel by summing the products of the diagonals. Therefore, we utilize the determinant as the kernel pooling method. Furthermore, we propose to remove the non-linear activation function behind the depthwise convolution (i.e., linear depthwise convolution) to preserve informative features for reconstructing faithful high-resolution (HR) images. Built on this recipe, we introduce kernel-attentive linear depthwise separable convolutions (KDSConvs), which exhibits promising super-resolution performance. Our experimental results underline the potential of depthwise convolution in super-resolution tasks by demonstrating that the proposed KDSConvs significantly outperforms DSConvs in terms of both quantitative measurements and visual quality. The code will be released at https://github.com/AwesomeHwang/KALDN.

Prediction model with multi-point relationship fusion via graph convolutional network: A case study on mining-induced surface subsidence.

Accurate prediction of surface subsidence is of significance for analyzing the pattern of mining-induced surface subsidence, and for mining under buildings, railways, and water bodies. To address the problem that the existing prediction models ignore the correlation between subsidence points, resulting in large prediction errors, a Multi-point Relationship Fusion prediction model based on Graph Convolutional Networks (MRF-GCN) for mining-induced subsidence was proposed. Taking the surface subsidence in 82/83 mining area of Yuandian No. 2 Mine in Anhui Province in eastern China as an example, the surface deformation data obtained from 250 InSAR images captured by Sentinel-1A satellite from 2018 to 2022, combined with GNSS observation data, were used for modeling. The deformation pattern of each single observation point was obtained by feeding their deformation observation data into the LSTM encoder, after that, the relationship graph was created based on the correlation between points in the observation network and MRF-GCN was established. Then the prediction results came out through a nonlinear activation function of neural network. The research shows that the R2R2 value of MRF-GCN model was 0.865 0, much larger than that of Long-Short Term Memory (LSTM) and other conventional models, while mean square error (MSE) of MRF-GCN model was 1.59 899, much smaller than that of LSTM and other conventional models. Therefore, the MRF-GCN model has better prediction accuracy than other models and can be applied to predicting surface subsidence in large areas.

A Scalable Small-Footprint Time-Space-Pipelined Architecture for Reservoir Computing

Reservoir computing (RC) is a lightweight machine learning algorithm for edge applications, which features a lower computation workload and one-time training process compared with the recurrent neural network (RNN) and Transformer. The existing RC accelerators still suffer considerable hardware overhead, which cannot satisfy the scalability on various edge FPGA/ASIC resource limits. This brief proposes a scalable small-footprint time-space-pipelined architecture for the cycle RC paradigm. The RC workload can be distributed onto a configurable number of processing elements (PE) under scalable resource limits. A time-space-pipelined PE is designed to efficiently execute the cycle RC computation, with the power-of-2 piecewise linearization circuits for non-linear activation function. Experimental results show that the proposed architecture can match a large variety of FPGA resource limits and ASIC power/area limits, with 12.6× energy efficiency improvement compared with the state-of-the-art RC accelerator.

Robustness Verification of Swish Neural Networks Embedded in Autonomous Driving Systems

With the applications of deep learning in safety-critical domains such as autonomous driving systems gaining ground, it demands rigorous verification to guarantee the safety and reliability of corresponding systems. As the intelligent component in such systems, neural networks (NNs) must be robust in that their outputs are not affected by minor perturbation to inputs. Many research studies have shown that formal methods are effective ways to the robustness verification of NNs. However, most of the existing approaches are focused on NNs that contain monotonic activation functions, such as ReLU, Tanh, and Sigmoid. In this work, we propose an approach to verify the robustness of NNs with the nonmonotonic activation function called Swish. Such networks have been proved to have a better performance on image classification than other NNs. In our approach, we turn the robustness verification problem into a constraint-solving problem using the linear approximation technique. We first model the affine function of an NN into a linear constraint model. Then, for nonlinear activation functions, we leverage an efficient approximation strategy to linearly approximate them. Finally, we utilize the constraint solver gurobi to solve the model, which reveals that the model satisfies the robustness property. We develop a prototype tool and evaluate it with open-sourced NNs. Experimental results showed the effectiveness and efficiency of our approach.

SSCK-Net: Spine segmentation in MRI based on cross attention and key-points recognition-assisted learner

Multiple spine segmentation of vertebrae and intervertebral discs for magnetic resonance images (MRI) plays a significant role in various spinal diseases diagnosis and treatments of spine disorder. However, the inherent complexity and characteristics of the spine pose challenges in balancing the inter-class similarity and intra-class variety of spine, improving generalization ability, learning rate, and accuracy. To address the above issues, a spine segmentation in MRI based on cross attention and key-points recognition-assisted learner (SSCK-Net) is proposed. Firstly, a multi-channel cross attention (MCCA) mechanism is put forward to generate a comprehensive description of the spine by fusing complementary inter-class feature and intra-class feature. Secondly, a key-points recognition-assisted learner (KRAL) is designed, which includes mixed-supervision recognition-assisted label fusion (RALF), to reduce dependence on a single dataset and improve the generalization ability of the network. Subsequently, adaptive transformer (AT) is presented to perform parallel computation through selective kernel fusion, nonlinear activation function with simple inverse and rescaling layer normalization, which reduces training time and prevents performance degradation due to long-term dependence. Finally, a feature enhances unit (FEU) is proposed to fuse the key point features and the segmented features while constraining the key-point features to the fused features again by the refine unit. Our experiments on T2-weighted volumetric MRI show that SSCK-Net achieves impressive performance with a mean Dice similarity coefficient (DSC) 96.12% for 5 vertebral bodies and 95.07% for 5 intervertebral discs on public datasets, outperforming state-of-the-art techniques.

Using normalized echo state network to detect abnormal ECG patterns

AbstractCardiovascular disease (CVD) stands as a prominent contributor to human mortality. Electrocardiogram (ECG) represents a widely adopted noninvasive method employed by clinicians to detect and diagnose CVDs. Nonetheless, conventional ECG‐based detection approaches for cardiac disorders tend to be time‐consuming and inefficient, necessitating the need for more effective solutions. Recent studies have highlighted the effectiveness of the echo state network (ESN) in detecting abnormal ECG patterns. However, traditional ESN models often face challenges such as unstable training and convergence difficulties due to variations in the range of reservoir state values. To address this issue, this study introduces a novel approach called the normalized echo state network (NESN). The NESN method normalizes the states of all neurons within the reservoir before applying the nonlinear activation function. In our study, we conducted performance evaluations of the proposed model using the MIT‐BIH arrhythmia database. We performed a synergistical analysis to investigate the impact of reservoir parameters on the network performance. The experimental results demonstrated promising outcomes, with an accuracy of 99.1% and an F1‐score of 96.4%. Specifically, for detecting abnormal ECG patterns, our model achieved a sensitivity of 90.2%, a positive predictive value of 96.6%, and a specificity of 99.8%. These results highlight the superior performance of our classifier compared to most traditional mainstream heartbeat detection methods and ring topology ESN model.

Low-threshold all-optical nonlinear activation function based on injection locking in distributed feedback laser diodes.

We experimentally demonstrate an all-optical nonlinear activation unit based on the injection-locking effect of distributed feedback laser diodes (DFB-LDs). The nonlinear carrier dynamics in the unit generates a low-threshold nonlinear activation function with optimized operating conditions. The unit can operate at a low threshold of -15.86 dBm and a high speed of 1 GHz, making it competitive among existing optical nonlinear activation approaches. We apply the unit to a neural network task of solving the second-order ordinary differential equation. The fitting error is as low as 0.0034, verifying the feasibility of our optical nonlinear activation approach. Given that the large-scale fan-out of optical neural networks (ONNs) will significantly reduce the optical power in one channel, our low-threshold scheme is suitable for the development of high-throughput ONNs.

Fixed-time convergence integral-enhanced ZNN for calculating complex-valued flow matrix Drazin inverse

The complex-valued flow matrix Drazin inverse has recently attracted considerable interest from researchers due to its great academic value. In this paper, a fixed-time convergence integral-enhanced zeroing neural network (FTCIEZNN) model is proposed and investigated for calculating the Drazin inverse of complex-valued flow matrix. Since the FTCIEZNN model possesses fixed-time convergence, its upper limit of convergence time is irrelevant to initial conditions and can be adjusted by specified system parameters. Meanwhile, by adopting the newly designed reformed nonlinear activation function (RNAF) and variable parameters, the FTCIEZNN model converges rapidly in a relatively fast fixed-time and its robustness is dramatically strengthened. In addition, the upper limit of the convergence time in the absence of noise and the upper limit of the steady-state error in the presence of time-varying bounded noise are given by a scrupulous mathematical logic calculation. Furthermore, the outcomes of the numerical simulations demonstrate that the FTCIEZNN model outshines existing zeroing neural network models in calculating complex-valued flow matrix Drazin inverse. Finally, an application based on the FTCIEZNN model in image encryption fully illustrates the practical value of the FCIEZNN model.

ReAFM: A Reconfigurable Nonlinear Activation Function Module for Neural Networks

Deep neural networks (DNNs) with various nonlinear activation functions (NAFs) have achieved unprecedented successes, sparking interest in efficient DNN hardware implementation. However, most existing NAF implementations focus on one type of functions with dedicated architectures, which is not suited for supporting versatile DNN accelerators. In this brief, based on a proposed reciprocal approximation optimization (RAO) method, an efficient reconfigurable nonlinear activation function module (ReAFM) is devised to implement various NAFs. The computational logic and dataflow of certain NAFs are merged and reused to minimize hardware consumption by leveraging the correlations among different NAFs. In addition, a precision adjustable exponential unit (PAEU) is developed to obtain a good tradeoff between the approximation accuracy and hardware cost. Compared to the prior art, the experimental results demonstrate that the proposed ReAFM can support many more NAF types with comparable or even better performance. Furthermore, evaluation results on some prevalent neural networks show that the proposed approximation method causes negligible accuracy loss (< 0.1%).

A method to improve the computational performance of nonlinear all—optical diffractive deep neural network model

To further improve the computational performance of the diffractive deep neural network (D2NN) model, we use the ReLU function to limit the phase parameters, which effectively solves the problem of vanishing gradient that occurs in the mitigation model. We add various commonly used nonlinear activation functions to the hidden layer of the model and establish the ReLU phase-limit nonlinear diffractive deep neural network (ReLU phase-limit N-D2NN) model. We evaluate the model by comparing the performance of various nonlinear activation functions, where confusion matrix and accuracy are used as evaluation methods. The numerical simulation results show that the model achieves better classification performance on the MNIST and Fashion-MNIST datasets, respectively. In particular, the highest classification performance is obtained by the ReLU phase-limit N-D2NN model, in which the hidden layer uses PReLU, with 98.38% and 90.14%, respectively. This paper provides a theoretical basis for applying the nonlinear D2NN systems in natural scenes.

A High Performance Reconfigurable Hardware Architecture for Lightweight Convolutional Neural Network

Since the lightweight convolutional neural network EfficientNet was proposed by Google in 2019, the series of models have quickly become very popular due to their superior performance with a small number of parameters. However, the existing convolutional neural network hardware accelerators for EfficientNet still have much room to improve the performance of the depthwise convolution, squeeze-and-excitation module and nonlinear activation functions. In this paper, we first design a reconfigurable register array and computational kernel to accelerate the depthwise convolution. Next, we propose a vector unit to implement the nonlinear activation functions and the scale operation. An exchangeable-sequence dual-computational kernel architecture is proposed to improve the performance and the utilization. In addition, the memory architectures are designed to complete the hardware accelerator for the above computing architecture. Finally, in order to evaluate the performance of the hardware accelerator, the accelerator is implemented based on Xilinx XCVU37P. The results show that the proposed accelerator can work at the main system clock frequency of 300 MHz with the DSP kernel at 600 MHz. The performance of EfficientNet-B3 in our architecture can reach 69.50 FPS and 255.22 GOPS. Compared with the latest EfficientNet-B3 accelerator, which uses the same FPGA development board, the accelerator proposed in this paper can achieve a 1.28-fold improvement of single-core performance and 1.38-fold improvement of performance of each DSP.

Formula omitted] master–slave synchronization for delayed impulsive implicit hybrid neural networks based on memory-state feedback control

This paper investigates the H∞ master–slave synchronization problem for delayed impulsive implicit hybrid neural networks based on memory-state feedback control. By developing a more holistic stochastic impulse-time-dependent Lyapunov–Krasovskii functional and dealing with the nonlinear neuron activation function, the stochastic admissibility and prescribed H∞ performance index for the synchronization error closed-loop system are achieved. In addition, the desired mode-dependent memory-state feedback synchronization controller is acquired in the form of linear matrix inequalities. The free-weighting matrix technique is adopted to remove the inherent limitation of time-varying delay derivative for the implicit delayed systems, and the derivative of time-varying delay is relaxed enough to be greater than 1. The simulation of genetic regulatory network in bio-economic system is given to verify validity of the derived results.

Validating neural networks for spectroscopic classification on a universal synthetic dataset

To aid the development of machine learning models for automated spectroscopic data classification, we created a universal synthetic dataset for the validation of their performance. The dataset mimics the characteristic appearance of experimental measurements from techniques such as X-ray diffraction, nuclear magnetic resonance, and Raman spectroscopy among others. We applied eight neural network architectures to classify artificial spectra, evaluating their ability to handle common experimental artifacts. While all models achieved over 98% accuracy on the synthetic dataset, misclassifications occurred when spectra had overlapping peaks or intensities. We found that non-linear activation functions, specifically ReLU in the fully-connected layers, were crucial for distinguishing between these classes, while adding more sophisticated components, such as residual blocks or normalization layers, provided no performance benefit. Based on these findings, we summarize key design principles for neural networks in spectroscopic data classification and publicly share all scripts used in this study.

Front Transport Reduction for Complex Moving Fronts

This work addresses model order reduction for complex moving fronts, which are transported by advection or through a reaction–diffusion process. Such systems are especially challenging for model order reduction since the transport cannot be captured by linear reduction methods. Moreover, topological changes, such as splitting or merging of fronts pose difficulties for many nonlinear reduction methods and the small non-vanishing support of the underlying partial differential equations dynamics makes most nonlinear hyper-reduction methods infeasible. We propose a new decomposition method together with a hyper-reduction scheme that addresses these shortcomings. The decomposition uses a level-set function to parameterize the transport and a nonlinear activation function that captures the structure of the front. This approach is similar to autoencoder artificial neural networks, but additionally provides insights into the system, which can be used for efficient reduced order models. In addition to the presented decomposition method, we outline a tailored hyper-reduction method that is based on the reduced integration domain method. The capability of the approach is illustrated by various numerical examples in one and two spatial dimensions, including an advection–reaction–diffusion system with a Kolmogorov–Petrovsky–Piskunov reaction term and real life application to a two-dimensional Bunsen flame.

Nonlinear function activated GNN versus ZNN for online solution of general linear matrix equations

In this work, the gradient-based design method is generalized to establish a finite-/fixed-time GNN (gradient-based neural network) to obtain an online solution of the general linear matrix equation (LME): P1X(t)Q1+P2X(t)Q2=G. Two nonlinear activation functions (AFs), which are broadly investigated in ZNN (zeroing neural network), are introduced and put to use for the construction of the GNN model. Theoretical analyses and comparisons are given to illustrate their impacts on the GNN and the corresponding ZNN models in terms of convergence. It is shown that both the GNN model and the ZNN model exhibit the same convergence property (i.e., finite-/fixed-time convergence) under the action of the same AF. A tighter estimate of the upper bound of fixed convergence time is also obtained for the ZNN model. Comparative verification is conducted to confirm the validation of theoretical analysis.

A Generalized Deep Learning Clustering Algorithm Based on Non-Negative Matrix Factorization

Clustering is a popular research topic in the field of data mining, in which the clustering method based on non-negative matrix factorization (NMF) has been widely employed. However, in the update process of NMF, there is no learning rate to guide the update as well as the update depends on the data itself, which leads to slow convergence and low clustering accuracy. To solve these problems, a generalized deep learning clustering (GDLC) algorithm based on NMF is proposed in this article. Firstly, a nonlinear constrained NMF (NNMF) algorithm is constructed to achieve sequential updates of the elements in the matrix guided by the learning rate. Then, the gradient values corresponding to the element update are transformed into generalized weights and generalized biases, by inputting the elements as well as their corresponding generalized weights and generalized biases into the nonlinear activation function to construct the GDLC algorithm. In addition, for improving the understanding of the GDLC algorithm, its detailed inference procedure and algorithm design are provided. Finally, the experimental results on eight datasets show that the GDLC algorithm has efficient performance.

A Segmented Variable-Parameter ZNN for Dynamic Quadratic Minimization With Improved Convergence and Robustness.

As a category of the recurrent neural network (RNN), zeroing neural network (ZNN) can effectively handle time-variant optimization issues. Compared with the fixed-parameter ZNN that needs to be adjusted frequently to achieve good performance, the conventional variable-parameter ZNN (VPZNN) does not require frequent adjustment, but its variable parameter will tend to infinity as time grows. Besides, the existing noise-tolerant ZNN model is not good enough to deal with time-varying noise. Therefore, a new-type segmented VPZNN (SVPZNN) for handling the dynamic quadratic minimization issue (DQMI) is presented in this work. Unlike the previous ZNNs, the SVPZNN includes an integral term and a nonlinear activation function, in addition to two specially constructed time-varying piecewise parameters. This structure keeps the time-varying parameters stable and makes the model have strong noise tolerance capability. Besides, theoretical analysis on SVPZNN is proposed to determine the upper bound of convergence time in the absence or presence of noise interference. Numerical simulations verify that SVPZNN has shorter convergence time and better robustness than existing ZNN models when handling DQMI.

Applying and dissecting LSTM neural networks and regularized learning for dynamic inferential modeling

Deep learning models such as the long short-term memory (LSTM) network have been applied for dynamic inferential modeling. However, many studies apply LSTM as a black-box approach without examining the necessity and usefulness of the internal LSTM gates for inferential modeling. In this paper, we use LSTM as a state space realization and compare it with linear state space modeling and statistical learning methods, including N4SID, partial least squares, the Lasso, and support vector regression. Two case studies on an industrial 660 MW boiler and a debutanizer column process indicate that LSTM underperforms all other methods. LSTM is shown to be capable of outperforming linear methods for a simulated reactor process with severely excited nonlinearity in the data. By dissecting the sub-components of a simple LSTM model, the effectiveness of the LSTM gates and nonlinear activation functions is scrutinized.

B-LNN: Inference-time linear model for secure neural network inference

Machine Learning as a Service (MLaaS) provides clients with well-trained neural networks for predicting private data. Conventional prediction processes of MLaaS require clients to send sensitive inputs to the server, or proprietary models must be stored on the client-side device. The former reveals client privacy, while the latter harms the interests of model providers. Existing works on privacy-preserving MLaaS introduce cryptographic primitives to allow two parties to perform neural network inference without revealing either party's data. However, nonlinear activation functions bring high computational overhead and response delays to the inference process of these schemes.In this paper, we analyze the mechanism by which activation functions enhance model expressivity, and design an activation function S-cos that is friendly to secure neural network inference. Our proposed S-cos can be re-parameterized into a linear layer during the inference phase. Further, we propose an inference-time linear model called Beyond Linear Neural Network (B-LNN) equipped with S-cos, which exhibits promising performance on several benchmark datasets.

Low power resource efficient CORDIC enabled neuron architecture using 45 nm CMOS technology

In this paper problem is addressed in the current study by providing resource-efficient CORDIC enabled neuron architecture (RECON) that can be customized to calculate both block of multiply-accumulate (MAC) unit and non-linear activation function (AF) operations. The CORDIC-enabled architecture implements MAC and AF operations using linear and trigonometric relationships, respectively. All physical parameters of the proposed design are built and verified using Cadence Virtuoso @ 45 nm technology. As compared to the conventional art of MAC design, our implementation of the signed fixed-point 8-bit MAC results in a 70% reduction in area, latency, and power product (ALP), as well as a 45 percent reduction in area, a 28% reduction in power dissipation, and a 20% reduction in latency. Both the process adjustments and the device mismatch are subjected to Monte-Carlo simulations. The proposed design is based on resource-intensive components such as multipliers and non-linear Activation Functions, modern hardware implementations of DNNs require more space (AFs). To access input features, weights, and biases, and improved on-chip quantized log2 based memory addressing approach is implemented. The bandwidth needs of DNNs' external memory are therefore decreased and dynamically adjusted. The Taylor series is also used to extract intensive higher speed and resource-efficient memory components for the various activation functions, and its order expansion has been altered for increased test accuracy. The MNIST dataset is used in earlier studies.