Nonlinear Activation Function Research Articles

Echo State Network Based Model Predictive Control for Active Vibration Control of Hybrid Electric Vehicle Powertrains

Reservoir computing refers to a computational framework based on recurrent neural networks that can process time-series data. In an echo state network (ESN), which is a type of reservoir computing framework, the reservoir consists of a recursive network of artificial neurons with nonlinear activation functions. A model predictive control (MPC) technique can determine the control signals by solving the optimization problem of a system using the finite-time domain of each control period. However, real-time optimization cannot be achieved unless the optimal control problem can be solved within the next control period. To overcome this limitation, we propose a new control method based on MPC that explicitly incorporates the predicted disturbance of a time-varying trajectory using ESN to achieve the active vibration control of hybrid electric vehicle (HEV) powertrains. Once the ESN has been trained, the associated MPC explicitly satisfies the constraints over a moving horizon without further training. Instead of completing the real-time optimization within the control period, ESN predicts the future disturbance and applies it to the MPC in the future control period. Based on the predicted future disturbance, the system calculates the optimal control signals required for the future. Thus, real-time control can be realized because the optimal signals are determined before the subsequent control period occurs. The proposed method can be implemented in MPC even if the control period is too short to optimize as long as the disturbance can be reasonably measured and predicted. In this study, the simulation approach was demonstrated using the engine start condition in an HEV powertrain. The importance of this study is that the limitation of MPC relevant to real-time optimization can be relaxed by applying our proposed method.

Recognition of honeycomb lung in CT images based on improved MobileNet model.

The research is to improve the efficiency and accuracy of recognition of honeycomb lung in CT images. Deep learning methods are used to achieve automatic recognition of honeycomb lung in CT images, however, are time consuming and less accurate due to the large amount of structural parameters. In this paper, a novel recognition method based on MobileNetV1 network, multiscale feature fusion method (MSFF), and dilated convolution is explored to deal with honeycomb lung in CT image classification. Firstly, the dilated convolution with different dilated rate is used to extract features to obtain receptive fields of different sizes, and then fuse the features of different scales at multiscale feature fusion block is used to solve the problem of feature loss and incomplete feature extraction. After that, by using linear activation functions (Sigmoid) instead of nonlinear activation functions (ReLu) in the improved deep separable convolution blocks to retain the feature information of each channel. Finally, by reducing the number of improved deep separable blocks to reduce the computation and resource consumption of the model. The experimental results show that improved MobileNet model has the best performance and the potential for recognition of honeycomb lung image datasets, which includes 6318 images. By comparing with 4 traditional models (SVM, RF, decision tree, and KNN) and 11 deep learning models (LeNet-5, AlexNet, VGG-16, GoogleNet, ResNet18, DenseNet121, SENet18, InceptionV3, InceptionV4, Xception, and MobileNetV1), our model achieved the performance with an accuracy of 99.52%, a sensitivity of 99.35%, and a specificity of 99.89%. Improved MobileNet model is designed for the automatic recognition and classification of honeycomb lung in CT images. Through experiments comparative analysis of other models of machine learning and deep learning, it is proved that the proposed improved MobileNet method has the best recognition accuracy with fewer the model parameters and less the calculation time.

A Deep Latent Factor Model for High-Dimensional and Sparse Matrices in Recommender Systems

Recommender systems (RSs) commonly adopt a user-item rating matrix to describe users' preferences on items. With users and items exploding, such a matrix is usually high-dimensional and sparse (HiDS). Recently, the idea of deep learning has been applied to RSs. However, current deep-structured RSs suffer from high computational complexity. Enlightened by the idea of deep forest, this paper proposes a deep latent factor model (DLFM) for building a deep-structured RS on an HiDS matrix efficiently. Its main idea is to construct a deep-structured model by sequentially connecting multiple latent factor (LF) models instead of multilayered neural networks through a nonlinear activation function. Thus, the computational complexity grows linearly with its layer count, which is easy to resolve in practice. The experimental results on four HiDS matrices from industrial RSs demonstrate that when compared with state-of-the-art LF models and deep-structured RSs, DLFM can well balance the prediction accuracy and computational efficiency, which well fits the desire of industrial RSs for fast and right recommendations.

An automatic retinal vessel segmentation approach based on Convolutional Neural Networks

The image segmentation concept is a mechanism of image processing that allows cutting an image into several image sections or parts. Numerous techniques are applied to achieve image segmentation, such as color, pixel value, deep Convolutional Neural Networks, and others. We are interested in Convolutional Neural Network (CNN) based approaches. In this paper, we present an automatic method for blood vessel segmentation in retina images; we propose the use of deep CNNs to achieve this goal. Our proposed model is a multi-encoder decoder architecture composed of two encoder units with convolutional and maxpoling layers and a decoder unit with convolutional and deconvolutional layers; it takes an RGB retina image directly as input. The model, then, is trained to build feature maps from convolution, deconvolution operations and nonlinear activation function in order to reconstruct the feature map and ultimately obtain the output layer as segmentation of retina vessels. Structured retina analysis (STARE) and Digital images of the retina for vessel extraction (DRIVE) are used as datasets for training and evaluation. We implement the model with keras in python. An overview of the model architecture, training with data augmentation techniques, validation and prediction is presented. The results are compared to existing labeled data; and the performance of our model with different metrics like F1 score, accuracy, sensitivity, specificity and precision is evaluated. The results obtained for the different metrics listed above are respectively 0.8321, 0.9716, 0.8214, 0.9860 and 0.8466. In addition, the results of our work are submitted into Drive Challenge in order to get an external evaluation based on dice coefficient calculation with the dice-mean score being 0.8283. The results show that our method is, to a great extent, much better than other CNN-based methods, with a precision of 0.9707 for DRIVE datasets versus 0.9568 and 0.9469 for the others. The results obtained from our proposed model indicate that our method has a promising potential for practical applications like computer analysis of retinal images such as automated screening for precocious detection of diabetic retinopathy. These results are due to the model’s segmentation performance, and, in particular, its enhanced specificity, accuracy and precision.

RETRACTED: Breast cancer diagnosis using multiple activation deep neural network

Deep Neural Network (DNN) stands for multilayered Neural Network (NN) that is capable of progressively learn the more abstract and composite representations of the raw features of the input data received, with no need for any feature engineering. They are advanced NNs having repetitious hidden layers between the initial input and the final layer. The working principle of such a standard deep classifier is based on a hierarchy formed by the composition of linear functions and a defined nonlinear Activation Function (AF). It remains uncertain (not clear) how the DNN classifier can function so well. But it is clear from many studies that within DNN, the AF choice has a notable impact on the kinetics of training and the success of tasks. In the past few years, different AFs have been formulated. The choice of AF is still an area of active study. Hence, in this study, a novel deep Feed forward NN model with four AFs has been proposed for breast cancer classification: hidden layer 1: Swish, hidden layer, 2:-LeakyReLU, hidden layer 3: ReLU, and final output layer: naturally Sigmoidal. The purpose of the study is twofold. Firstly, this study is a step toward a more profound understanding of DNN with layer-wise different AFs. Secondly, research is also aimed to explore better DNN-based systems to build predictive models for breast cancer data with improved accuracy. Therefore, the benchmark UCI dataset WDBC was used for the validation of the framework and evaluated using a ten-fold CV method and various performance indicators. Multiple simulations and outcomes of the experimentations have shown that the proposed solution performs in a better way than the Sigmoid, ReLU, and LeakyReLU and Swish activation DNN in terms of different parameters. This analysis contributes to producing an expert and precise clinical dataset classification method for breast cancer. Furthermore, the model also achieved improved performance compared to many established state-of-the-art algorithms/models.

Logish: A new nonlinear nonmonotonic activation function for convolutional neural network

Activation function is an important component of the convolutional neural network. Recently, nonlinear nonmonotonic activation functions such as Swish and Mish have illustrated good performance in deep learning structures. In this paper, we propose a new nonlinear nonmonotonic activation function called Logish, which can be represented by f(x)=x·ln[1+sigmoid(x)]. Firstly, we take the logarithmic operation to reduce the numerical range of sigmoid(x)+1, then we employ variable x to make the negative output have a strong regularization effect. Furthermore, we evaluate the image classification performance of Logish and its variant f(x)=αx·ln[1+sigmoid(βx)] in simple and complex networks with top–1 accuracy. Experimental results demonstrate that Logish’s variant (α=1,β=10) can achieve 94.8% top-1 accuracy with ResNet–50 network on CIFAR 10 dataset, and can reach 99.24% top-1 accuracy with DenseNet on MNIST dataset and 88.52% top-1 accuracy with SE-Inception-v4 network on SVHN dataset respectively. It is higher than the Sigmoid, Tanh, ReLU, Swish and Mish activation functions in the corresponding dataset. It also verifies the performance and effectiveness of Logish.

A novel hybrid Zhang neural network model for time-varying matrix inversion

A new hybrid Zhang neural network (HZNN) model is formulated to solve time-varying matrix inversion problem. The network is designed such that it encompasses the goodness of both gradient neural network (GNN) and Zhang neural network (ZNN). The stability of the network and convergence results are established theoretically. The rate of convergence of HZNN model is much faster compared to the classical GNN model and ZNN model for time-varying matrix inversion problem. Nonlinear activation functions for HZNN need not have to be odd and monotonically increasing for the network to converge, however with certain restrictions. Simulations are carried out to test the efficiency of HZNN over GNN and ZNN. Also, HZNN with odd and non-odd activation functions are examined for convergence. It is observed that the network converges much faster with the proposed non-odd function 3 (nof3). Furthermore, the HZNN model with odd and non-odd functions are applied to the kinematic control of a two-link planar manipulator for tracking a circular path.

High-order error function designs to compute time-varying linear matrix equations

This paper devotes to solving time-varying linear matrix equations (TVLMEs) from the viewpoint of high-order neural networks. For this purpose, high-order zeroing neural network (ZNN) models are designed and studied to solve TVLMEs. Compared with the first-order ZNN model for TVLMEs, the proposed high-order ZNN models are based on the design of the high-order error functions, and different order choices will generate different high-order ZNN models. Two nonlinear activation functions [i.e., tunable activation function (TunAF) and sign-bi-power activation function (SBPAF)] are used to speedup the high-order ZNN models for achieving the finite-time convergence. Furthermore, the strict theoretical analyses are provided to show that high-order ZNN models have better properties (especially in terms of convergence), when the nonlinear activation functions are used. Two numerical simulations are given to reveal the superior convergence property of the proposed high-order ZNN models, as compared to the first-order ZNN model for solving TVLMEs.

Sparse Neural Networks for Inference of Interwell Connectivity and Production Prediction

Summary A new neural network-based proxy model is presented for prediction of well production performance and interpretation of interwell connectivity in large oil fields. The workflow consists of two stages. The first stage uses feature learning to describe the general input-output relations that exist among the wells and to characterize the interwell connectivity. In the second stage, the identified interwell connectivity patterns are used as network topology to develop a multilayer neural network proxy model, with nonlinear activation functions, to predict the production performance of each producer. The estimation of connectivity patterns in the first stage serves as an interpretable feature-learning step to improve the effectiveness of the proxy model in the second stage. Identification of interwell connectivity is based on the selection property of the ℓ1-norm minimization by promoting sparsity in the estimated connectivity weights. The sparsity of the network is motivated by the domain knowledge that each production well is mainly supported by a few nearby injection wells. That is, a proxy model that allows each producer to communicate with all the other wells in the field is inherently redundant and must have an unknown sparse representation. The sparse structure of the connection weights in the resulting network is detected by promoting sparsity during the training process. Two synthetic numerical examples, with known solutions, are first used to demonstrate the functionality and effectiveness of ℓ1-norm regularization for interwell connectivity identification. The workflow is then applied to a real field waterflooding example in Long Beach to predict oil production and to infer interwell connectivity information. Overall, the workflow provides a proxy model that effectively combines the implicit physical information from simulated data with reservoir engineering insight to identify interwell connectivity and to predict well production trends.

COVSeg-NET: A deep convolution neural network for COVID-19 lung CT image segmentation.

COVID‐19 is a new type of respiratory infectious disease that poses a serious threat to the survival of human beings all over the world. Using artificial intelligence technology to analyze lung images of COVID‐19 patients can achieve rapid and effective detection. This study proposes a COVSeg‐NET model that can accurately segment ground glass opaque lesions in COVID‐19 lung CT images. The COVSeg‐NET model is based on the fully convolutional neural network model structure, which mainly includes convolutional layer, nonlinear unit activation function, maximum pooling layer, batch normalization layer, merge layer, flattening layer, sigmoid layer, and so forth. Through experiments and evaluation results, it can be seen that the dice coefficient, sensitivity, and specificity of the COVSeg‐NET model are 0.561, 0.447, and 0.996 respectively, which are more advanced than other deep learning methods. The COVSeg‐NET model can use a smaller training set and shorter test time to obtain better segmentation results.

Scalability of All-Optical Neural Networks Based on Spatial Light Modulators

Optical implementation of artificial neural networks has been attracting great attention due to its potential in parallel computation at speed of light. Although all-optical deep neural networks (AODNNs) with a few neurons have been experimentally demonstrated with acceptable errors recently, the feasibility of large scale AODNNs remains unknown because error might accumulate inevitably with increasing number of neurons and connections. Here, we demonstrate a scalable AODNN with programmable linear operations and tunable nonlinear activation functions. We verify its scalability by measuring and analyzing errors propagating from a single neuron to the entire network. The feasibility of AODNNs is further confirmed by recognizing handwritten digits and fashions respectively.

Joint Equalization of Linear and Nonlinear Impairments for PAM4 Short-Reach Direct Detection Systems

Various neural network (NN)-based equalizers have been proposed as effective digital signal processing (DSP) tools for short-reach optical direct detection (DD) communication systems. However, they are mainly designed for solving nonlinear problems due to the intrinsic nonlinear activation function. In this letter, we propose novel DSP schemes to deal with both linear and nonlinear impairments in short-reach DD optical links, including cascade feedforward NN (FNN)/cascade recurrent NN (RNN)-based equalizers, and combination of feedforward equalization (FFE) and traditional non-cascade FNN/RNN-based equalizers. A 50-Gb/s pulse amplitude modulation (PAM)-4 directly modulated laser (DML)-based system is experimentally carried out and the bit-error-rate (BER) performance of the proposed DSP algorithms are compared. Experimental results show that cascade FNN/RNN and FNN/RNN-FFE could help improve the BER performance compared with traditional non-cascade FNN/RNN, since linear impairments are taken into account by either FFE or the cascade structure. With the help of the joint optimization of both linear and nonlinear effects through the network training process, cascade FNN/RNN shows great superiority over other DSP schemes. We also investigate on the impact of decision feedback equalization (DFE) concatenated with FNN/RNN. Compared with using FNN/RNN followed by FFE, no significant BER improvement can be found using DFE in place of FFE.

Global Exponential Stability of Takagi-Sugeno Fuzzy Cohen-Grossberg Neural Network With Time-Varying Delays

In this letter, global exponential stability of Takagi-Sugeno fuzzy Cohen-Grossberg Neural Network (CGNN) with time-varying delay factor has been investigated based on the criteria of non-singular M-matrix and the Lyapunov stability technique. The stability inequality is derived with the help of Lipschitz condition for the nonlinear activation functions and a sufficient condition is shown to verify the criterion of the exponential stability condition for the CGNN with time-varying delay terms, which is described in the presence of delay terms of T-S Fuzzy model. Thus, the global exponential stability for T-S fuzzy CGNN in the presence of time-varying delay terms is derived in an easy way. This letter contains quite a new result for delayed CGNN for the T-S Fuzzy model. Finally, a numerical example is taken to validate the efficiency and unwavering quality, and to exhibit the superiority of the considered method as compared to the existing method for particular cases.

Adaptive sigmoid-like and PReLU activation functions for all-optical perceptron.

We present an approach for the generation of an adaptive sigmoid-like and PReLU nonlinear activation function of an all-optical perceptron, exploiting the bistability of an injection-locked Fabry-Perot semiconductor laser. The profile of the activation function can be tailored by adjusting the injection-locked side-mode order, frequency detuning of the input optical signal, Henry factor, or bias current. The universal fitting function for both families of the activation functions is presented.

Electrical load-temperature CNN for residential load forecasting

Residential load forecasting is a challenging problem due to complex relations among the hourly electrical load values along the time and also nonlinear relationships among the consumed electricity values and their associated temperature values. A nonlinear relationship extraction (NRE) method is proposed in this work. NRE obtains a load cube where each hourly load value is surrounded by load values of past, present and future hours in previous, same and next days of the same week and previous week. Then, a convolutional neural network (CNN) is used to extract the nonlinear relationships among the load values. In addition, a load-temperature cube is composed from the hourly load and temperature values of a week. Another CNN is trained by using the load-temperature cubes to learn the hidden nonlinear load-temperature features. The extracted features are given to a support vector regression (SVR) for load forecasting. The two dimensional convolutional operator is utilized for local feature extraction from the neighborhood regions; the nonlinear activation function is used for nonlinear feature extraction; and the SVR with Gaussian kernel is employed for minimizing the forecasting error. The forecasting results show the superior performance of the proposed method compared to several outstanding forecasters.

Multiple asymptotic stability of fractional-order quaternion-valued neural networks with time-varying delays

In this paper, the multiple asymptotic stability is investigated for fractional-order quaternion-valued neural networks (FQVNNs) with time-varying delays. The activation function is a nonmonotonic piecewise nonlinear activation function. By applying the Hamilton rules, the FQVNNs are transformed into real-valued systems. Then, according to the Brouwer’s fixed point theorem, three new conditions are proposed to ensure that there exist 34n equilibrium points. Moreover, by virtue of fractional-order Razumikhin theorem and Lyapunov function, a new condition is derived to guarantee the FQVNNs have 24n locally asymptotic stable equilibrium points. For the first time, the multiple asymptotic stability of delayed FQVNNs is investigated. Contrast to multistability analysis of integer-order quaternion-valued neural networks, this paper present different conclusions. Finally, two numerical simulations demonstrate the validity of the results.

Non‐linear activation function approximation using a REMEZ algorithm

Here a more accurate piecewise approximation (PWA) scheme for non-linear activation function is proposed. It utilizes a precision-controlled recursive algorithm to predict a sub-range; after that, the REMEZ algorithm is used to find the corresponding approximation function. The PWA realized in three ways: using first-order functions, that is, piecewise linear model, second-order functions (piecewise non-linear model), and hybrid-order model (a mixture of first-order and second-order functions). The hybrid-order approximation employs the second-order derivative of non-linear activation function to decide the linear and non-linear sub-regions, correspondingly the first-order and second-order functions are predicted, respectively. The accuracy is compared to the present state-of-the-art approximation schemes. The multi-layer perceptron model is designed to implement XOR-gate, and it uses an approximate activation function. The hardware utilization is measured using the TSMC 0.18-μm library with the Synopsys Design Compiler. Result reveals that the proposed approximation scheme efficiently approximates the non-linear activation functions.

Energy-efficient Mott activation neuron for full-hardware implementation of neural networks.

To circumvent the von Neumann bottleneck, substantial progress has been made towards in-memory computing with synaptic devices. However, compact nanodevices implementing non-linear activation functions are required for efficient full-hardware implementation of deep neural networks. Here, we present an energy-efficient and compact Mott activation neuron based on vanadium dioxide and its successful integration with a conductive bridge random access memory (CBRAM) crossbar array in hardware. The Mott activation neuron implements the rectified linear unit function in the analogue domain. The neuron devices consume substantially less energy and occupy two orders of magnitude smaller area than those of analogue complementary metal-oxide semiconductor implementations. The LeNet-5 network with Mott activation neurons achieves 98.38% accuracy on the MNIST dataset, close to the ideal software accuracy. We perform large-scale image edge detection using the Mott activation neurons integrated with a CBRAM crossbar array. Our findings provide a solution towards large-scale, highly parallel and energy-efficient in-memory computing systems for neural networks.

Kernelized Linear Autoencoder

This work proposes a new representation learning model called kernelized linear autoencoder. Instead of modeling non-linearity by the non-linear activation functions, we employ linear activations but account for non-linearity by the kernel trick. We propose four variants. The first one is the basic unsupervised kernelized linear autoencoder. The second one is a label consistent version for classification. The third one incorporates sparse sub-space clustering into the autoencoder formulation. Finally, we proposed a coupled version for domain adaptation problems. For each task, we compare our proposed technique with shallow and deep representation learning methods on benchmark problems. For each task, we perform significantly better than the methods compared against.

Research on The Model of UBI Car Insurance Rates Rating Based on CNN-Softmax Algorithm

With the support of vehicular networking technology, the rating of UBI auto insurance rate has a certain guiding significance for accurate pricing of the rate and personalized demand. Based on CNN-Softmax algorithm, a rating model of UBI auto insurance rate is proposed in this paper. The model first performs a series of operations such as convolution, pooling, and non-linear activation function mapping on CNN algorithm to enable it to extract features based on UBI customers’ driving behavior data, and then uses Softmax algorithm to classify customers according to their driving behaviors, thereby obtain UBI customer’s auto insurance rate rating. The empirical results show that compared with, CNN algorith, BP neural network algorithm and SVM algorithm, CNN-Softmax algorithm has a higher discrimination accuracy in the risk assessment of UBI customers’ driving behavior. And it is easy to implement in establishing rating model of UBI auto insurance rate. What’s more, it also has a good robustness, which can achieve better grade evaluations.