Nonlinear Activation Function Research Articles

Applying and dissecting LSTM neural networks and regularized learning for dynamic inferential modeling

Deep learning models such as the long short-term memory (LSTM) network have been applied for dynamic inferential modeling. However, many studies apply LSTM as a black-box approach without examining the necessity and usefulness of the internal LSTM gates for inferential modeling. In this paper, we use LSTM as a state space realization and compare it with linear state space modeling and statistical learning methods, including N4SID, partial least squares, the Lasso, and support vector regression. Two case studies on an industrial 660 MW boiler and a debutanizer column process indicate that LSTM underperforms all other methods. LSTM is shown to be capable of outperforming linear methods for a simulated reactor process with severely excited nonlinearity in the data. By dissecting the sub-components of a simple LSTM model, the effectiveness of the LSTM gates and nonlinear activation functions is scrutinized.

B-LNN: Inference-time linear model for secure neural network inference

Machine Learning as a Service (MLaaS) provides clients with well-trained neural networks for predicting private data. Conventional prediction processes of MLaaS require clients to send sensitive inputs to the server, or proprietary models must be stored on the client-side device. The former reveals client privacy, while the latter harms the interests of model providers. Existing works on privacy-preserving MLaaS introduce cryptographic primitives to allow two parties to perform neural network inference without revealing either party's data. However, nonlinear activation functions bring high computational overhead and response delays to the inference process of these schemes.In this paper, we analyze the mechanism by which activation functions enhance model expressivity, and design an activation function S-cos that is friendly to secure neural network inference. Our proposed S-cos can be re-parameterized into a linear layer during the inference phase. Further, we propose an inference-time linear model called Beyond Linear Neural Network (B-LNN) equipped with S-cos, which exhibits promising performance on several benchmark datasets.

Low power resource efficient CORDIC enabled neuron architecture using 45 nm CMOS technology

In this paper problem is addressed in the current study by providing resource-efficient CORDIC enabled neuron architecture (RECON) that can be customized to calculate both block of multiply-accumulate (MAC) unit and non-linear activation function (AF) operations. The CORDIC-enabled architecture implements MAC and AF operations using linear and trigonometric relationships, respectively. All physical parameters of the proposed design are built and verified using Cadence Virtuoso @ 45 nm technology. As compared to the conventional art of MAC design, our implementation of the signed fixed-point 8-bit MAC results in a 70% reduction in area, latency, and power product (ALP), as well as a 45 percent reduction in area, a 28% reduction in power dissipation, and a 20% reduction in latency. Both the process adjustments and the device mismatch are subjected to Monte-Carlo simulations. The proposed design is based on resource-intensive components such as multipliers and non-linear Activation Functions, modern hardware implementations of DNNs require more space (AFs). To access input features, weights, and biases, and improved on-chip quantized log2 based memory addressing approach is implemented. The bandwidth needs of DNNs' external memory are therefore decreased and dynamically adjusted. The Taylor series is also used to extract intensive higher speed and resource-efficient memory components for the various activation functions, and its order expansion has been altered for increased test accuracy. The MNIST dataset is used in earlier studies.

A Comparative Study to Estimate Fuel Consumption: A Simplified Physical Approach against a Data-Driven Model

Two methods were compared to predict a ship’s fuel consumption: the simplified naval architecture method (SNAM) and the deep neural network (DNN) method. The SNAM relied on limited operational data and employed a simplified technique to estimate a ship’s required power by determining its resistance in calm water. Here, the Holtrop–Mennen technique obtained hydrostatic information for each selected voyage, the added resistance in the encountered natural seaways, and the brake power required for each scenario. Additional characteristics, such as efficiency factors, were derived from literature surveys and from assumed working hypotheses. The DNN method comprised multiple fully connected layers with the nonlinear activation function rectified linear unit (ReLU). This machine-learning-based method was trained on more than 12,000 sample voyages, and the tested data were validated against realistic operational data. Our results demonstrated that, for some ship topologies (general cargo and containerships), the physical models predicted more accurately the realistic data than the machine learning approach despite the lack of relevant operational parameters. Nevertheless, the DNN method was generally capable of yielding reasonably accurate predictions of fuel consumption for oil tankers, bulk carriers, and RoRo ships.

Energy Efficient and Secure Neural Network–based Disease Detection Framework for Mobile Healthcare Network

Adopting mobile healthcare network (MHN) services such as disease detection is fraught with concerns about the security and privacy of the entities involved and the resource restrictions at the Internet of Things (IoT) nodes. Hence, the essential requirements for disease detection services are to (i) produce accurate and fast disease detection without jeopardizing the privacy of health clouds and medical users and (ii) reduce the computational and transmission overhead (energy consumption) of the IoT devices while maintaining the privacy. For privacy preservation of widely used neural network– (NN) based disease detection, existing literature suggests either computationally heavy public key fully homomorphic encryption (FHE), or secure multiparty computation, with a large number of interactions. Hence, the existing privacy-preserving NN schemes are energy consuming and not suitable for resource-constrained IoT nodes in MHN. This work proposes a lightweight, fully homomorphic, symmetric key FHE scheme (SkFhe) to address the issues involved in implementing privacy-preserving NN. Based on SkFhe, widely used non-linear activation functions ReLU and Leaky ReLU are implemented over the encrypted domain. Furthermore, based on the proposed privacy-preserving linear transformation and non-linear activation functions, an energy-efficient, accurate, and privacy-preserving NN is proposed. The proposed scheme guarantees privacy preservation of the health cloud’s NN model and medical user’s data. The experimental analysis demonstrates that the proposed solution dramatically reduces the overhead in communication and computation at the user side compared to the existing schemes. Moreover, the improved energy efficiency at the user is accomplished with reduced diagnosis time without sacrificing classification accuracy.

Forecasting the Tourism Demand of Türkiye Using Artificial Neural Network (ANN) Approach

Modelling and forecasting the tourism demand is an important research area for both public and private sectors for planning the travel and accommodation facilities in advance. Therefore, modelling the tourism demand in terms of the number of tourists is an active research area for both tourism, economics and financial studies. In this work, the tourism demand of Türkiye is modelled using artificial neural network methods. The tourism demand represented by the number of tourists is taken as the dependent variable while the past values of the number of tourists is considered as the inputs. Furthermore, the tourism revenue is also modelled employing the same autoregressive artificial neural network approach. Monthly data in the period of 2008-2022 which are taken from the official sources are used as the research material. The multilayer perceptron type artificial neural networks with nonlinear neural activation functions is selected for the accurate modelling of the highly seasonal tourism demand data. The modelling and forecasting results show that the developed artificial neural network reaches high accuracy for modelling both the tourism demand and the tourism revenue such that the coefficient of determination of the developed artificial neural network models have the values of R2=0.949 and R2=0.840 for the tourism demand and tourism revenue, respectively, providing an objective assessment of the accuracy of the developed models. It is concluded that the multilayer perceptron based artificial neural network methods can be used to model the tourism demand therefore this type of modelling is expected to be helpful for tourism planners by providing them the tourism demand forecast in advance.

Computational Complexity-based Fractional-Order Neural Network Models for the Diagnostic Treatments and Predictive Transdifferentiability of Heterogeneous Cancer Cell Propensity

Neural networks and fractional order calculus are powerful tools for system identification through which there exists the capability of approximating nonlinear functions owing to the use of nonlinear activation functions and of processing diverse inputs and outputs as well as the automatic adaptation of synaptic elements through a specified learning algorithm. Fractional-order calculus, concerning the differentiation and integration of non-integer orders, is reliant on fractional-order thinking which allows better understanding of complex and dynamic systems, enhancing the processing and control of complex, chaotic and heterogeneous elements. One of the most characteristic features of biological systems is their different levels of complexity; thus, chaos theory seems to be one of the most
 applicable areas of life sciences along with nonlinear dynamic and complex systems of living and non-living environment. Biocomplexity, with multiple scales ranging from molecules to cells and organisms, addresses complex structures and behaviors which emerge from nonlinear interactions of active biological agents. This sort of emergent complexity is concerned with the organization of molecules
 into cellular machinery by that of cells into tissues as well as that of individuals to communities. Healthy systems sustain complexity in their lifetime and are chaotic, so complexity loss or chaos loss results in diseases. Within the mathematics-informed frameworks, fractional-order calculus based Artificial Neural Networks (ANNs) can be employed for accurate understanding of complex biological
 processes. This approach aims at achieving optimized solutions through the maximization of the model’s accuracy and minimization of computational burden and exhaustive methods. Relying on a transdifferentiable mathematics-informed framework and multifarious integrative methods concerning computational complexity, this study aims at establishing an accurate and robust model based upon
 integration of fractional-order derivative and ANN for the diagnosis and prediction purposes for cancer cell whose propensity exhibits various transient and dynamic biological properties. The other aim is concerned with showing the significance of computational complexity for obtaining the fractional-order derivative with the least complexity in order that optimized solution could be achieved. The multifarious
 scheme of the study, by applying fractional-order calculus to optimization methods, the advantageous aspect concerning model accuracy maximization has been demonstrated through the proposed method’s applicability and predictability aspect in various domains manifested by dynamic and nonlinear nature displaying different levels of chaos and complexity.

Image sensing with multilayer nonlinear optical neural networks

Optical imaging is commonly used for both scientific and technological applications across industry and academia. In image sensing, a measurement, such as of an object's position, is performed by computational analysis of a digitized image. An emerging image-sensing paradigm breaks this delineation between data collection and analysis by designing optical components to perform not imaging, but encoding. By optically encoding images into a compressed, low-dimensional latent space suitable for efficient post-analysis, these image sensors can operate with fewer pixels and fewer photons, allowing higher-throughput, lower-latency operation. Optical neural networks (ONNs) offer a platform for processing data in the analog, optical domain. ONN-based sensors have however been limited to linear processing, but nonlinearity is a prerequisite for depth, and multilayer NNs significantly outperform shallow NNs on many tasks. Here, we realize a multilayer ONN pre-processor for image sensing, using a commercial image intensifier as a parallel optoelectronic, optical-to-optical nonlinear activation function. We demonstrate that the nonlinear ONN pre-processor can achieve compression ratios of up to 800:1 while still enabling high accuracy across several representative computer-vision tasks, including machine-vision benchmarks, flow-cytometry image classification, and identification of objects in real scenes. In all cases we find that the ONN's nonlinearity and depth allowed it to outperform a purely linear ONN encoder. Although our experiments are specialized to ONN sensors for incoherent-light images, alternative ONN platforms should facilitate a range of ONN sensors. These ONN sensors may surpass conventional sensors by pre-processing optical information in spatial, temporal, and/or spectral dimensions, potentially with coherent and quantum qualities, all natively in the optical domain.

Application of Improved Image Processing Technology in New Media Short Video Quality Improvement in Film and Television Postproduction

With the development of film and television industry and the rise of new media short video, film and television postproduction process has higher and higher requirements for image quality. In film and television postproduction, optimizing the image quality can enhance the resolution and make the image more vivid and detailed. High-quality image can fully embody the value of film and television, as well as promote the development of new media short videos. This paper optimizes image quality by improving image processing technology, thus improving the quality and value of film and television and new media short videos. In this paper, a convolutional neural network combined with a nonlinear activation function is used to establish an improved image processing technology model to efficiently extract image features. This technology can enhance the ability to extract image features, improve the accuracy of image feature extraction, and then improve the image resolution and details, thus improving the image quality in the process of film and television postproduction. The results show that the average value of PSNR is 30.29. The average value of PSNR of the proposed algorithm is higher than that of other algorithms, indicating that the error between the image processed by the proposed algorithm and the original image is small. The average SSIM of the algorithm in this paper is 0.903, which is closer to 1. Compared with other algorithms, the structure processed by the algorithm in this paper is more similar to the original structure, resulting in a better graph. The algorithm in this paper has the best performance on both the peak signal-to-noise ratio (PSNR) and the structural similarity (SSIM). The improved image processing technology proposed in this paper can effectively improve the accuracy of image feature extraction, making film and television or new media short video images of higher quality.

Deep learning soliton dynamics and complex potentials recognition for 1D and 2D [formula omitted]-symmetric saturable nonlinear Schrödinger equations

In this paper, we firstly extend the physics-informed neural networks (PINNs) to learn data-driven stationary and non-stationary solitons of 1D and 2D saturable nonlinear Schrödinger equations (SNLSEs) with two fundamental PT-symmetric Scarf-II and periodic potentials in optical fibers. Secondly, the data-driven inverse problems are studied for PT-symmetric potential functions discovery rather than just potential parameters in the 1D and 2D SNLSEs. Particularly, we propose a modified PINNs (mPINNs) scheme to identify directly the PT potential functions of the 1D and 2D SNLSEs by the solution data. And the inverse problems about 1D and 2D PT-symmetric potentials depending on propagation distance z are also investigated using mPINNs method. We also identify the potential functions by the PINNs applied to the stationary equation of the SNLSE. Furthermore, two network structures are compared under different parameter conditions such that the predicted PT potentials can achieve the similar high accuracy. These results illustrate that the established deep neural networks can be successfully used in 1D and 2D SNLSEs with high accuracies. Moreover, some main factors affecting neural networks performance are discussed in 1D and 2D PT Scarf-II and periodic potentials, including activation functions, structures of the networks, and sizes of the training data. In particular, twelve different nonlinear activation functions are in detail analyzed containing the periodic and non-periodic functions such that it is concluded that selecting activation functions according to the form of solution and equation usually can achieve better effect.

Towards non-linearly activated ZNN model for constrained manipulator trajectory tracking

As a powerful method for time-varying problems solving, the zeroing neural network (ZNN) is widely applied in many practical applications that can be modeled as time-varying linear matrix equations (TVLME). Generally, existing ZNN models solve these TVLME problems in the ideal no noise situation without inequality constraints, but the TVLME with noises and inequality constraints are rarely considered. Therefore, a non-linear activation function is designed, and based on the non-linear activation function, a non-linearly activated ZNN (NAZNN) model is proposed for solving constrained TVLME (CTVLME) problems. The convergence and robustness of the proposed NAZNN model are verified theoretically, and simulation results further demonstrate the effectiveness and superiority of the NAZNN model in dealing with CTVLME and the constrained robot manipulator trajectory tracking problems. In addition, the wheeled robot trajectory tracking fault problems with physical constraints are also analyzed theoretically, and the proposed NAZNN model is also applied to the manipulator trajectory tracking fault problem, and the experimental results prove that the NAZNN model also deal with the manipulator trajectory tracking fault problem effectively.

Universality Laws for High-Dimensional Learning With Random Features

We prove a universality theorem for learning with random features. Our result shows that, in terms of training and generalization errors, a random feature model with a nonlinear activation function is asymptotically equivalent to a surrogate linear Gaussian model with a matching covariance matrix. This settles a so-called Gaussian equivalence conjecture based on which several recent papers develop their results. Our method for proving the universality theorem builds on the classical Lindeberg approach. Major ingredients of the proof include a leave-one-out analysis for the optimization problem associated with the training process and a central limit theorem, obtained via Stein’s method, for weakly correlated random variables.

Delay-Variation-Dependent Criteria on Extended Dissipativity for Discrete-Time Neural Networks With Time-Varying Delay.

This article is concerned with the extended dissipativity of discrete-time neural networks (NNs) with time-varying delay. First, the necessary and sufficient condition on matrix-valued polynomial inequalities reported recently is extended to a general case, where the variable of the polynomial does not need to start from zero. Second, a novel Lyapunov functional with a delay-dependent Lyapunov matrix is constructed by taking into consideration more information on nonlinear activation functions. By employing the Lyapunov functional method, a novel delay and its variation-dependent criterion are obtained to investigate the effects of the time-varying delay and its variation rate on several performances, such as performance, passivity, and performance, of a delayed discrete-time NN in a unified framework. Finally, a numerical example is given to show that the proposed criterion outperforms some existing ones.

All-Optical Nonlinear Activation Function Based on Germanium Silicon Hybrid Asymmetric Coupler

Nonlinear activation functions are crucial for optical neural networks (ONNs) to achieve more various functions. However, the current nonlinear functions suffer from some dilemma, including high power consumption, high loss, and limited bandwidth. Here, we propose and demonstrate an all-optical implementation of a nonlinear activation function based on germanium silicon hybrid integration. The principle lies in the intrinsic absorption and the carrier-induced refractive index change of germanium in C -band. It has a large operating bandwidth and a response frequency of 70 MHz, with a loss of 4.28 dB and a threshold power of 5.1 mW. Adopting it to the MNIST handwriting data set classification, it shows an improvement in accuracy from 91.6% to 96.8%. This proves that our scheme has great potential for advanced ONN applications.

Bias-Scalable Near-Memory CMOS Analog Processor for Machine Learning

Bias-scalable analog computing is attractive for implementing machine learning (ML) processors with distinct power-performance specifications. For instance, ML implementations for server workloads are focused on higher computational throughput for faster training, whereas ML implementations for edge devices are focused on energy-efficient inference. In this paper, we demonstrate the implementation of bias-scalable approximate analog computing circuits using the generalization of the margin-propagation principle called shape-based analog computing (S-AC). The resulting S-AC core integrates several near-memory compute elements, which include: (a) non-linear activation functions; (b) inner-product compute circuits; and (c) a mixed-signal compressive memory, all of which can be scaled for performance or power while preserving its functionality. Using measured results from prototypes fabricated in a 180nm CMOS process, we demonstrate that the performance of computing modules remains robust to transistor biasing and variations in temperature. In this paper, we also demonstrate the effect of bias-scalability and computational accuracy on a simple ML regression task.

Stationary states of a dissipative two-qubit quantum channel and their applications for quantum machine learning

Entangled-state preparation and preservation are the cornerstones of any quantum information platform. However, the strongest adversaries in quantum information science are unwanted environmental effects such as decoherence and dissipation. Here, we address how to control and harness these unwanted effects that arise from the coupling of a system with its environment, to provide stationary entangled states for quantum machine learning (QML). To do so, we design a dissipative quantum channel, i.e., a two-qubit system interacting with a squeezed vacuum field reservoir, and study the output state of the channel by solving the corresponding master equation, especially, in the small squeezing regime. We show that the time-dependent output state of the channel is the so-called two-qubit X-states that generalize many families of entangled two-qubit states. Also, by considering a general Bell diagonal state as the initial state of the system, we reveal that this dissipative channel generates two well-known classes of entangled mixed state and Werner-like states in the steady-state regime. Moreover, this channel provides an efficient way to determine whether a given initial state results in a stationary entangled state or not. Finally, we examine the potential application of the designed two-qubit channel for QML. In this line, we propose a general theoretical scheme for quantum neural networks (QNNs) implemented with the variational quantum circuits, which encode data in continuous variables (CVs) of the two-qubit states. The linear (also referred to as affine) and nonlinear (activation function) transformations are enacted in the QNN using the stationary states of the two-qubit channel and measurement process, respectively. Finally, we proceed to test our proposed model by solving some supervised binary classification tasks. Integrating the non-unitary transformation and parallel-processed neural computing on such a two-qubit channel establishes the requirements for a meaningful QNN. Such a CV-QNN model with sufficient layers may execute any algorithm implementable on a universal CV quantum computer.

Mixed [formula omitted] and passive exponential synchronization for singular Markov jump neural networks based on hybrid event trigger scheme

This article considers the driving-response-based exponential synchronization issue for delayed singular Markov jump neural networks by virtue of hybrid event triggering strategy. By employing singular value decomposition method, the regularity and impulse-freeness of synchronization error system are obtained. Under the framework of hybrid event-triggered scheme, the novel exponential admissibility conditions, mixed H∞ and passive performance are acquired by establishing improved Lyapunov–Krasovskill functional and dealing with nonlinear neuron activation function. Furthermore, the desired synchronization state feedback controller is designed by virtue of linear matrix inequalities. Finally, a numerical example and an analog resistance-capacitance network circuit are used to verify the effectiveness and practicability of the method.

Growing axons: greedy learning of neural networks with application to function approximation

Abstract We propose a new method for learning deep neural network models, which is based on a greedy learning approach: we add one basis function at a time, and a new basis function is generated as a non-linear activation function applied to a linear combination of the previous basis functions. Such a method (growing deep neural network by one neuron at a time) allows us to compute much more accurate approximants for several model problems in function approximation.

Robustness quantification method for network intrusion detection models

Deep learning techniques have been successfully applied to network intrusion detection tasks, but as in the case of autonomous driving and face recognition, the reliability of the system itself has become a pressing issue. Robustness is a key attribute to determine whether a deep learning system is secure and reliable, and we also choose to explore the security of intrusion detection models from a new perspective of robustness quantification. In this paper, we focus on the intrusion detection model based on long and short-term memory, and use a fine-grained linear approximation method to derive a more accurate robustness bound on the nonlinear activation function with tighter linear constraints. We can use this bound to quantitatively measure the robustness of the detection model and determine whether the model is susceptible to the influence of adversarial samples. In our experiments, we test networks with various structures on the MNIST dataset, and the results show that our proposed method can effectively deduce the robustness bounds of output elements, and has good scalability and applicability.

Intrinsic Lexical Intentionality and the Mathematics of Homomorphism

Moisl [1, 2] proposed a model of how the brain implements intrinsic intentionality with respect to lexical and sentence meaning, where 'intrinsic' is understood as 'independent of interpretation by observers external to the cognitive agent'. The discussion in both was mainly philosophical and qualitative; the present paper gives a mathematical account of the distance structure preservation that underlies the proposed mechanism of intrinsic intentionality. The three-layer autoassociative multilayer perceptron (aMLP) architecture with nonlinear hidden and linear output layers is the component in the model which generates representations homomorphic with the environment. The discussion first cites existing work which identifies the aMLP as an implementation architecture for principal component analysis (PCA), and then goes on to argue that the homomorphism characteristic of linear functions like PCA extends to aMLPs with nonlinear activation functions in the hidden layer. The discussion is in two main parts: the first part outlines the model, and the second presents the mathematical account.