Single Hidden Layer Perceptron Research Articles

Add a SideNet to your MainNet

As the performance and popularity of deep neural networks has increased, so too has their computational cost. There are many effective techniques for reducing a network’s computational footprint--quantisation, pruning, knowledge distillation--, but these lead to models whose computational cost is the same regardless of their input. Our human reaction times vary with the complexity of the tasks we perform: easier tasks--e.g. telling apart dogs from boats--are executed much faster than harder ones--e.g. telling apart two similar-looking breeds of dogs. Driven by this observation, we develop a method for adaptive network complexity by attaching a small classification layer, which we call SideNet, to a large pretrained network, which we call MainNet. Given an input, the SideNet returns a classification if its confidence level, obtained via softmax, surpasses a user-determined threshold, and only passes it along to the large MainNet for further processing if its confidence is too low. This allows us to flexibly trade off the network’s performance with its computational cost. Experimental results show that simple single hidden layer perceptron SideNets added onto pretrained ResNet and BERT MainNets allow for substantial decreases in compute with minimal drops in performance on image and text classification tasks.

Autopilot design for tilt-rotor unmanned aerial vehicle with nacelle mounted wing extension using single hidden layer perceptron neural network

Single hidden layer perceptron neural network controllers combined with dynamic inversion are applied to the tilt-rotor unmanned aerial vehicle and its variant model with the nacelle mounted wing extension. The bandwidths of the inner loop and outer loop of the controller are designed using the timescale separation approach, which uses the combined analysis of the two loops. The bandwidth of each loop is selected to be close to each other using a combination of the pseudo-control-hedging and the pole-placement method. Similar to the previous studies on sigma-pi neural network, the dynamic inversion at hover conditions of the original tilt-rotor model is used as a baseline for both aircraft, and the compatible solution to the Lyapunov equation is suggested. The single hidden layer perceptron neural network minimizes the error of the inversion model through the back-propagation adaptation. The waypoint guidance is applied to the outermost loop of the neural network controller for autonomous flight which includes vertical take-off and landing as well as nacelle conversion. The simulation results under the two wind conditions for the tilt-rotor aircraft and its variant are presented. The south and north-west wind directions are simulated in order to compare with the results from the existing sigma-pi neural network, and the estimation results of the wind are presented.

Approximation by neural networks with weights varying on a finite set of directions

Approximation properties of the MLP (multilayer feedforward perceptron) model of neural networks have been investigated in a great deal of works over the last 30 years. It has been shown that for a large class of activation functions, a neural network can approximate arbitrarily well any given continuous function. The most significant result on this problem belongs to Leshno, Lin, Pinkus and Schocken. They proved that the necessary and sufficient condition for any single hidden layer network to have the u.a.p. (universal approximation property) is that its activation function not be a polynomial. Some authors (White, Stinchcombe, Ito, and others) showed that a single hidden layer perceptron with some bounded weights can also have the u.a.p. Thus the weights required for u.a.p. are not necessary to be of an arbitrarily large magnitude. But what if they are too restricted? How can one learn approximation properties of networks with arbitrarily restricted set of weights? The current paper makes a first step in solving this general problem. We consider neural networks with sets of weights consisting of a finite number of directions. Our purpose is to characterize compact sets X in the d-dimensional space such that the network can approximate any continuous function over X. In a special case, when weights vary only on two directions, we give a lower bound for the approximation error and find a sufficient condition for the network to be a best approximation.

Free-knot Splines Approximation of s -monotone Functions

Let I be a finite interval and r,s∈N. Given a set M, of functions defined on I, denote by Δ+sM the subset of all functions y∈M such that the s-difference Δτsy(⋅) is nonnegative on I, ∀τ>0. Further, denote by Δ+sWpr, the class of functions x on I with the seminorm ‖x(r)‖Lp≤1, such that Δτsx≥0, τ>0. Let Mn(hk):={∑i=1ncihk(wit−θi)∣ci,wi, θi∈R, be a single hidden layer perceptron univariate model with n units in the hidden layer, and activation functions hk(t)=t+k, t∈R, k∈N0. We give two-sided estimates both of the best unconstrained approximation E(Δ+sWpr,Mn(hk))Lq, k=r−1,r, s=0,1,...,r+1, and of the best s-monotonicity preserving approximation E(Δ+sWpr,Δ+sMn(hk))Lq, k=r−1,r, s=0,1,...,r+1. The most significant results are contained in theorem 2.2.

Approximate models for nonlinear dynamical systems and their generalization properties

In this paper, a methodology to obtain approximate models from input-output data for nonlinear, causal, time invariant discrete systems having a certain type of continuity condition called fading memory is presented. The region or domain of the input space, where the model can be applicable, is studied, as well as the importance of this study in applications as data processing and the qualification of the model quality. The structure is synthesized using a finite set of discrete Kautz systems, followed by a single hidden layer perceptron. The number of the Kautz systems is evaluated by Lipschitz quotients, while the number of hidden neurons is bounded using a pruning technique. Examples illustrating the proposed methodology are presented.

State‐space nonlinear process modeling: Identification and universality

AbstractA practical approach was used to identify nonlinear models from available data. Knowing the true dynamics of a process allows implementation of a better and more efficient control scheme. Since this knowledge is not always available, a valid alternative to obtain a model is to identify the system. For that purpose, the DABNet (Decoupled A‐B Net) structure was used that is composed of a decoupled linear dynamic system followed by a nonlinear static map. The linear dynamic system was initially spanned by a set of discrete Laguerre systems and then cascaded with a single hidden layer Perceptron. A model reduction technique (linear balancing) used on the hidden nodes of the neural network as part of the identification process makes it possible not only to identify the main time constants, but reduce the dimensionality of the Perceptron input. It was proven that the DABNet structure can approximate every nonlinear, causal, discrete time invariant, multiple‐input single‐output system with fading memory. The final model consists of a linear state‐space system whose states, decoupled by input, are mapped by a neural network. Results concerning the application of the methodology to the approximation of a CSTR and a polymer process are presented.