Optimal Rational Approximation Research Articles

Rational approximation of operator semigroups via the [formula omitted]-calculus

We improve the classical results by Brenner and Thomée on rational approximations of operator semigroups. In the setting of Hilbert spaces, we introduce a finer regularity scale for initial data, provide sharper stability estimates, and obtain optimal approximation rates. Moreover, we strengthen a result due to Egert-Rozendaal on subdiagonal Padé approximations of operator semigroups. Our approach is direct and based on the theory of the B- functional calculus developed recently. On the way, we elaborate a new and simple approach to construction of the B-calculus thus making the paper essentially self-contained.

Optimal Rates of Approximation by Shallow ReLU$$^k$$ Neural Networks and Applications to Nonparametric Regression

AbstractWe study the approximation capacity of some variation spaces corresponding to shallow ReLU$$^k$$ k neural networks. It is shown that sufficiently smooth functions are contained in these spaces with finite variation norms. For functions with less smoothness, the approximation rates in terms of the variation norm are established. Using these results, we are able to prove the optimal approximation rates in terms of the number of neurons for shallow ReLU$$^k$$ k neural networks. It is also shown how these results can be used to derive approximation bounds for deep neural networks and convolutional neural networks (CNNs). As applications, we study convergence rates for nonparametric regression using three ReLU neural network models: shallow neural network, over-parameterized neural network, and CNN. In particular, we show that shallow neural networks can achieve the minimax optimal rates for learning Hölder functions, which complements recent results for deep neural networks. It is also proven that over-parameterized (deep or shallow) neural networks can achieve nearly optimal rates for nonparametric regression.

Approximating smooth and sparse functions by deep neural networks: Optimal approximation rates and saturation

Constructing neural networks for function approximation is a classical and longstanding topic in approximation theory. In this paper, we aim at constructing deep neural networks with three hidden layers using a sigmoidal activation function to approximate smooth and sparse functions. Specifically, we prove that the constructed deep nets with controllable magnitude of free parameters can reach the optimal approximation rate in approximating both smooth and sparse functions. In particular, we prove that neural networks with three hidden layers can avoid the phenomenon of saturation, i.e., the phenomenon that for some neural network architectures, the approximation rate stops improving for functions of very high smoothness.

Low-Memory Krylov Subspace Methods for Optimal Rational Matrix Function Approximation

We describe a Lanczos-based algorithm for approximating the product of a rational matrix function with a vector. This algorithm, which we call the Lanczos method for optimal rational matrix function approximation (Lanczos-OR), returns the optimal approximation from a given Krylov subspace in a norm depending on the rational function's denominator, and can be computed using the information from a slightly larger Krylov subspace. We also provide a low-memory implementation which only requires storing a number of vectors proportional to the denominator degree of the rational function. Finally, we show that Lanczos-OR can be used to derive algorithms for computing other matrix functions, including the matrix sign function and quadrature based rational function approximations. In many cases, it improves on the approximation quality of prior approaches, including the standard Lanczos method, with little additional computational overhead.

Time response optimal rational approximation: Improvement of time responses of MSBL based approximate fractional order derivative operators by using gradient descent optimization

Fractional calculus has become a popular topic in recent years. Its applications have been seen frequently in the analysis and solution of many engineering problems. Since, the exact realization of fractional order (FO) operators is not computationally feasible, several integer order approximations methods such as Oustaloup, Matsuda, Curve Fitting, modified stability boundary locus (MSBL) are utilized for frequency domain approximation of FO operators. These methods present advantages in frequency response approximation. However, real-world engineering applications require more accurate time responses to implement FO operators. Thus, any improvement in the approximate transfer function of FO systems will positively contribute to realization performance of FO system models in real-world applications. In this work, time responses of MSBL based approximate FO derivative models are improved by using gradient descent optimization (GDO) technique. Frequency domain approximation methods do not guarantee optimal approximation performance for the step response of FO elements. To deal with this weakness, author proposes a new GDO based rational approximation scheme that can considerably enhance the step response performance of MSBL method without deteriorating frequency response approximation performance. Results for the improved approximate FO model obtained by GDO (GDO-MSBL) are compared with the results of approximate FO models for popular FO approximation methods. The analog circuit realization is performed for the derivative operator, and the performance improvements for the proposed method are shown on the FO-PID controller and FO filter.

“Near-best” polynomial approximation of harmonic functions on compact sets in [formula omitted

For a function u that is continuous on a compact set K and harmonic in its interior we construct a sequence of polynomial approximants that give almost optimal rate of approximation on K, and in the interior points of K converge faster than on the whole of K. We also show that the geometric convergence inside K is possible for sets whose boundary is an analytically bounded Jordan domain.

Optimal approximation rate of ReLU networks in terms of width and depth

This paper concentrates on the approximation power of deep feed-forward neural networks in terms of width and depth. It is proved by construction that ReLU networks with width O(max⁡{d⌊N1/d⌋,N+2}) and depth O(L) can approximate a Hölder continuous function on [0,1]d with an approximation rate O(λd(N2L2ln⁡N)−α/d), where α∈(0,1] and λ>0 are Hölder order and constant, respectively. Such a rate is optimal up to a constant in terms of width and depth separately, while existing results are only nearly optimal without the logarithmic factor in the approximation rate. More generally, for an arbitrary continuous function f on [0,1]d, the approximation rate becomes O(dωf((N2L2ln⁡N)−1/d)), where ωf(⋅) is the modulus of continuity. We also extend our analysis to any continuous function f on a bounded set. Particularly, if ReLU networks with depth 31 and width O(N) are used to approximate one-dimensional Lipschitz continuous functions on [0,1] with a Lipschitz constant λ>0, the approximation rate in terms of the total number of parameters, W=O(N2), becomes O(λWln⁡W), which has not been discovered in the literature for fixed-depth ReLU networks.

A deep network construction that adapts to intrinsic dimensionality beyond the domain

We study the approximation of two-layer compositions f(x)=g(ϕ(x)) via deep networks with ReLU activation, where ϕ is a geometrically intuitive, dimensionality reducing feature map. We focus on two intuitive and practically relevant choices for ϕ: the projection onto a low-dimensional embedded submanifold and a distance to a collection of low-dimensional sets. We achieve near optimal approximation rates, which depend only on the complexity of the dimensionality reducing map ϕ rather than the ambient dimension. Since ϕ encapsulates all nonlinear features that are material to the function f, this suggests that deep nets are faithful to an intrinsic dimension governed by f rather than the complexity of the domain of f. In particular, the prevalent assumption of approximating functions on low-dimensional manifolds can be significantly relaxed using functions of type f(x)=g(ϕ(x)) with ϕ representing an orthogonal projection onto the same manifold.

Rational Implementation of Fractional Calculus Operator Based on Quadratic Programming

When fractional calculus operators and models are implemented rationally, there exist some problems such as low approximation accuracy of rational approximation function, inability to specify arbitrary approximation frequency band, or poor robustness. Based on the error criterion of the least squares method, a quadratic programming method based on the frequency-domain error is proposed. In this method, the frequency-domain error minimization between the fractional operator s ± r and its rational approximation function is transformed into a quadratic programming problem. The results show that the construction method of the optimal rational approximation function of fractional calculus operator is effective, and the optimal rational approximation function constructed can effectively approximate the fractional calculus operator and model for the specified approximation frequency band.

Curve Based Approximation of Measures on Manifolds by Discrepancy Minimization

The approximation of probability measures on compact metric spaces and in particular on Riemannian manifolds by atomic or empirical ones is a classical task in approximation and complexity theory with a wide range of applications. Instead of point measures we are concerned with the approximation by measures supported on Lipschitz curves. Special attention is paid to push-forward measures of Lebesgue measures on the unit interval by such curves. Using the discrepancy as distance between measures, we prove optimal approximation rates in terms of the curve’s length and Lipschitz constant. Having established the theoretical convergence rates, we are interested in the numerical minimization of the discrepancy between a given probability measure and the set of push-forward measures of Lebesgue measures on the unit interval by Lipschitz curves. We present numerical examples for measures on the 2- and 3-dimensional torus, the 2-sphere, the rotation group on mathbb R^3 and the Grassmannian of all 2-dimensional linear subspaces of {mathbb {R}}^4. Our algorithm of choice is a conjugate gradient method on these manifolds, which incorporates second-order information. For efficient gradient and Hessian evaluations within the algorithm, we approximate the given measures by truncated Fourier series and use fast Fourier transform techniques on these manifolds.

Optimal Approximation of Fractional-Order Butterworth Filter Based on Weighted Sum of Classical Butterworth Filters

In this paper, a new two-steps design strategy is introduced for the optimal rational approximation of the fractional-order Butterworth filter. At first, the weighting factors of the summation between the n <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">th</sup> -order and the (n + 1) <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">th</sup> -order Butterworth filters are optimally determined. Subsequently, this model is employed as an initial point for another optimization routine, which minimizes the magnitude-frequency error relative to the (n+α) <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">th</sup> -order, where α ∈ (0, 1), Butterworth filter. The proposed approximant demonstrates improved performance about the magnitude mean squared error compared to the state-of-the-art design for six decades of bandwidth, but the introduced approach does not require a fractional-order transfer function model and the approximant of the s <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">α</sup> operator. The proposed strategy also avoids the use of the cascading technique to yield higher-order fractional-order Butterworth filter models. The performance of the proposed 1.5 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">th</sup> -order Butterworth filter in follow-the-leader feedback topology is verified through SPICE simulations and its hardware implementation based on Analog Devices AD844AN-type current feedback operational amplifier.

Approximation of Smoothness Classes by Deep Rectifier Networks

We consider approximation rates of sparsely connected deep rectified linear unit (ReLU) and rectified power unit (RePU) neural networks for functions in Besov spaces $B^\alpha_{q}(L^p)$ in arbitrary dimension $d$, on general domains. We show that \alert{deep rectifier} networks with a fixed activation function attain optimal or near to optimal approximation rates for functions in the Besov space $B^\alpha_{\tau}(L^\tau)$ on the critical embedding line $1/\tau=\alpha/d+1/p$ for \emph{arbitrary} smoothness order $\alpha>0$. Using interpolation theory, this implies that the entire range of smoothness classes at or above the critical line is (near to) optimally approximated by deep ReLU/RePU networks.

Optimal Modelling of (1 + α) Order Butterworth Filter under the CFE Framework

This paper presents the optimal rational approximation of (1+α) order Butterworth filter, where α ∊ (0,1) under the continued fraction expansion framework, by employing a new cost function. Two simple techniques based on the constrained optimization and the optimal pole-zero placements are proposed to model the magnitude-frequency response of the fractional-order lowpass Butterworth filter (FOLBF). The third-order FOLBF approximants achieve good agreement to the ideal characteristic for six decades of design bandwidth. Circuit realization using the current feedback operational amplifier is presented, and the modelling efficacy is validated in the OrCAD PSPICE platform.

Design of Fractional-Order PID for Stabilized Sight System via Internal Model Control Approach

This paper aims at the problem of accuracy reduction of stabilized sight system caused by uncertain factors and disturbances under complex working conditions. Here, a fractional-order PID (FOPID) controller design method based on internal model control (IMC) approach is proposed for stabilized sight system. First, the IMC-PID is constructed by IMC approach, and then the fractional order is added to its differential and integral terms to obtain the FOPID. The traditional FOPID requires five parameters for tuned, while the proposed FOPID has only three tuning parameters, which can be obtained through the CRONE control technology. The optimal rational approximation algorithm (ORAA) with a high degree of fit is used to approximate the fractional-order calculus operator. Matlab simulation experiments show that the proposed controller outperforms other controllers and can obtain good robustness, disturbance rejection and tracking performance.

Optimal approximation of asymmetric type fractional‐order bandpass Butterworth filter using decomposition technique

SummaryIn this letter, the optimal rational approximation of fractional‐order bandpass Butterworth filter (FOBBF) is presented. The transfer function of the FOBBF is decomposed into a multiplication of first‐order and second‐order terms. As a result, the design stability conditions can be easily satisfied using only the variable boundary constraints. The proposed technique generalizes the symmetric fractional‐order roll‐off characteristic as only a special case of the asymmetric one. Several examples are presented to validate the modeling efficacy.

$$H^1$$, $$H(\mathrm {curl})$$ and $$H(\mathrm {div})$$ conforming elements on polygon-based prisms and cones

The conation and extrusion techniques were proposed by Bossavit (Math Comput Simul 80:1567–1577, 2010) for constructing $$(m+1)$$ -dimensional Whitney forms on prisms/cones from m-dimensional ones defined on the base shape. We combine the conation and extrusion techniques with the 2D polygonal $$H(\mathrm {div})$$ conforming finite element proposed by Chen and Wang (Math Comput 307:2053–2087, 2017), and construct the lowest-order $$H^1$$ , $$H(\mathrm {curl})$$ and $$H(\mathrm {div})$$ conforming elements on polygon-based prisms and cones. The elements have optimal approximation rates. Despite of the relatively sophisticated theoretical analysis, the construction itself is easy to implement. As an example, we provide a 100-line Matlab code for evaluating the shape functions of $$H^1$$ , $$H(\mathrm {curl})$$ and $$H(\mathrm {div})$$ conforming elements as well as their exterior derivatives on polygon-based cones. Note that all convex and some non-convex 3D polyhedra can be divided into polygon-based cones by connecting the vertices with a chosen interior point. Thus our construction also provides composite elements for all such polyhedra.

Optimal rational approximation of bandpass Butterworth filter with symmetric fractional-order roll-off

In this paper, a fractional-order bandpass Butterworth filter (FBPBF) exhibiting symmetric roll-off characteristic is approximated as an integer-order transfer function using a metaheuristic optimization approach. A novel cost function is proposed by considering the exact FBPBF magnitude characteristic obtained through cascading the (n+α), where n is an integer and 0<α<1, order lowpass and highpass Butterworth filter responses. Design examples considering various values of n and α are presented to illustrate the performance efficacy of the proposed approach. PSPICE simulations for the models realized using the current feedback operational amplifiers confirm proximity to the theoretical characteristics.

Optimal Monte Carlo integration on closed manifolds

The worst case integration error in reproducing kernel Hilbert spaces of standard Monte Carlo methods with n random points decays as n^{-1/2}. However, the re-weighting of random points, as exemplified in the Bayesian Monte Carlo method, can sometimes be used to improve the convergence order. This paper contributes general theoretical results for Sobolev spaces on closed Riemannian manifolds, where we verify that such re-weighting yields optimal approximation rates up to a logarithmic factor. We also provide numerical experiments matching the theoretical results for some Sobolev spaces on the sphere {mathbb {S}}^2 and on the Grassmannian manifold {mathcal {G}}_{2,4}. Our theoretical findings also cover function spaces on more general sets such as the unit ball, the cube, and the simplex.

Revisiting the use of squared magnitude function for the optimal approximation of (1 + α)-order Butterworth filter

Optimal rational approximation of the fractional-order Butterworth filter (FBF) based on a two-step design procedure is proposed. Firstly, the coefficients of the squared magnitude function of an approximant which matches the squared magnitude response of the ideal (1 + α)-order FBF, where 0<α<1, are determined using the Genetic Algorithm (GA). Then, the stable model is used as an initial point for Powell’s conjugate direction algorithm (PCDA). The computational efficiency and the robustness of the suggested strategy are justified using illustrative examples. The proposed designs show a marked improvement in solution quality compared to the state-of-the-art. PSPICE responses for the FBFs realized using current feedback operational amplifiers (CFOA) confirm a close match with the theoretical characteristic. Python code for implementing the proposed designs using the Powell’s method is also provided.

Optimal approximation of piecewise smooth functions using deep ReLU neural networks

We study the necessary and sufficient complexity of ReLU neural networks – in terms of depth and number of weights – which is required for approximating classifier functions in an Lp-sense.As a model class, we consider the set Eβ(Rd) of possibly discontinuous piecewise Cβ functions f:[−12,12]d→R, where the different “smooth regions” of f are separated by Cβ hypersurfaces. For given dimension d≥2, regularity β>0, and accuracy ε>0, we construct artificial neural networks with ReLU activation function that approximate functions from Eβ(Rd) up to an L2 error of ε. The constructed networks have a fixed number of layers, depending only on d and β, and they have O(ε−2(d−1)∕β) many nonzero weights, which we prove to be optimal. For the proof of optimality, we establish a lower bound on the description complexity of the class Eβ(Rd). By showing that a family of approximating neural networks gives rise to an encoder for Eβ(Rd), we then prove that one cannot approximate a general function f∈Eβ(Rd) using neural networks that are less complex than those produced by our construction.In addition to the optimality in terms of the number of weights, we show that in order to achieve this optimal approximation rate, one needs ReLU networks of a certain minimal depth. Precisely, for piecewise Cβ(Rd) functions, this minimal depth is given – up to a multiplicative constant – by β∕d. Up to a log factor, our constructed networks match this bound. This partly explains the benefits of depth for ReLU networks by showing that deep networks are necessary to achieve efficient approximation of (piecewise) smooth functions.Finally, we analyze approximation in high-dimensional spaces where the function f to be approximated can be factorized into a smooth dimension reducing feature map τ and classifier function g – defined on a low-dimensional feature space – as f=g∘τ. We show that in this case the approximation rate depends only on the dimension of the feature space and not the input dimension.