Continuous Piecewise Linear Functions Research Articles

Spurious Local Minima are Common for Deep Neural Networks With Piecewise Linear Activations.

In this article, theoretically, it is shown that spurious local minima are common for deep fully connected networks and average-pooling convolutional neural networks (CNNs) with piecewise linear activations and datasets that cannot be fit by linear models. Motivating examples are given to explain why spurious local minima exist: each output neuron of deep fully connected networks and CNNs with piecewise linear activations produces a continuous piecewise linear (CPWL) function, and different pieces of the CPWL output can optimally fit disjoint groups of data samples when minimizing the empirical risk. Fitting data samples with different CPWL functions usually results in different levels of empirical risk, leading to the prevalence of spurious local minima. The results are proved in general settings with arbitrary continuous loss functions and general piecewise linear activations. The main proof technique is to represent a CPWL function as maximization over minimization of linear pieces. Deep networks with piecewise linear activations are then constructed to produce these linear pieces and implement the maximization over minimization operation.

Fixed-Time Adaptive Parameter Estimation for Hammerstein Systems Subject to Dead-Zone

The Hammerstein model has proven its ability for modeling many industrial servo systems, but the corresponding parameter estimation remains as a challenging problem, especially for the purpose of achieving fast convergence. This paper presents a fixed-time (FxT) adaptive parameter estimation scheme for Hammerstein systems with an asymmetric dead-zone to ensure the convergence of estimated parameters in fixed-time. A continuous piecewise linear (CPL) function is adopted to model the nonlinear dead-zone dynamics to remedy the use of intermediate variable. To avoid using the system state information, a K-filter is used to construct a relationship between the unknown system parameters and the input/output signals. Then a new adaptive law based on the parameter error and the FxT scheme is developed to online estimate the lumped unknown parameters and ensure the fixed-time convergence, where an online updated auxiliary variable of regressor is suggested to eliminate the impact of the regressor on the convergence performance. Both numerical simulations and experimental results are given to demonstrate the effectiveness of proposed methods.

Chance-Constrained Optimization for a Green Multimodal Routing Problem with Soft Time Window under Twofold Uncertainty

This study investigates a green multimodal routing problem with soft time window. The objective of routing is to minimize the total costs of accomplishing the multimodal transportation of a batch of goods. To improve the feasibility of optimization, this study formulates the routing problem in an uncertain environment where the capacities and carbon emission factors of the travel process and the transfer process in the multimodal network are considered fuzzy. Taking triangular fuzzy numbers to describe the uncertainty, this study proposes a fuzzy nonlinear programming model to deal with the specific routing problem. To make the problem solvable, this study adopts the fuzzy chance-constrained programming approach based on the possibility measure to remove the fuzziness of the proposed model. Furthermore, we use linear inequality constraints to reformulate the nonlinear equality constraints represented by the continuous piecewise linear functions and realize the linearization of the nonlinear programming model to improve the computational efficiency of problem solving. After model processing, we can utilize mathematical programming software to run exact solution algorithms to solve the specific routing problem. A numerical experiment is given to show the feasibility of the proposed model. The sensitivity analysis of the numerical experiment further clarifies how improving the confidence level of the chance constraints to enhance the possibility that the multimodal route planned in advance satisfies the real-time capacity constraint in the actual transportation, i.e., the reliability of the routing, increases both the total costs and carbon emissions of the route. The numerical experiment also finds that charging carbon emissions is not absolutely effective in emission reduction. In this condition, bi-objective analysis indicates the conflicting relationship between lowering transportation activity costs and reducing carbon emissions in routing optimization. The sensitivity of the Pareto solutions concerning the confidence level reveals that reliability, economy, and environmental sustainability are in conflict with each other. Based on the findings of this study, the customer and the multimodal transport operator can organize efficient multimodal transportation, balancing the above objectives using the proposed model.

Efficient P1-FEM for Any Space Dimension in Matlab

Abstract This paper deals with the efficient implementation of the finite element method with continuous piecewise linear functions (P1-FEM) in R d \mathbb{R}^{d} ( d ∈ N d\in\mathbb{N} ). Although at present there does not seem to be a very high practical demand for finite element methods that use higher-dimensional simplicial partitions, there are some advantages in studying the efficient implementation of the method independent of the dimension. For instance, it provides additional insights into necessary data structures and the complexity of implementations. Throughout, the focus is on an efficient realization using Matlab built-in functions and vectorization. The fast and vectorized Matlab function can be easily implemented in many other vector languages and is provided in Julia, too. The complete implementation of the adaptive FEM is given, including assembling stiffness matrix, building load vector, error estimation, and adaptive mesh-refinement. Numerical experiments underline the efficiency of our freely available code which is observed to be of a slightly more than linear complexity with respect to the number of elements when memory limits are not exceeded.

On the enriched mixed Galerkin method with θ scheme for the elastic wave equation

In this paper, we propose the application of an enriched Galerkin (EG) method to solve the elastic wave equation in the “displacement pseudo-pressure” mixed form, which is known for being locking-free. Unlike the continuous Galerkin (CG) method and the discontinuous Galerkin (DG) method, the EG method is based on the weak form of the DG method, and the enriched space combines a continuous piecewise linear vector-valued function space with some discontinuous piecewise linear functions. Specifically, we employ the enriched space to discretize the displacement, while the pseudo-pressure is discretized using the P0 element space. As a result, compared to the DG method, the EG method significantly reduces the number of degrees of freedom. Furthermore, we adopt a more general θ-scheme for time discretization, which behaves as an implicit scheme for 14≤θ≤1 and as an explicit scheme for 0≤θ<14. We establish error estimates for both the semi-discrete and fully-discrete schemes. The accuracy of the proposed method is validated through numerical examples, which not only confirm the theoretical analysis but also provide simulations of elastic wave propagation.

A new analytical method for estimating hydraulic parameters of a nonlinear consolidated aquitard with variable drawdown in adjacent aquifers

Estimation of hydraulic parameters and groundwater depletion of aquitards is necessary to evaluate the groundwater resources and land subsidence. In this study, an analytical solution for water release from a nonlinear consolidated aquitard was derived by approximating the drawdown history as a continuous piecewise linear drawdown function (PLD) with several stages, while the water level in adjacent aquifers varies arbitrarily over time. According to dimensionless and parameter sensitivity analyses, groundwater depletion of an aquitard increases with an increase in compression index, aquitard thickness, and water drawdown rate in adjacent aquifers, decreases with an increase in initial void ratio and effective stress, and is not affected by the consolidation coefficient. In addition, we proposed a type-curve fitting method to calculate the hydraulic parameters of the aquitard under the condition of PLD in adjacent aquifers. The proposed method was applied to estimate the hydraulic parameters of aquitards at two field sites in the Yangtze River Delta in China, i.e., Changzhou and Shanghai. Results show that the compression index of the aquitards in Changzhou and Shanghai are 0.074 and 0.24, respectively, while the consolidation coefficients are 7.62 × 10-7 m2/s and 2.16 × 10-6 m2/s, respectively. The hydraulic conductivity and specific storage estimated by the proposed method are in good agreement with previous studies and the results of the geological method of Konikow and Neuzil (2007). The proposed analytical method for estimating hydraulic parameters and groundwater depletion of aquitards will contribute to the sustainable management of groundwater resources in multi-layer aquifer systems.

Piecewise linear functions representable with infinite width shallow ReLU neural networks

This paper analyzes representations of continuous piecewise linear functions with infinite width, finite cost shallow neural networks using the rectified linear unit (ReLU) as an activation function. Through its integral representation, a shallow neural network can be identified by the corresponding signed, finite measure on an appropriate parameter space. We map these measures on the parameter space to measures on the projective n n -sphere cross R \mathbb {R} , allowing points in the parameter space to be bijectively mapped to hyperplanes in the domain of the function. We prove a conjecture of Ongie et al. [A Function Space View of Bounded Norm Infinite Width ReLU Nets: The Multivariate Case, arXiv, 2019] that every continuous piecewise linear function expressible with this kind of infinite width neural network is expressible as a finite width shallow ReLU neural network.

Fractal dimensions of continuous piecewise linear iterated function systems

We consider iterated function systems on the real line that consist of continuous, piecewise linear functions. Under a mild separation condition, we show that the Hausdorff and box dimensions of the attractor are equal to the minimum of 1 and the exponent which comes from the most natural system of covers of the attractor.

De Rham compatible Deep Neural Network FEM

On general regular simplicial partitions T of bounded polytopal domains Ω⊂Rd, d∈{2,3}, we construct exact neural network (NN) emulations of all lowest order finite element spaces in the discrete de Rham complex. These include the spaces of piecewise constant functions, continuous piecewise linear (CPwL) functions, the classical “Raviart–Thomas element”, and the “Nédélec edge element”. For all but the CPwL case, our network architectures employ both ReLU (rectified linear unit) and BiSU (binary step unit) activations to capture discontinuities. In the important case of CPwL functions, we prove that it suffices to work with pure ReLU nets. Our construction and DNN architecture generalizes previous results in that no geometric restrictions on the regular simplicial partitions T of Ω are required for DNN emulation. In addition, for CPwL functions our DNN construction is valid in any dimension d≥2. Our “FE-Nets” are required in the variationally correct, structure-preserving approximation of boundary value problems of electromagnetism in nonconvex polyhedra Ω⊂R3. They are thus an essential ingredient in the application of e.g., the methodology of “physics-informed NNs” or “deep Ritz methods” to electromagnetic field simulation via deep learning techniques. We indicate generalizations of our constructions to higher-order compatible spaces and other, non-compatible classes of discretizations, in particular the “Crouzeix–Raviart” elements and Hybridized, Higher Order (HHO) methods.

Rate-independent Computation in Continuous Chemical Reaction Networks

Understanding the algorithmic behaviors that are in principle realizable in a chemical system is necessary for a rigorous understanding of the design principles of biological regulatory networks. Further, advances in synthetic biology herald the time when we will be able to rationally engineer complex chemical systems and when idealized formal models will become blueprints for engineering. Coupled chemical interactions in a well-mixed solution are commonly formalized as chemical reaction networks (CRNs). However, despite the widespread use of CRNs in the natural sciences, the range of computational behaviors exhibited by CRNs is not well understood. Here, we study the following problem: What functions f : ℝ k → ℝ can be computed by a CRN, in which the CRN eventually produces the correct amount of the “output” molecule, no matter the rate at which reactions proceed? This captures a previously unexplored but very natural class of computations: For example, the reaction X 1 + X 2 → Y can be thought to compute the function y = min ( x 1 , x 2 ). Such a CRN is robust in the sense that it is correct whether its evolution is governed by the standard model of mass-action kinetics, alternatives such as Hill-function or Michaelis-Menten kinetics, or other arbitrary models of chemistry that respect the (fundamentally digital) stoichiometric constraints (what are the reactants and products?). We develop a reachability relation based on a broad notion of “what could happen” if reaction rates can vary arbitrarily over time. Using reachability, we define stable computation analogously to probability 1 computation in distributed computing and connect it with a seemingly stronger notion of rate-independent computation based on convergence in the limit t → ∞ under a wide class of generalized rate laws. Besides the direct mapping of a concentration to a nonnegative analog value, we also consider the “dual-rail representation” that can represent negative values as the difference of two concentrations and allows the composition of CRN modules. We prove that a function is rate-independently computable if and only if it is piecewise linear (with rational coefficients) and continuous (dual-rail representation), or non-negative with discontinuities occurring only when some inputs switch from zero to positive (direct representation). The many contexts where continuous piecewise linear functions are powerful targets for implementation, combined with the systematic construction we develop for computing these functions, demonstrate the potential of rate-independent chemical computation.

Accuracy Estimates for Bilinear Optimal Control Problems Governed by Ordinary Differential Equations

First- and second-order accuracy estimates for an optimal control problem governed by a system of ordinary differential equations with a bilinear control mechanism are presented. The numerical time discretization scheme under consideration is the finite element method with continuous piecewise linear functions. Central to this work is a first- and second-order analysis of optimality of the continuous and approximated optimal control problems. In the case of box constraints on the control, first-order error estimates for the control function are obtained assuming a piecewise constant approximation of the control, whereas second-order accuracy can be obtained in the case of a continuous, piecewise polynomial approximation. Numerical evidence is presented that supports the theoretical findings.

Adaptive VEM: Stabilization-Free A Posteriori Error Analysis and Contraction Property

In the present paper we initiate the challenging task of building a mathematically sound theory for adaptive virtual element methods (AVEMs). Among the realm of polygonal meshes, we restrict our analysis to triangular meshes with hanging nodes in two dimensions—the simplest meshes with a systematic refinement procedure that preserves shape regularity and optimal complexity. A major challenge in the a posteriori error analysis of AVEMs is the presence of the stabilization term, which is of the same order as the residual-type error estimator but prevents the equivalence of the latter with the energy error. Under the assumption that any chain of recursively created hanging nodes has uniformly bounded length, we show that the stabilization term can be made arbitrarily small relative to the error estimator provided the stabilization parameter of the scheme is sufficiently large. This quantitative estimate leads to stabilization-free upper and lower a posteriori bounds for the energy error. This novel and crucial property of VEMs hinges on the largest subspace of continuous piecewise linear functions and the delicate interplay between its coarser scales and the finer ones of the VEM space. An important consequence for piecewise constant data is a contraction property between consecutive loops of AVEMs, which we also prove. Our results apply to -conforming (lowest order) VEMs of any kind, including the classical and enhanced VEMs.

Constructive Approximation on Graded Meshes for the Integral Fractional Laplacian

We consider the homogeneous Dirichlet problem for the integral fractional Laplacian $$(-\Delta )^s$$ . We prove optimal Sobolev regularity estimates in Lipschitz domains provided the solution is $$C^s$$ up to the boundary. We present the construction of graded bisection meshes by a greedy algorithm and derive quasi-optimal convergence rates for approximations to the solution of such a problem by continuous piecewise linear functions. The nonlinear Sobolev scale dictates the relation between regularity and approximability.

Сравнение способов аппроксимации функции множителей Лагранжа при решении контактных задач методом с независимой границей контакта

The paper considers the contact problem of the elasticity theory in a static spatial two-dimensional formulation without considering friction. For discretization of the elasticity theory equations, the finite element method was introduced using a triangular unstructured grid and linear and quadratic basis functions. To account for the contact boundary conditions, a modified method of Lagrange multipliers with independent contact boundary is proposed. This method implies the ability to construct a contact boundary with the smoothness degree required for the solution precision and to execute approximation of the Lagrange multiplier function independent of the grids inside the contacting bodies. Various types of the Lagrange multiplier function approximations were studied, including piecewise constant, continuous piecewise linear functions and piecewise linear functions with discontinuities at the difference cells boundaries. Examples of test calculations are provided both for problems with rectilinear and curvilinear contact boundaries. In both cases, the use of discontinuous approximations of the Lagrange multiplier function makes it possible to obtain a numerical solution with fewer artificial oscillations and higher rate of convergence at the grid refinement. It is shown that the numerical solution precision could be improved by more detailed discretization of the contact boundary without changing the grids inside the contacting bodies

Approximation in AC(sigma )

For a nonempty compact subset sigma in the plane, the space AC(sigma ) is the closure of the space of complex polynomials in two real variables under a particular variation norm. In the classical setting, AC[0, 1] contains several other useful dense subsets, such as continuous piecewise linear functions, C^1 functions and Lipschitz functions. In this paper, we examine analogues of these results in this more general setting.

An iterative constrained least squares method for continuous piecewise linear approximation

In this work, a novel iterative least squares method is proposed to approximate nonlinear functions with continuous piecewise linear (CPWL) functions, where the continuity is guaranteed by constrained least squares. A typical least-squares-based method for CPWL fitting consists of two major steps: to perform least squares for a fixed set of breakpoints (i.e. for a specific partition of function domain), and to update breakpoints to reduce the overall fitting error. The proposed method aims to improve the existing CPWL method, the one with canonical representation, by modifying both major steps. Instead of performing ordinary least squares with canonical representation, partitioned least squares and constrained least squares are successively performed to reduce the computational complexity. For the update of breakpoints, an iterative procedure is proposed, where gradient descent with momentum method is employed to improve the convergence characteristics. The advantages of proposed method are illustrated using illustrative examples.

Generating optimal robust continuous piecewise linear regression with outliers through combinatorial Benders decomposition

Using piecewise linear (PWL) functions to model discrete data has applications for example in healthcare, engineering and pattern recognition. Recently, mixed-integer linear programming (MILP) approaches have been used to optimally fit continuous PWL functions. We extend these formulations to allow for outliers. The resulting MILP models rely on binary variables and big- constructs to model logical implications. The combinatorial Benders decomposition (CBD) approach removes the dependency on the big- constraints by separating the MILP model into a master problem of the complicating binary variables and a linear sub problem over the continuous variables, which feeds combinatorial solution information into the master problem. We use the CBD approach to decompose the proposed MILP model and solve for optimal PWL functions. Computational results show that vast speedups can be found using this robust approach, with problem-specific improvements including smart initialization, strong cut generation and special branching approaches leading to even faster solve times, up to more than 12,000 times faster than the standard MILP approach.

An Embedding of ReLU Networks and an Analysis of Their Identifiability

Neural networks with the rectified linear unit (ReLU) nonlinearity are described by a vector of parameters $$\theta $$ and realized as a piecewise linear continuous function $$\varvec{R}_{\theta }: x \in \mathbb {R}^{d} \mapsto \varvec{R}_{\theta }(x) \in \mathbb {R}^{k}$$ . Natural scalings and permutations operations on the parameters $$\theta $$ leave the realization unchanged, leading to equivalence classes of parameters that yield the same realization. These considerations in turn lead to the notion of identifiability—the ability to recover (the equivalence class of) $$\theta $$ from the sole knowledge of its realization $$\varvec{R}_{\theta }$$ . The overall objective of this paper is to introduce an embedding for ReLU neural networks of any depth, $${\varvec{\Phi }}(\theta )$$ , that is invariant to scalings and that provides a locally linear parameterization of the realization of the network. Leveraging these two key properties, we derive some conditions under which a deep ReLU network is indeed locally identifiable from the knowledge of the realization on a finite set of samples $$x_{i} \in \mathbb {R}^{d}$$ . We study the shallow case in more depth, establishing necessary and sufficient conditions for the network to be identifiable from a bounded subset $$\mathcal {X}\subseteq \mathbb {R}^{d}$$ .

Error estimates for the numerical approximation of optimal control problems with nonsmooth pointwise-integral control constraints

Abstract The numerical approximation of an optimal control problem governed by a semilinear parabolic equation and constrained by a bound on the spatial $L^1$-norm of the control at every instant of time is studied. Spatial discretizations of the controls by piecewise constant and continuous piecewise linear functions are investigated. Under finite element approximations, the sparsity properties of the continuous solutions are preserved in a natural way using piecewise constant approximations of the control, but suitable numerical integration of the objective functional and of the constraint must be used to keep the sparsity pattern when using spatially continuous piecewise linear approximations. We also obtain error estimates and finally present some numerical examples.

Neural Network Approaches to Point Lattice Decoding

We characterize the complexity of the lattice decoding problem from a neural network perspective. The notion of Voronoi-reduced basis is introduced to restrict the space of solutions to a binary set. On the one hand, this problem is shown to be equivalent to computing a continuous piecewise linear (CPWL) function restricted to the fundamental parallelotope. On the other hand, it is known that any function computed by a ReLU feed-forward neural network is CPWL. As a result, we count the number of affine pieces in the CPWL decoding function to characterize the complexity of the decoding problem. It is exponential in the space dimension <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$n$ </tex-math></inline-formula> , which induces shallow neural networks of exponential size. For structured lattices we show that folding, a technique equivalent to using a deep neural network, enables to reduce this complexity from exponential in <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$n$ </tex-math></inline-formula> to polynomial in <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$n$ </tex-math></inline-formula> . Regarding unstructured MIMO lattices, in contrary to dense lattices many pieces in the CPWL decoding function can be neglected for quasi-optimal decoding on the Gaussian channel. This makes the decoding problem easier and it explains why shallow neural networks of reasonable size are more efficient with this category of lattices (in low to moderate dimensions).