Nonsmooth Functions Research Articles

Shaped pattern synthesis for hybrid analog–digital arrays via manifold optimization-enabled block coordinate descent

Hybrid analog–digital (HAD) architecture is a promising means to realize large-scale arrays owing to the judicious trade-off between system performance and hardware complexity. This paper investigates the beampattern shaping of HAD arrays through minimizing the beampattern matching error between desired and actual patterns. Due to the nonconvex constraints on analog and digital weights and the nonconvex nonsmooth objective function, the resultant codesign is NP-hard. To address this issue, we create an efficient algorithm by leveraging block coordinate descent (BCD) and Riemannian manifold optimization. We first equivalently represent the objective function as a smooth bi-quadratic function by introducing a uni-modulus auxiliary variable. Subsequently, we alternatingly optimize the scaling factor, digital weights, analog weights and auxiliary variable under the BCD framework, where the simultaneous update of analog weights and auxiliary variable is recast as uni-modular constrained quadratic programming which is efficiently solved by Riemannian Newton method, and the digital weights have a global optimal solution via Lagrangian multiplier method. We also derive an explicit convergence condition of this algorithm. Numerical results demonstrate that the proposed algorithm has superior performance and faster convergence speed than alternative algorithms, and produces nearly the same mainlobe gain as fully digital arrays with significantly fewer radio-frequency chains.

Deep learning as Ricci flow.

Deep neural networks (DNNs) are powerful tools for approximating the distribution of complex data. It is known that data passing through a trained DNN classifier undergoes a series of geometric and topological simplifications. While some progress has been made toward understanding these transformations in neural networks with smooth activation functions, an understanding in the more general setting of non-smooth activation functions, such as the rectified linear unit (ReLU), which tend to perform better, is required. Here we propose that the geometric transformations performed by DNNs during classification tasks have parallels to those expected under Hamilton's Ricci flow-a tool from differential geometry that evolves a manifold by smoothing its curvature, in order to identify its topology. To illustrate this idea, we present a computational framework to quantify the geometric changes that occur as data passes through successive layers of a DNN, and use this framework to motivate a notion of 'global Ricci network flow' that can be used to assess a DNN's ability to disentangle complex data geometries to solve classification problems. By training more than 1500 DNN classifiers of different widths and depths on synthetic and real-world data, we show that the strength of global Ricci network flow-like behaviour correlates with accuracy for well-trained DNNs, independently of depth, width and data set. Our findings motivate the use of tools from differential and discrete geometry to the problem of explainability in deep learning.

Level sets of nonsmooth functions, Part 2: Lipschitz and piecewise-differentiable manifolds

This paper introduces piecewise-differentiable (PCr) manifolds according to a unified general framework that also applies to nonsmooth Lipschitz manifolds and smooth manifolds. We present definitions of nonsmooth manifolds and embedded submanifolds for abstract sets as well as for subsets of Rn, and explore the relationships between them. The PCr and Lipschitz Rank Theorems from Part 1 of this series, in terms of the Clarke Jacobian and B-subdifferential generalized derivative sets, are used to characterize level sets of functions between nonsmooth manifolds as embedded submanifolds. We illustrate how the Level Set Theorems developed in this paper can be applied to functions on Euclidean space, including a piecewise-differentiable process model for distillation columns.

Physics-Inspired Evolutionary Machine Learning Method: From the Schrödinger Equation to an Orbital-Free-DFT Kinetic Energy Functional.

We introduce a machine learning (ML)-supervised model function (which is in fact a functional rather than a regular function) that is inspired by the variational principle of physics. This ML hypothesis evolutionary method, termed ML-Ω, allows us to go from data to differential equation(s) underlying the physical (chemical, engineering, etc.) phenomena from which the data are derived from. The fundamental equations of physics can be derived from this ML-Ω evolutionary method when the proper training data is used. By training the ML-Ω model function with only three hydrogen-like atom energies, the method can find Schrödinger's exact functional and, from it, Schrödinger's fundamental equation. Then, in the field of density functional theory (DFT), when the model function is trained with the energies from the known Thomas-Fermi (TF) formula , it correctly finds the exact TF functional. Finally, the method is applied to find a local orbital-free (OF) functional expression of the independent electron kinetic energy functional Ts based on the γTFλvW model. By considering the theoretical energies of only five atoms (He, Be, Ne, Mg, and Ar) as the training set, the evolutionary ML-Ω method finds an ML-Ω-OF-DFT local Ts functional (γTFλvW(0.964,1/4)) that outperforms all the OF-DFT functionals of a representative group. Moreover, our ML-Ω-OF functional overcomes the difficulty of LDA's and some local generalized gradient approximation (GGA)-DFT's functionals to describe the stretched bond region at the correct spin configuration of diatomic molecules. Nonsmooth and nonclosed form functionals can be considered in the ML-Ω model function and still be effectively trained. Although our evolutionary ML-Ω model function can work without an explicit prior-form functional, by using the techniques of symbolic regression, in this work, we exploit prior-form functional expressions to make the training process simpler and faster. The ML-Ω method can be considered at the intersection of ML and the natural sciences.

Bregman Proximal Linearized ADMM for Minimizing Separable Sums Coupled by a Difference of Functions

AbstractIn this paper, we develop a splitting algorithm incorporating Bregman distances to solve a broad class of linearly constrained composite optimization problems, whose objective function is the separable sum of possibly nonconvex nonsmooth functions and a smooth function, coupled by a difference of functions. This structure encapsulates numerous significant nonconvex and nonsmooth optimization problems in the current literature including the linearly constrained difference-of-convex problems. Relying on the successive linearization and alternating direction method of multipliers (ADMM), the proposed algorithm exhibits the global subsequential convergence to a stationary point of the underlying problem. We also establish the convergence of the full sequence generated by our algorithm under the Kurdyka–Łojasiewicz property and some mild assumptions. The efficiency of the proposed algorithm is tested on a robust principal component analysis problem and a nonconvex optimal power flow problem.

Three Ellipsoids for External Approximation of the Half-Ball

Introduction. Ellipsoids that approximate the half-ball in Rn (n≥2) and their volume is smaller than the volume of the ball can be used to construct algorithms for solving the problem of minimization of a convex (smooth or non-smooth) function, the problem of minimization of a convex function on a sphere, a general problem convex programming, saddle point problems of convex-concave functions and others. The speed of convergence of such algorithms will be determined by the ratio of the volume of the approximating ellipsoid to the volume of the ball. The purpose of the work is to describe properties of three ellipsoids for approximation of a n-dimensional half-ball, the volume reduction factor of which depends only on n - dimension of space. The first is the well-known ellipsoid of minimum volume, which is used in the classical Yudin-Nemirovsky-Shor ellipsoid method. It provides a minimum volume reduction factor and can be used if n≥2. If n=1, then the classical ellipsoid method replaces the dichotomy method. The other two ellipsoids are close to the minimum volume ellipsoid and at large n guarantee volume reduction coefficients close to the minimum. The results. It is shown that when n=1 the first approximate ellipsoid provides a coefficient of reduction of the segment length equal to 2–√2 ≈ 0.5858. The second approximate ellipsoid is new and the volume reduction factor for it is slightly larger than the factor for the first approximate ellipsoid. If n=1, then it provides a coefficient of reduction of the segment length equal to (√5–1)/2 ≈ 0.6180. For three ellipsoids, comparative results are given in terms of volume reduction coefficients (n=1/20) and the number of iterations for solving problems with relative accuracy 1e-10 (n=1/30). Conclusions. A new ellipsoid has been constructed to approximate the n-dimensional half-ball, which is close in volume to the minimal ellipsoid. The volume reduction factor of the proposed ellipsoid is approximated by an asymptotic formula 1–1/(2n)+1/n², which differs little from the formula 1–1/(2n) for an ellipsoid of minimum volume.

A modified inertial proximal minimization algorithm for structured nonconvex and nonsmooth problem

We introduce an enhanced inertial proximal minimization algorithm tailored for a category of structured nonconvex and nonsmooth optimization problems. The objective function in question is an aggregation of a smooth function with an associated linear operator, a nonsmooth function dependent on an independent variable, and a mixed function involving two variables. Throughout the iterative procedure, parameters are selected employing a straightforward approach, and weak inertial terms are incorporated into two subproblems within the update sequence. Under a set of lenient conditions, we demonstrate that the sequence engendered by our algorithm is bounded. Furthermore, we establish the global and strong convergence of the algorithmic sequence, contingent upon the assumption that the principal function adheres to the Kurdyka–Łojasiewicz (KL) property. Ultimately, the numerical outcomes corroborate the algorithm’s feasibility and efficacy.

A quasi-reversibility method for solving nonhomogeneous sideways heat equation

Abstract We consider solving a severely ill-posed problem for determining the surface temperature and heat flux distribution of the nonhomogeneous sideways heat equation by a quasi-reversibility regularization method. An analytical solution is deduced based on the Fourier transform, and then, the logarithmic-type error estimate for the regularized solution is achieved. Finally, three numerical examples, including smooth and non-smooth functions, validate the feasibility and effectiveness of the proposed approach.

Impact of mixed boundary conditions and nonsmooth data on layer‐originated nonpremixed combustion problems: Higher‐order convergence analysis

AbstractThis work explores the theoretical and computational impacts of mixed‐type flux conditions and nonsmooth data on boundary/interior layer‐originated singularly perturbed semilinear reaction–diffusion problems. Such problems are prevalent in nonpremixed combustion models and catalytic reaction models. The inclusion of arbitrarily small diffusion terms results in boundary layers influenced by specific flux conditions normalized by perturbation parameters. We have demonstrated theoretically that the sharpness of the boundary layer is significantly reduced when this normalization is independent of the diffusion parameter. In addition, the presence of a nonsmooth source function gives rise to an interior layer in the current problem. We show that using upwind discretizations for mixed boundary fluxes achieves nearly second‐order accuracy if the first derivatives are not normalized concerning perturbation parameters. This outcome arises from the bounds of a prior derivative of the continuous solution. Furthermore, it is theoretically shown that nearly second‐order uniform accuracy can be attained for reaction‐dominated semilinear problems using a hybrid scheme at the discontinuity point. To ensure the uniform stability of the discrete solution, a transformation is necessary for the corresponding discrete problem. Theoretical results are supported by various experiments on nonlinear problems, illustrating the pointwise rates and highlighting both linear and higher‐order accuracy at specific points.

Convergence analysis of iteratively regularized Landweber iteration with uniformly convex constraints in Banach spaces

In Banach spaces, the convergence analysis of iteratively regularized Landweber iteration (IRLI) is recently studied via conditional stability estimates. But the formulation of IRLI does not include general non-smooth convex penalty functionals, which is essential to capture special characteristics of the sought solution. In this paper, we formulate a generalized form of IRLI so that its formulation includes general non-smooth uniformly convex penalty functionals. We study the convergence analysis and derive the convergence rates of the generalized method solely via conditional stability estimates in Banach spaces for both the perturbed and unperturbed data. We also discuss few examples of inverse problems on which our method is applicable.

Optimality analysis on ℓ0/1 -sparsity regularized nonsmooth optimization

ABSTRACT We consider the problem of minimizing the sum of a nonsmooth function and an ℓ 0 / 1 -sparsity function, which is called the ℓ 0 / 1 -sparsity regularized nonsmooth optimization ( ℓ 0 / 1 -RNO). In this paper, by defining and analyzing two classes of stationary points, we present the first-order necessary and sufficient optimality conditions for ℓ 0 / 1 -RNO. Moreover, applying these optimality results on ℓ 0 / 1 -RNO, we derive the first-order necessary and sufficient optimality conditions for both the nonsmooth M-estimation of linear regression with positive variable sparsity, and ℓ 1 -penalty minimum of 0/1 composite optimization. Especially, we obtain that the given ℓ 1 -penalty is strongly exact penalty of 0/1 composite optimization.

Non-linear filtration model with splitting front

We consider filtration of a suspension in a homogeneous porous medium which is described by a macroscopic 1-D model including the mass exchange equation and the kinetic equation for deposit growth. The standard model assumes that suspended particles move at the same speed as the carrier fluid. However, in some experiments, a lag between the front boundary of suspended particles and the front of the carrier fluid was observed. This article proposes a modification of the standard model that provides a description of the separation between the fluid front and the particle front. For this purpose, a non-smooth non-linear function depending on the concentration of the suspension is introduced into the deposit growth equation. In case of a non-smooth suspension function, the concentration of suspended particles at the front decreases to zero in a finite time. At this moment the united front splits into the front of pure injected water and the front of suspended and retained particles. The particle front moves slower than the pure water front. In case of a linear filtration function (Langmuir coefficient), an exact solution is constructed in a closed form. For the filtration problem with a suspension function in the form of a square root, explicit analytical formulae are obtained. In case of a non-smooth filtration function, the filtration time is finite. The curvilinear boundary separates the filtration domain, where concentrations increase with time, from the stabilization domain, where the concentrations of suspended and retained particles have reached their limits. The limit speeds of the stabilization border and of the particle front coincide.

A Markov random field model for change points detection

Detecting Change Points (CPs) in data sequences is a challenging problem that arises in a variety of disciplines, including signal processing and time series analysis. While many methods exist for PieceWise Constant (PWC) signals, relatively fewer address PieceWise Linear (PWL) signals due to the challenge of preserving sharp transitions. This paper introduces a Markov Random Field (MRF) model for detecting changes in slope. The number of CPs and their locations are unknown. The proposed method incorporates PWL prior information using MRF framework with an additional boolean variable called Line Process (LP), describing the presence or absence of CPs. The solution is then estimated in the sense of maximum a posteriori. The LP allows us to define a non-convex non-smooth energy function that is algorithmically hard to minimize. To tackle the optimization challenge, we propose an extension of the combinatorial algorithm DPS, initially designed for CP detection in PWC signals. Also, we present a shared memory implementation to enhance computational efficiency. Numerical studies show that the proposed model produces competitive results compared to the state-of-the-art methods. We further evaluate the performance of our method on three real datasets, demonstrating superior and accurate estimates of the underlying trend compared to competing methods.

A modified inertial proximal alternating direction method of multipliers with dual-relaxed term for structured nonconvex and nonsmooth problem

In this research, we introduce a novel optimization algorithm termed the dual-relaxed inertial alternating direction method of multipliers (DR-IADM), tailored for handling nonconvex and nonsmooth problems. These problems are characterized by an objective function that is a composite of three elements: a smooth composite function combined with a linear operator, a nonsmooth function, and a mixed function of two variables. To facilitate the iterative process, we adopt a straightforward parameter selection approach, integrate inertial components within each subproblem, and introduce two relaxed terms to refine the dual variable update step. Within a set of reasonable assumptions, we establish the boundedness of the sequence generated by our DR-IADM algorithm. Furthermore, leveraging the Kurdyka–Łojasiewicz (KŁ) property, we demonstrate the global convergence of the proposed method. To validate the practicality and efficacy of our algorithm, we present numerical experiments that corroborate its performance. In summary, our contribution lies in proposing DR-IADM for a specific class of optimization problems, proving its convergence properties, and supporting the theoretical claims with numerical evidence.

Analysis of a Class of Minimization Problems Lacking Lower Semicontinuity

The minimization of nonlower semicontinuous functions is a difficult topic that has been minimally studied. Among such functions is a Heaviside composite function that is the composition of a Heaviside function with a possibly nonsmooth multivariate function. Unifying a statistical estimation problem with hierarchical selection of variables and a sample average approximation of composite chance constrained stochastic programs, a Heaviside composite optimization problem is one whose objective and constraints are defined by sums of possibly nonlinear multiples of such composite functions. Via a pulled-out formulation, a pseudostationarity concept for a feasible point was introduced in an earlier work as a necessary condition for a local minimizer of a Heaviside composite optimization problem. The present paper extends this previous study in several directions: (a) showing that pseudostationarity is implied by (and thus, weaker than) a sharper subdifferential-based stationarity condition that we term epistationarity; (b) introducing a set-theoretic sufficient condition, which we term a local convexity-like property, under which an epistationary point of a possibly nonlower semicontinuous optimization problem is a local minimizer; (c) providing several classes of Heaviside composite functions satisfying this local convexity-like property; (d) extending the epigraphical formulation of a nonnegative multiple of a Heaviside composite function to a lifted formulation for arbitrarily signed multiples of the Heaviside composite function, based on which we show that an epistationary solution of the given Heaviside composite program with broad classes of B-differentiable component functions can in principle be approximately computed by surrogation methods. Funding: The work of Y. Cui was based on research supported by the National Science Foundation [Grants CCF-2153352, DMS-2309729, and CCF-2416172] and the National Institutes of Health [Grant 1R01CA287413-01]. The work of J.-S. Pang was based on research supported by the Air Force Office of Scientific Research [Grant FA9550-22-1-0045].

Second Order Dynamics Featuring Tikhonov Regularization and Time Scaling

In a Hilbert setting we aim to study a second order in time differential equation, combining viscous and Hessian-driven damping, containing a time scaling parameter function and a Tikhonov regularization term. The dynamical system is related to the problem of minimization of a nonsmooth convex function. In the formulation of the problem as well as in our analysis we use the Moreau envelope of the objective function and its gradient and heavily rely on their properties. We show that there is a setting where the newly introduced system preserves and even improves the well-known fast convergence properties of the function and Moreau envelope along the trajectories and also of the gradient of Moreau envelope due to the presence of time scaling. Moreover, in a different setting we prove strong convergence of the trajectories to the element of minimal norm from the set of all minimizers of the objective. The manuscript concludes with various numerical results.

A stochastic primal–dual algorithm for composite constrained optimization

This paper studies the decentralized stochastic optimization problem over an undirected network, where each agent owns its local private functions made up of two non-smooth functions and an expectation-valued function. A decentralized stochastic primal–dual algorithm is proposed, by combining the variance-reduced method and the stochastic approximation method. The local gradients are estimated by using the mean of a variable number of sample gradients and the stochastic error decreases with the number of samples in the stochastic approximation process. The highlight of this paper is the extension of the primal–dual algorithm to the stochastic optimization problems. The effectiveness of the proposed algorithm and the correctness of the theory are verified by numerical experiments.

A NEW CONVERGENCE ANALYSIS FOR MINIMUM NORM SOLUTION OF SPLIT SYSTEM OF NONSMOOTH MINIMIZATION PROBLEMS

This work presents a novel self-adaptive steepest-descent type algorithm for solving the split system of minimization problem (SSMP) related to convex nonsmooth functions. The algorithm includes a self-adaptive step size mechanism, which uses a step size that does not need prior information about the operator norm. Under certain weakened assumptions of parameters, a strong convergence theorem is established and proved for the algorithm. Specifically, the sequence generated by this new algorithm strongly converges towards the minimum norm element of the SSPM. To assess the implementation of our algorithm, a numerical example is provided. According to the numerical results, our algorithm showcases effectiveness and simplicity in its implementation. Furthermore, the primary numerical experiment results indicate that our proposed algorithm surpasses some existing results in the literature in terms of CPU time and iteration count. Our result represents an extension and enhancement of recent findings in this area.

Piecewise nonlinear approximation for non-smooth functions

Piecewise affine or linear approximation has garnered significant attention as a technique for approximating piecewise-smooth functions. In this study, we propose a novel approach: piecewise non-linear approximation based on rational approximation, aimed at approximating non-smooth functions. We introduce a method termed piecewise Padé Chebyshev (PiPC) tailored for approximating univariate piecewise smooth functions. Our investigation focuses on assessing the effectiveness of PiPC in mitigating the Gibbs phenomenon during the approximation of piecewise smooth functions. Additionally, we provide error estimates and convergence results of PiPC for non-smooth functions. Notably, our technique excels in capturing singularities, if present, within the function with minimal Gibbs oscillations, without necessitating the explicit specification of singularity locations. To the best of our knowledge, prior research has not explored the use of piecewise non-linear approximation for approximating non-smooth functions. Finally, we validate the efficacy of our methods through numerical experiments, employing PiPC to reconstruct a non-trivial non-smooth function, thus demonstrating its capability to significantly alleviate the Gibbs phenomenon.

On Mean-Optimal Robust Linear Discriminant Analysis

Linear discriminant analysis (LDA) is widely used for dimensionality reduction under supervised learning settings. Traditional LDA objective aims to minimize the ratio of the squared Euclidean distances that may not perform optimally on noisy datasets. Multiple robust LDA objectives have been proposed to address this problem, but their implementations have two major limitations. One is that their mean calculations use the squared \(\ell_{2}\) -norm distance to center the data, which is not valid when the objective depends on other distance functions. The second problem is that there is no generalized optimization algorithm to solve different robust LDA objectives. In addition, most existing algorithms can only guarantee that the solution is locally optimal rather than globally optimal. In this article, we review multiple robust loss functions and propose a new and generalized robust objective for LDA. Besides, to remove the mean value within data better, our objective uses an optimal way to center the data through learning. As one important algorithmic contribution, we derive an efficient iterative algorithm to optimize the resulting non-smooth and non-convex objective function. We theoretically prove that our solution algorithm guarantees that both the objective and the solution sequences converge to globally optimal solutions at a sub-linear convergence rate. The results of comprehensive experimental evaluations demonstrate the effectiveness of our new method, achieving significant improvements compared to the other competing methods.