Nonconvex Objective Function Research Articles

Jointly Modeling and Clustering Tensors in High Dimensions

Tackling High-Dimensional Tensor Clustering In the paper “Jointly Modeling and Clustering Tensors in High Dimensions,” Cai, Zhang, and Sun address the challenge of jointly modeling and clustering tensors by introducing a high-dimensional tensor mixture model with heterogeneous covariances. The proposed mixture model exploits the intrinsic structures of tensor data. The authors develop a computationally efficient high-dimensional expectation conditional maximization (HECM) algorithm and show that the HECM iterates, with an appropriate initialization, converge geometrically to a neighborhood that is within statistical precision of the true parameter. The theoretical analysis is nontrivial because of the dual nonconvexity arising from both the expectation maximization-type estimation and the nonconvex objective function in the M step. They also study the convergence rate of the algorithm when the number of clusters is overspecified and when the signal-to-noise ratio diminishes with sample size. The efficacy of the proposed method is demonstrated through numerical experiments and a real-world medical data application.

An optimization-free approximation Framework for Connected and Automated Vehicles Eco-Trajectory Planning Under limited computing capacity

The trajectory planning problem (TPP) has become increasingly crucial in the research of next-generation transportation systems, but it presents challenges due to the non-linearity its constraints. One specific case within TPP, namely the Eco-trajectory Planning Problem (EPP), poses even greater computational difficulties due to its nonlinear, high-order, and non-convex objective function. This paper proposes an optimization-free framework to address the eco-trajectory planning problem of connected and automated vehicles (CAVs) under a limited computing capacity scenario. The framework consists of an offline module and an online module. In the offline module, an optimal eco-trajectory batch is constructed by solving a sequence of simplified optimization problems to minimize fuel consumption, considering various initial and terminal system states. Each candidate trajectory in the batch yields the lowest fuel consumption subject to a specific travel time from the vehicle entry to the departure from the intersection. In the online module, an approximation framework is proposed, which greatly improves the computational efficiency of planning and only suffers from a limited extent of optimality losses through a batch-based selection process because optimization and calculation are pre-computed in the offline module. Numerical tests are presented and discussed to evaluate the computational quality and efficiency of the optimization-free approximation framework under a mixed-traffic flow environment that incorporates human-driving vehicles (HDV) and connected and automated vehicles (CAVs) with different market penetration rates (MPR) and the stochasticity of HDVs.

Euler Kernel Mapping for Hyperspectral Image Clustering via Self-Paced Learning

Clustering, as a classical unsupervised artificial intelligence technology, is commonly employed for hyperspectral image clustering tasks. However, most existing clustering methods designed for remote sensing tasks aim to solve a non-convex objective function, which can be optimized iteratively, beginning with random initializations. Consequently, during the learning phase of the clustering model, it may easily fall into bad local optimal solutions and finally hurt the clustering performance. Additionally, prevailing approaches often exhibit limitations in capturing the intricate structures inherent in hyperspectral images and are very sensitive to noise and outliers that widely exist in remote sensing data. To address these issues, we proposed a novel Euler kernel mapping for hyperspectral image clustering via self-paced learning (EKM-SPL). EKM-SPL first employs self-paced learning to learn the clustering model in a meaningful order by progressing samples from easy to complex, which can help to remove bad local optimal solutions. Secondly, a probabilistic soft weighting scheme is employed to measure complexity across the data sample, which makes the optimization process more reasonable. Thirdly, in order to more accurately characterize the intricate structure of hyperspectral images, Euler kernel mapping is used to convert the original data into a reproduced kernel Hilbert space, where the nonlinearly inseparable clusters may become linearly separable. Moreover, we innovatively integrate the coordinate descent technique into the optimization algorithm to circumvent the computational inefficiencies and information loss typically associated with conventional kernel methods. Extensive experiments conducted on classic benchmark hyperspectral image datasets illustrate the effectiveness and superiority of our proposed model.

Shaped pattern synthesis for hybrid analog–digital arrays via manifold optimization-enabled block coordinate descent

Hybrid analog–digital (HAD) architecture is a promising means to realize large-scale arrays owing to the judicious trade-off between system performance and hardware complexity. This paper investigates the beampattern shaping of HAD arrays through minimizing the beampattern matching error between desired and actual patterns. Due to the nonconvex constraints on analog and digital weights and the nonconvex nonsmooth objective function, the resultant codesign is NP-hard. To address this issue, we create an efficient algorithm by leveraging block coordinate descent (BCD) and Riemannian manifold optimization. We first equivalently represent the objective function as a smooth bi-quadratic function by introducing a uni-modulus auxiliary variable. Subsequently, we alternatingly optimize the scaling factor, digital weights, analog weights and auxiliary variable under the BCD framework, where the simultaneous update of analog weights and auxiliary variable is recast as uni-modular constrained quadratic programming which is efficiently solved by Riemannian Newton method, and the digital weights have a global optimal solution via Lagrangian multiplier method. We also derive an explicit convergence condition of this algorithm. Numerical results demonstrate that the proposed algorithm has superior performance and faster convergence speed than alternative algorithms, and produces nearly the same mainlobe gain as fully digital arrays with significantly fewer radio-frequency chains.

Centroid opposition-based backtracking search algorithm for global optimization and engineering problems

Evolutionary algorithms (EAs) have a lot of potential to handle nonlinear and non-convex objective functions. Particularly, the backtracking search algorithm (BSA) is a popular nature-based evolutionary optimization method that has attracted many researchers due to its simple structure and efficiency in problem-solving across diverse fields. However, like other optimization algorithms, BSA is also prone to reduced diversity, local optima, and inadequate intensification capabilities. To overcome the flaws and increase the performance of BSA, this research proposes a centroid opposition-based backtracking search algorithm (CoBSA) for global optimization and engineering design problems. In CoBSA, specific individuals simultaneously acquire current and historical population knowledge to preserve population variety and improve exploration capability. On the other hand, other individuals execute the position from the current population's centroid opposition to progress convergence speed and exploitation potential. In addition, an elite process based on logistic chaotic local search was developed to improve the superiority of the current individuals. The suggested CoBSA was validated on a set of benchmark functions and then employed in a set of application examples. According to extensive numerical results and assessments, CoBSA outperformed the other state-of-the-art methods in terms of accurateness, reliability, and execution capability.

A nonlinear spectral core–periphery detection method for multiplex networks

Core–periphery detection aims to separate the nodes of a complex network into two subsets: a core that is densely connected to the entire network and a periphery that is densely connected to the core but sparsely connected internally. The definition of core–periphery structure in multiplex networks that record different types of interactions between the same set of nodes on different layers is non-trivial since a node may belong to the core in some layers and to the periphery in others. We propose a nonlinear spectral method for multiplex networks that simultaneously optimizes a node and a layer coreness vector by maximizing a suitable non-convex homogeneous objective function by a provably convergent alternating fixed-point iteration. We derive a quantitative measure for the quality of a given multiplex core–periphery structure that allows the determination of the optimal core size. Numerical experiments on synthetic and real-world networks illustrate that our approach is robust against noisy layers and significantly outperforms baseline methods while improving the latter with our novel optimized layer coreness weights. As the runtime of our method depends linearly on the number of edges of the network, it is scalable to large-scale multiplex networks.

Solving Optimal Power Flow Using New Efficient Hybrid Jellyfish Search and Moth Flame Optimization Algorithms

This paper presents a new optimization technique based on the hybridization of two meta-heuristic methods, Jellyfish Search (JS) and Moth Flame Optimizer (MFO), to solve the Optimal Power Flow (OPF) problem. The JS algorithm offers good exploration capacity but lacks performance in its exploitation mechanism. To improve its efficiency, we combined it with the Moth Flame Optimizer, which has proven its ability to exploit good solutions in the search area. This hybrid algorithm combines the advantages of both algorithms. The performance and precision of the hybrid optimization approach (JS-MFO) were investigated by minimizing well-known mathematical benchmark functions and by solving the complex OPF problem. The OPF problem was solved by optimizing non-convex objective functions such as total fuel cost, total active transmission losses, total gas emission, total voltage deviation, and the voltage stability index. Two test systems, the IEEE 30-bus network and the Mauritanian RIM 27-bus transmission network, were considered for implementing the JS-MFO approach. Experimental tests of the JS, MFO, and JS-MFO algorithms on eight well-known benchmark functions, the IEEE 30-bus, and the Mauritanian RIM 27-bus system were conducted. For the IEEE 30-bus test system, the proposed hybrid approach provides a percent cost saving of 11.4028%, a percent gas emission reduction of 14.38%, and a percent loss saving of 50.60% with respect to the base case. For the RIM 27-bus system, JS-MFO achieved a loss percent saving of 50.67% and percent voltage reduction of 62.44% with reference to the base case. The simulation results using JS-MFO and obtained with the MATLAB 2009b software were compared with those of JS, MFO, and other well-known meta-heuristics cited in the literature. The comparison report proves the superiority of the JS-MFO method over JS, MFO, and other competing meta-heuristics in solving difficult OPF problems.

On convergence of a q-random coordinate constrained algorithm for non-convex problems

We propose a random coordinate descent algorithm for optimizing a non-convex objective function subject to one linear constraint and simple bounds on the variables. Although it is common use to update only two random coordinates simultaneously in each iteration of a coordinate descent algorithm, our algorithm allows updating arbitrary number of coordinates. We provide a proof of convergence of the algorithm. The convergence rate of the algorithm improves when we update more coordinates per iteration. Numerical experiments on large scale instances of different optimization problems show the benefit of updating many coordinates simultaneously.

Online Distributed Optimization With Nonconvex Objective Functions Under Unbalanced Digraphs: Dynamic Regret Analysis

Online Distributed Optimization With Nonconvex Objective Functions Under Unbalanced Digraphs: Dynamic Regret Analysis

Online distributed nonconvex optimization with stochastic objective functions: High probability bound analysis of dynamic regrets

In this paper, the problem of online distributed optimization with stochastic and nonconvex objective functions is studied by employing a multi-agent system. When making decisions, each agent only has access to a noisy gradient of its own objective function in the previous time and can only communicate with its immediate neighbors via a time-varying digraph. To handle this problem, an online distributed stochastic projection-free algorithm is proposed. Of particular interest is that the dynamic regrets are employed to measure the performance of the online algorithm. Existing works on online distributed algorithms involving stochastic gradients only provide the sublinearity results of regrets in expectation. Different from them, we study the high probability bounds of dynamic regrets, i.e., the sublinear bounds of dynamic regrets are characterized by the natural logarithm of the failure probability’s inverse. Under mild assumptions on the graph and objective functions, we prove that if the variations in both the objective function sequence and its gradient sequence grow within a certain rate, then the high probability bounds of the dynamic regrets grow sublinearly. Finally, a simulation example is carried out to demonstrate the effectiveness of our theoretical results.

Research on multi-source sparse optimization method and its application on gearbox compound fault detection

In general, gearbox is prone to occur compound fault frequently because of its harsh working environment, its fault vibration signal often contains polymorphic-oscillatory components and is corrupted by heavy background noise, which brings great difficulty to diagnose fault. Sparse decomposition is often utilized to extract weak fault feature among heavy background noise. In order to solve the problems of traditional sparse decomposition method, such as lacking signal fidelity, causing local optimal solution by using the non-convex objective function, and presenting poor universality, a multi-source sparse optimization objective function with convexity is constructed based on the generalized mini-max concave penalty function. By using forward–backward splitting algorithm combination with Laplace wavelet dictionary, Morlet wavelet dictionary and DFT dictionary, the sparse coefficients corresponding to polymorphic-oscillatory components can be computed efficiently and each oscillatory component can be extracted accurately. Finally, simulation and experimental signal validate that the proposed method can decompose fault signal according to oscillatory property and diagnose gearbox compound fault without the prior knowledge of specific fault numbers.

Continuous Equality Knapsack with Probit-Style Objectives

We study continuous, equality knapsack problems with uniform separable, non-convex objective functions that are continuous, antisymmetric about a point, and have concave and convex regions. For example, this model captures a simple allocation problem with the goal of optimizing an expected value where the objective is a sum of cumulative distribution functions of identically distributed normal distributions (i.e., a sum of inverse probit functions). We prove structural results of this model under general assumptions and provide two algorithms for efficient optimization: (1) running in linear time and (2) running in a constant number of operations given preprocessing of the objective function.

A Collaborative Neurodynamic Optimization Approach to Distributed Chiller Loading.

In this article, we present a collaborative neurodynamic optimization approach to distributed chiller loading in the presence of nonconvex power consumption functions and binary variables associated with cardinality constraints. We formulate a cardinality-constrained distributed optimization problem with nonconvex objective functions and discrete feasible regions, based on an augmented Lagrangian function. To overcome the difficulty caused by the nonconvexity in the formulated distributed optimization problem, we develop a collaborative neurodynamic optimization method based on multiple coupled recurrent neural networks reinitialized repeatedly using a meta-heuristic rule. We elaborate on experimental results based on two multi-chiller systems with the parameters from the chiller manufacturers to demonstrate the efficacy of the proposed approach in comparison to several baselines.

On Mean-Optimal Robust Linear Discriminant Analysis

Linear discriminant analysis (LDA) is widely used for dimensionality reduction under supervised learning settings. Traditional LDA objective aims to minimize the ratio of the squared Euclidean distances that may not perform optimally on noisy datasets. Multiple robust LDA objectives have been proposed to address this problem, but their implementations have two major limitations. One is that their mean calculations use the squared \(\ell_{2}\) -norm distance to center the data, which is not valid when the objective depends on other distance functions. The second problem is that there is no generalized optimization algorithm to solve different robust LDA objectives. In addition, most existing algorithms can only guarantee that the solution is locally optimal rather than globally optimal. In this article, we review multiple robust loss functions and propose a new and generalized robust objective for LDA. Besides, to remove the mean value within data better, our objective uses an optimal way to center the data through learning. As one important algorithmic contribution, we derive an efficient iterative algorithm to optimize the resulting non-smooth and non-convex objective function. We theoretically prove that our solution algorithm guarantees that both the objective and the solution sequences converge to globally optimal solutions at a sub-linear convergence rate. The results of comprehensive experimental evaluations demonstrate the effectiveness of our new method, achieving significant improvements compared to the other competing methods.

A novel improved harbor seal whiskers algorithm for solving hybrid dynamic economic environmental dispatch considering uncertainty of renewable energy generation

The originality of this work is introducing a novel Improved Harbor Seal Whiskers Algorithm (IHSWA) as an innovative and improved optimizer for solving complex hybrid dynamic economic-environmental dispatch (HDEED) considering uncertainty of renewable energy generation (REG) and various system constraints. IHSWA is implemented by hybridizing opposition-based algorithm (OBA) and local search algorithm (LSA) approaches to enhance the algorithm's search efficiency and overcome the challenges of other counterpart approaches. IHSWA has an optimal balancing feature between searching modes (exploring and exploiting modes). This feature makes the proposed algorithms more advanced than their counterpart optimizing techniques. The HDEED is modeled using a non-convex and non-smooth objective function, considering various equality and inequality constraints such as ramp rate bounds (RRLs), valve-point effects (VPE), production limits (PL), spinning reserves (SRLs), prohibited operating zones (POZs), and power balance (PB). The proposed approach is verified on a hybrid modified IEEE 57-bus 7-unit power system incorporating solar and wind energy generation. The proposed approach is compared with various modern optimization techniques, such as the traditional Harbor Seal Whiskers Algorithm (HSWA), the Lemurs Optimization Algorithm (LOA), the Nutcracker Optimization Algorithm (NOA), and the Artificial Hummingbird Algorithm (AHA). The incorporation of REG sources produces a significant reduction in operative costs and a remarkable decline in emissions. The convergence rate of the proposed approach indicates that the novel hybrid algorithm converges rapidly towards the optimal solution, achieving the best global optimum solution. The ANOVA test and the Tukey test are used to analyze the results statistically. The IHSWA approach shows superior advanced features with a best fuel cost of 1640.5890 $ and a minimum std. value of 1708.4895. While for emission objective minimizing, IHSWA attained a minimum emission cost and std. value of 1080.1629 kg and 1777.0770, respectively. The statistical analysis of the results approves the stability, reliability, and consistency of the proposed strategy with significant convergence. The appreciated contribution of this work is introducing a novel hybrid approach for solving HDEED efficiently, incorporating REG, and considering various system constraints. IHSWA has advanced features over its counterpart approaches. The contribution holds specific importance for the economic development of countries. It creates the path for increased self-sufficiency by reducing their need for fossil fuels for energy production. The study emphasizes the transition from being reliant on fossil fuels to the development of sustainable urban economies.

Hypergraph-based convex semi-supervised unconstraint symmetric matrix factorization for image clustering

Semi-supervised symmetric nonnegative matrix factorization (SNMF) has been extensively utilized in both linear and nonlinear data clustering tasks. However, the current SNMF model's non-convex objective function faces challenges in global optimization and time efficiency. In this study, we leverage label information to propose a convex and unconstrained symmetric matrix factorization (SMF) model that is thoroughly analyzed for its convexity properties. In order to capture high-order relationships among data, a hypergraph is utilized in the model, which is computationally simple, translation invariant, and naturally normalized. Moreover, based on the analysis and the corresponding experiments in the paper, the model exhibits robustness towards outliers to some extent. Due to the convexity of our proposed model without constraint, it can be efficiently optimized using the Conjugate Gradient (CG) method, one of the most efficient methods available. Therefore, we propose a novel Convex Combination-based Sufficient Descent CG (CSDCG) method, which outperforms other methods across 284 optimization problems within the CUTEst library. In order to evaluate the effectiveness of the proposed method, the semi-supervised clustering experiments are conducted on the eight datasets by comparison with ten state-of-the-art matrix factorization (MF) methods. The experiment results demonstrate its superiority over the other compared methods to handle the clustering problem with better performance and less computational time. The code is available at https://github.com/Pokemer/HCSSMF.

Phase retrieval for radar waveform design

Abstract The ability of a radar to discriminate in both range and Doppler velocity is completely characterized by the ambiguity function (AF) of its transmit waveform. Mathematically, it is obtained by correlating the waveform with its Doppler-shifted and delayed replicas. We consider the inverse problem of designing a radar transmit waveform that satisfies the specified AF magnitude. This process may be viewed as a signal reconstruction with some variation of phase retrieval methods. We provide a trust-region algorithm that minimizes a smoothed non-convex least-squares objective function to iteratively recover the underlying signal-of-interest for either time- or band-limited support. The method first approximates the signal using an iterative spectral algorithm and then refines the attained initialization based on a sequence of gradient iterations. Our theoretical analysis shows that unique signal reconstruction is possible using signal samples no more than thrice the number of signal frequencies or time samples. Numerical experiments demonstrate that our method recovers both time- and band-limited signals from sparsely and randomly sampled, noisy and noiseless AFs.

A stochastic gradient method with variance control and variable learning rate for Deep Learning

In this paper we study a stochastic gradient algorithm which rules the increase of the mini-batch size in a predefined fashion and automatically adjusts the learning rate by means of a monotone or non-monotone line search procedure. The mini-batch size is incremented at a suitable a priori rate throughout the iterative process in order that the variance of the stochastic gradients is progressively reduced. The a priori rate is not subject to restrictive assumptions, allowing for the possibility of a slow increase in the mini-batch size. On the other hand, the learning rate can vary non-monotonically throughout the iterations, as long as it is appropriately bounded. Convergence results for the proposed method are provided for both convex and non-convex objective functions. Moreover it can be proved that the algorithm enjoys a global linear rate of convergence on strongly convex functions. The low per-iteration cost, the limited memory requirements and the robustness against the hyperparameters setting make the suggested approach well-suited for implementation within the deep learning framework, also for GPGPU-equipped architectures. Numerical results on training deep neural networks for multi-class image classification show a promising behaviour of the proposed scheme with respect to similar state of the art competitors.

Nonconvex distributed feedback optimization for aggregative cooperative robotics

Distributed aggregative optimization is a recently emerged framework in which the agents of a network want to minimize the sum of local objective functions, each one depending on the agent decision variable (e.g., the local position of a team of robots) and an aggregation of all the agents’ variables (e.g., the team barycenter). In this paper, we address a distributed feedback optimization framework in which agents implement a local (distributed) policy to reach a steady-state minimizing an aggregative cost function. We propose AGGREGATIVE TRACKING FEEDBACK, i.e., a novel distributed feedback optimization law in which each agent combines a closed-loop gradient flow with a consensus-based dynamic compensator reconstructing the missing global information. By using tools from system theory, we prove that AGGREGATIVE TRACKING FEEDBACK steers the network to a stationary point of an aggregative optimization problem with (possibly) nonconvex objective function. The effectiveness of the proposed method is validated through numerical simulations on a multi-robot surveillance scenario.

A nonconvex sparse recovery method for DOA estimation based on the trimmed lasso

Sparse direction-of-arrival (DOA) estimation methods can be formulated as a group-sparse optimization problem. Meanwhile, sparse recovery methods based on nonconvex penalty terms have been a hot topic in recent years due to their several appealing properties. Herein, this paper studies a new nonconvex regularized approach called the trimmed lasso for DOA estimation. We define the penalty term of the trimmed lasso in the multiple measurement vector model by ℓ2,1-norm. First, we use the smooth approximation function to change the nonconvex objective function to the convex weighted problem. Next, we derive sparse recovery guarantees based on the extended Restricted Isometry Property and regularization parameter for the trimmed lasso in the multiple measurement vector problem. Our proposed method can control the desired level of sparsity of estimators exactly and give a more precise solution to the DOA estimation problem. Numerical simulations show that our proposed method overperforms traditional approaches, which is more close to the Cramer-Rao bound.