Nonconvex Function Research Articles

Continuation Multiple Instance Learning for Weakly and Fully Supervised Object Detection.

Weakly supervised object detection (WSOD) is a challenging task that requires simultaneously learning object detectors and estimating object locations under the supervision of image category labels. Many WSOD methods that adopt multiple instance learning (MIL) have nonconvex objective functions and, therefore, are prone to get stuck in local minima (falsely localize object parts) while missing full object extent during training. In this article, we introduce classical continuation optimization into MIL, thereby creating continuation MIL (C-MIL) with the aim to alleviate the nonconvexity problem in a systematic way. To fulfill this purpose, we partition instances into class-related and spatially related subsets and approximate MIL's objective function with a series of smoothed objective functions defined within the subsets. We further propose a parametric strategy to implement continuation smooth functions, which enables C-MIL to be applied to instance selection tasks in a uniform manner. Optimizing smoothed loss functions prevents the training procedure from falling prematurely into local minima and facilities learning full object extent. Extensive experiments demonstrate the superiority of CMIL over conventional MIL methods. As a general instance selection method, C-MIL is also applied to supervised object detection to optimize anchors/features, improving the detection performance with a significant margin.

Trajectory and Transmit Power Optimization for IRS-Assisted UAV Communication Under Malicious Jamming

In this letter, we investigate an unmanned aerial vehicle (UAV) communication system, where an intelligent reflecting surface (IRS) is deployed to assist in the transmission from a ground node (GN) to the UAV in the presence of a jammer. We aim to maximize the average rate of the UAV communication by jointly optimizing the GN's transmit power, the IRS's passive beamforming and the UAV's trajectory. However, the formulated problem is difficult to solve due to the non-convex objective function and the coupled optimization variables. Thus, to tackle it, we propose an alternating optimization (AO) based algorithm by exploiting the successive convex approximation (SCA) and semidefinite relaxation (SDR) techniques. Simulation results show that the proposed algorithm can significantly improve the average rate compared with the benchmark algorithms. Moreover, it also shows that when the jamming power is large and the number of IRS elements is relatively small, deploying the IRS near the jammer outperforms deploying it near the GN, and vice versa.

A new non-convex low rank minimization model to decompose an image into cartoon and texture components

Decomposition of an image into cartoon and texture components is frequently used in many image processing applications. Here, the cartoon component has been characterized by the frequently used total variation norm. However, it becomes very challenging to obtain the texture component due to the varying nature of the texture. In general, the texture component has oscillatory behavior locally or globally. Owing to this oscillatory behavior, the texture component has been characterized via low-rank regularization which is widely used to extract texture component from the image. In the works reported till now, convex nuclear norm has been frequently used as a surrogate of the matrix rank, which is suboptimal because of shrinking each singular value equally, while the non-convex surrogate of the rank treats each singular value adaptively. In this paper, we are introducing a new tightest non-convex surrogate of the rank that assigns different weights to each singular value. The new non-convex image decomposition minimization model provides us cartoon and texture components by minimizing the total variation norm and non-convex function simultaneously. This model can also work best for many image restoration problems such as image deblurring and inpainting. The conventional alternating direction method of multiplier (ADMM) has been exploited as the solver of the non-convex minimization model. The proposed model works well on both globally patterned and natural images. In the experimental section, we demonstrate the outperformance of the proposed model over the state-of-the-art methods.

Distributed stochastic nonsmooth nonconvex optimization

Distributed consensus optimization has received considerable attention in recent years and several distributed consensus-based algorithms have been proposed for (nonsmooth) convex and (smooth) nonconvex objective functions. However, the behavior of these distributed algorithms on nonconvex, nonsmooth and stochastic objective functions is not understood. Such class of functions and distributed setting are motivated by several applications, including problems in machine learning and signal processing. This paper presents the first convergence analysis of the decentralized stochastic subgradient method for such classes of problems, over networks modeled as undirected, fixed, graphs.

Vertiport Planning for Urban Aerial Mobility: An Adaptive Discretization Approach

Problem definition: Electric vertical-takeoff-and-landing (eVTOL) vehicles enable urban aerial mobility (UAM). This paper optimizes the number, locations, and capacities of vertiports in UAM systems while capturing interdependencies between strategic vertiport deployment, tactical operations, and passenger demand. Academic/practical relevance: The model includes a “tractable part” (based on mixed-integer second-order conic optimization) and also a nonconvex demand function. Methodology: We develop an exact algorithm that approximates nonconvex functions with piecewise constant segments, iterating between a conservative model (which yields a feasible solution) and a relaxed model (which yields a solution guarantee). We propose an adaptive discretization scheme that converges to a global optimum—because of the relaxed model. Results: Our algorithm converges to a 1% optimality gap, dominating static discretization benchmarks in terms of solution quality, runtimes, and solution guarantee. Managerial implications: We find that the most attractive structure for UAM is one that uses a few high-capacity vertiports, consolidating operations primarily to serve long-distance trips. Moreover, UAM profitability is highly sensitive to network planning optimization and to customer expectations, perhaps even more so than to vehicle specifications. Therefore, the success of UAM operations requires not only mature eVTOL technologies but also tailored analytics-based capabilities to optimize strategic planning and market-based efforts to drive customer demand. Supplemental Material: The online appendix is available at https://doi.org/10.1287/msom.2022.1148 .

Improved Harris hawks optimization for non-convex function optimization and design optimization problems

Harris hawks optimization (HHO) is a nature-inspired algorithm. It has the advantages of very few parameters, a simple structure, fast convergence and strong local search capability. The main drawback of the Harris hawks optimization is that it can easily fall into a local optimum. To solve this problem, a novel mutant strategy based on Brownian motion is proposed to combine with the original HHO. This mutant strategy is driven by exploiting the randomness of Brownian motion and does not require location information between populations and user-set parameters. As a result, it can guide the algorithm to better jump out of the local optimum trap. To verify the performance of the proposed algorithm, numerical experiments are carried out to compare the proposed algorithm with heuristic optimization algorithms for 54 non-convex functions and two classic engineering design problems. The results show that our algorithm not only escapes the local optimum trap, but also has better robustness and convergence.

Distributed robust cooperative scheduling of multi-region integrated energy system considering dynamic characteristics of networks

Due to the advantages of reliability and economy, multiple adjacent region integrated energy systems (IESs) are interconnected to form a multi-region IES through multi-energy network. However, the limitation on private data exchange, multi-uncertainty and the complex dynamic characteristics bring challenges to the optimal scheduling of the multi-region IES. This paper proposes a distributed robust cooperative scheduling method for the multi-region electricity–natural gas–heat IES considering the dynamic characteristics and the uncertainties of wind turbine (WT) and electricity load. Firstly, a robust scheduling model of the multi-region IES considering dynamic characteristics and the uncertainties is built. In particular, the dynamic transmission models of gas and heating networks are constructed, and a regulation model of virtual heat storage is built to control the virtual heat storage in district heating network (DHN). Further, to preserve the information privacy and decision-making independence for each sub-region IES, a distributed cooperative scheduling framework based on the consensus-based alternating direction method of multipliers (ADMM) is developed for the multi-region IES, where the original robust scheduling model is decomposed into several subproblems that are solved independently. Moreover, the nonconvex dynamic natural gas flow function is addressed by convex relaxation technique, and the max–min robust model with binary variables is solved by alternative optimization procedure (AOP) method and duality theory. Finally, the simulation results show that the proposed method can converge reliably, realize the balance between the operation cost and the heat loss and provide a flexible scheduling scheme for the multi-region IES.

Efficient Modified Meta-Heuristic Technique for Unconstrained Optimization Problems

In this paper, a new Modified Meta-Heuristic algorithm is proposed. This method contains some modifications to improve the performance of the simulated-annealing algorithm (SA). Most authors who deal with improving the SA algorithm presented some improvements and modifications to one or more of the five standard features of the SA algorithm. In this paper, we improve the SA algorithm by presenting some suggestions and modifications to all five standard features of the SA algorithm. Through these suggestions and modifications, we obtained a new algorithm that finds the approximate solution to the global minimum of a non-convex function. The new algorithm contains novel parameters, which are updated at each iteration. Therefore, the variety and alternatives in choosing these parameters demonstrated a noticeable impact on the performance of the proposed algorithm. Furthermore, it has multiple formulas by which the candidate solutions are generated. Diversity in these formulas helped the proposed algorithm to escape a local point while finding the global minimizer of a non-convex function. The efficiency of the proposed algorithm is reported through extensive numerical experiments on some well-known test problems. The performance profiles are used to evaluate and compare the performance of our proposed algorithm against the other five meta-heuristic algorithms. The comparison results between the performance of our suggested algorithm and the other five algorithms indicate that the proposed algorithm is competitive with, and in all cases superior to, the five algorithms in terms of the efficiency, reliability, and effectiveness for finding the global minimizers of non-convex functions. This superiority of the new proposed algorithm is due to those five modified standard features.

An Adaptive Alternating Direction Method of Multipliers

The alternating direction method of multipliers (ADMM) is a powerful splitting algorithm for linearly constrained convex optimization problems. In view of its popularity and applicability, a growing attention is drawn toward the ADMM in nonconvex settings. Recent studies of minimization problems for nonconvex functions include various combinations of assumptions on the objective function including, in particular, a Lipschitz gradient assumption. We consider the case where the objective is the sum of a strongly convex function and a weakly convex function. To this end, we present and study an adaptive version of the ADMM which incorporates generalized notions of convexity and penalty parameters adapted to the convexity constants of the functions. We prove convergence of the scheme under natural assumptions. To this end, we employ the recent adaptive Douglas–Rachford algorithm by revisiting the well-known duality relation between the classical ADMM and the Douglas–Rachford splitting algorithm, generalizing this connection to our setting. We illustrate our approach by relating and comparing to alternatives, and by numerical experiments on a signal denoising problem.

AFAFed—Asynchronous Fair Adaptive Federated learning for IoT stream applications

In this paper, we design, analyze the convergence properties, address the implementation aspects, and numerically test the performance of AFAFed. This is a novel Asynchronous Fair Adaptive Federated learning framework for stream-oriented IoT application environments, which are featured by time-varying operating conditions, heterogeneous resource-limited devices (i.e., coworkers), non-i.i.d. local training data and unreliable communication links. The key new of AFAFed is the synergic co-design of: (i) two sets of adaptively tuned tolerance thresholds and fairness coefficients at the coworkers and central server, respectively; and, (ii) a distributed adaptive mechanism, which allows each coworker to adaptively tune own communication rate. The convergence properties of AFAFed under (possibly) non-convex loss functions is guaranteed by a set of new analytical bounds, which formally unveil the impact on the resulting AFAFed convergence rate of a number of Federated Learning (FL) parameters, like, first and second moments of the per-coworker number of consecutive model updates, data skewness, communication packet-loss probability, and maximum/minimum values of the (adaptively tuned) mixing coefficient used for model aggregation. Extensive numerical tests show that AFAFed is capable to improve test accuracy by up to 20% and reduce training time by up to 50%, compared to state-of-the-art FL schemes, even under challenging learning scenarios featured by deep Machine Learning (ML) models, data skewness, coworker heterogeneity and unreliable communication.

A new nonconvex relaxation approach for low tubal rank tensor recovery

In this paper, we consider the tensor recovery problem within the low tubal rank framework. A new nonconvex surrogate is proposed to approximate the tensor tubal rank. The main advantage is that it uses a class of nonconvex functions to penalize the singular tubes directly instead of penalizing the singular values as in existing methods. Our proposed surrogate can continuously approximate the tubal rank without breaking its composition structure and keep the intrinsic structural information of the tensor better than existing methods. We then develop an efficient iteratively reweighted tube thresholding algorithm to solve the tensor recovery model equipped with the new tubal rank surrogate and provide the theoretical guarantee for convergence. Simulation results on synthetical and practical data demonstrate the superior performance of the proposed method over several widely used methods in the literature.

A unified framework for nonconvex nonsmooth sparse and low-rank decomposition by majorization-minimization algorithm

Recovering a low-rank matrix and a sparse matrix from an observed matrix, known as sparse and low-rank decomposition (SLRD), is becoming a hot topic in recent years. The most popular model for SLRD is to use the ℓ1 norm and nuclear norm for the sparse and low-rank approximation. Since this convex model has certain limitations, various nonconvex models have been explored and found to be very promising. In this paper, we introduce a generalized nonconvex nonsmooth model for SLRD which covers a wide range of nonconvex surrogate functions that are continuous, concave and monotonically increasing on [0,∞) to approximate both the ℓ0 norm and the rank function, such as ℓp norm (0<p<1), Logarithm, Geman, SCAD and MCP functions. The choice of the nonconvex surrogates for the sparse and low-rank components can be different. Due to the nonconvexity and extensive options of the surrogates, the optimization problem is untractable. Based on the majorization-minimization (MM) algorithm, we propose a unified framework named MM-ADMM algorithm to solve this problem, which can be applied to all eligible surrogates as long as their supergradients are available. The constrained majorizing problems established under the MM framework can be easily solved by the alternating direction method of multipliers (ADMM). The theoretical convergence properties are investigated and proved, including the convergence of the sequence of objective function values generated by the designed algorithm and a weak convergence result related to the inner ADMM-iterations. Experiments on the synthetic data and real-world applications demonstrate the effectiveness of our designed MM-ADMM algorithm.

Nonconvex multi-view subspace clustering via simultaneously learning the representation tensor and affinity matrix* *This research was supported by the National Natural Science Foundations of China (12071159, U1811464) and the NSF-DMS 1854638 of the United States.

Multi-view subspace clustering, which aims to partition a dataset into its relevant subspaces based on their multi-view features, has been widely applied to identify various characteristics of datasets. The typical model of multi-view subspace clustering in literature often makes use of the nuclear norm to seek the underlying low-rank representation. However, due to the sum property of the singular values defined by tensor nuclear norm, the existing multi-view subspace clustering does not well handle the noise and the illumination variations embedded in multi-view data. To address and improve the robustness and clustering performance, we propose a new nonconvex multi-view subspace clustering model via tensor minimax concave penalty (MCP) approximation associated with rank minimization (NMSC-MCP), which can simultaneously construct the low-rank representation tensor and affinity matrix in a unified framework. Specifically, the nonconvex MCP approximation rank function is adopted to as a tighter tensor rank approximation to discriminate the dimension of features so that better accuracy can be achieved. In addition, we also address the local structure by including both hyper-Laplacian regularization and auto-weighting scheme into the objective function to promote the clustering performance. A corresponding iterative algorithm is then developed to solve the proposed model and the constructed iterative sequence generated by the proposed algorithm is shown to converge to the desirable KKT critical point. Extensive experiments on benchmark datasets have demonstrate the highly desirable effectiveness of our proposed method.

Spectrum Sharing Among Multiple-Seller and Multiple-Buyer Operators of a Mobile Network: A Stochastic Geometry Approach

Sharing the spectrum among mobile network operators (MNOs) is a promising approach to improve the spectrum utilization and to increase the monetary income of MNOs. In this paper, we model a nonorthogonal spectrum sharing system for a large-scale cellular network where multiple seller MNOs lease their licensed sub-bands to several buyer MNOs. We first analyze the per-user expected rate and the per-MNO expected profit using stochastic geometry. Then, we formulate the joint problem of power control and licensed sub-band sharing to maximize the expected profit of all MNOs as a multiobjective optimization problem (MOOP) under the users' quality of service requirement and the nonnegative return on investment constraints. To transform the MOOP into a single objective form, we use a combination of the $\epsilon$-constraint and weighted sum methods. However, the transformed problem is nonconvex because of the presence of binary variables and nonconvex rate functions in the objective function and constraints. We address this problem by using a penalty function and approximating the nonconvex rate functions by a constrained stochastic successive convex approximation method. Finally, the numerical results show the correctness and performance of the proposed algorithm under various conditions.

High-accuracy low-cost privacy-preserving federated learning in IoT systems via adaptive perturbation

With the rapid development of the Internet of Things (IoT), federated learning (FL) has been widely used to obtain insights from collected data while preserving data privacy. Differential privacy (DP) is an additive noise scheme that has been widely studied as a privacy-preserving approach on FL. However, privacy protection under DP usually comes at the cost of model accuracy for the underlying FL process. In this paper, we propose a novel low-cost (for both communication and computational overhead) adaptive noise perturbation/masking scheme to protect FL clients’ privacy without degrading the global model accuracy. In particular, we set the magnitude of the additive noise to adaptively change with the magnitude of the local model updates. Then, a direction-based filtering scheme is used to accelerate the convergence of the FL model. A maximum tolerable noise bound for local clients is derived using the central limit theorem (CLT). The designed noise maximizes privacy protection for clients while preserving the accuracy and convergence rate of the FL model , as a result of the noise cancelling out and forming a more concentrated distribution after the aggregation operation on the server. We theoretically prove that FL with the proposed noise perturbation scheme retains the same accuracy and convergence rate (O(1/T) for convex loss functions and O(1/T) for non-convex loss functions) as that of non-private FL with SGD. We also evaluate the performance of the proposed scheme in terms of convergence behavior, computational efficiency, and privacy protection against state-of-the-art privacy inference attacks on real-world datasets. Experimental results show that FL with our proposed perturbation scheme outperforms DP in the accuracy and convergence rate of the FL model in both client dropout and non-client dropout scenarios. Compared with DP, our proposed scheme does not incur additional computational and communication overhead. Our approach provides DP-comparable or better effectiveness in defending against privacy attacks under the same global model accuracy.

Energy-Efficient Resource Allocation in Multi-UAV-Assisted Two-Stage Edge Computing for Beyond 5G Networks

Unmanned aerial vehicle (UAV)-assisted multi-access edge computing (MEC) has become one promising solution for energy-constrained devices to run the applications with high computation demand and stringent delay requirement in beyond 5G era. In this work, we study a multi-UAV-assisted two-stage MEC system in which UAVs provide the computing and relaying services to the mobile devices. Due to the limited computing resources, each UAV executes only a portion of the offloaded tasks from its associated MDs in the first stage. Hence, in the second stage, each UAV relays the portions of the tasks to the terrestrial base station (TBS) which has rich computing resources enough to handle all the tasks relayed to it. In this regard, we formulate a joint task offloading, communication and computation resource allocation problem to minimize the energy consumption of MDs and UAVs by considering the limited resources of UAVs and the tolerable latency of the tasks. The formulated problem is a mixed-integer non-convex problem which is NP hard. To solve the formulated optimization problem, we apply the Block Successive Upper-bound Minimization (BSUM) method which guarantees to obtain the stationary points of the non-convex objective function. Finally, the extensive evaluation results are conducted to show the superior performance of our proposed framework.

Improving defocus blur measures using robust regularization

This paper presents a robust regularization scheme for the enhancement of initial blur map in single image blur detection. The proposed technique considers the structural differences between guidance and initial blur map. Specifically, we propose to improve the initial blur map by optimizing a nonconvex energy function that jointly leverages structural information from guidance and blur map. Nonconvex problem is solved by majorize-minimize algorithm and an improved blur map is obtained that is adequately smooth and has better edge-preserving properties. Ultimately, this results in high-quality blur segmentation.

Estimation in the High Dimensional Additive Hazard Model with [formula omitted] Type of Penalty

High-dimensional data is commonly observed in survival data analysis. Penalized regression is widely applied for parameter selection given this type of data. The LASSO, SCAD and MCP methods are basic penalties developed in recent years in order to achieve more accurate selection of parameters. The l0 penalty, which selects the best subset of parameters and provides unbiased estimation, is relatively difficult to handle due to its NP-hard complexity resulted from the non-smooth and non-convex objective function. For the additive hazard model, most methods developed so far focus on providing a smoothed version of l0-norm. Instead of mimicking these methods, two augmented Lagrangian based algorithms, namely the ADMM-l0 method and the APM-l0 method, are proposed to approximate the optimal solution generated by the l0 penalty. The ADMM-l0 algorithm can achieve unbiased parameter estimation, while the two-step APM-l0 method is computationally more efficient. The convergence of ADMM-l0 can be proved under strict assumptions. Under moderate sample sizes, both methods perform well in selecting the best subset of parameters, especially in terms of controlling the false positive rate. Finally, both methods are applied to two real datasets.

Energy Efficiency and Resource Allocation Optimization with MIMONOMA and Backhaul Beam-forming in User-centric Ultra-dense Networks

Background: Non-orthogonal multiple access (NOMA) is viewed as the key multiple access technology for 5G and beyond networks, attracting the attention of academics and industries. NOMA and the multiple input multiple output (MIMO-NOMA) technology can improve a system’s throughput, latency, and energy efficiency (EE) in future-generation communication networks. Objective: The objective of this paper is to achieve maximum EE by applying the Max-min Power Control Algorithm (MMPCA) through sub-channel optimization, resource allocation (RA) optimization, access point selection (APS), and user association. The EE results obtained with and without using MMPCA are compared to the RA optimization from a conventional water-filling algorithm (WFA). Method: This paper formulates a framework for user-centric (UC) joint resource allocation, such as backhaul connection via beam-forming and Access point (AP) to user connection via MIMO-NOMA. The user without interference is decoded using the NOMA principle. The MMPCA was also used to optimize cooperative power allocation, sub-channel allocation, and efficient user association. The RA for EE is framed as a mixed non-convex and non-linear function using successive convex approximation and sum ratio decoupling converted into convex and linear. A bisection method was used to achieve optimal RA, user association, and sub-channel assignment. Results and Conclusion: The simulation shows energy efficiency (EE) improvement. Similarly, it is observed that MMPCA outperforms the WFA.

A Hybrid Stochastic Deterministic Algorithm for Solving Unconstrained Optimization Problems

In this paper, a new deterministic method is proposed. This method depends on presenting (suggesting) some modifications to existing parameters of some conjugate gradient methods. The parameters of our suggested method contain a mix of deterministic and stochastic parameters. The proposed method is added to a line search algorithm to make it a globally convergent method. The convergence analysis of the method is established. The gradient vector is estimated by a finite difference approximation approach, and a new step-size h of this approach is generated randomly. In addition, a set of stochastic parameter formulas is constructed from which some solutions are generated randomly for an unconstrained problem. This stochastic technique is hybridized with the new deterministic method to obtain a new hybrid algorithm that finds an approximate solution for the global minimization problem. The performance of the suggested hybrid algorithm is tested in two sets of benchmark optimization test problems containing convex and non-convex functions. Comprehensive comparisons versus four other hybrid algorithms are listed in this study. The performance profiles are utilized to evaluate and compare the performance of the five hybrid algorithms. The numerical results show that our proposed hybrid algorithm is promising and competitive for finding the global optimum point. The comparison results between the performance of our suggested hybrid algorithm and the other four hybrid algorithms indicate that the proposed algorithm is competitive with, and in all cases superior to, the four algorithms in terms of the efficiency, reliability, and effectiveness for finding the global minimizers of non-convex functions.