Theoretical Convergence Guarantees Research Articles

The ℓ2, regularized group sparse optimization: Lower bound theory, recovery bound and algorithms

In this paper, we consider an unconstrained ℓ 2 , q minimization for group sparse signal recovery. For this nonconvex and non-Lipschitz problem, we mainly focus on its local minimizers. Firstly, a uniform lower bound for nonzero groups of the local minimizers is presented. Secondly, under group restricted isometry property (GRIP) assumption, we provide a global recovery bound for points in a sublevel set of the objective function, as well as a local recovery bound for local minimizers. Thirdly, a sufficient condition for a stationary point to be a local minimizer is shown. Fourthly, inspired by the lower bound theory which indicates the sparsity of solutions, we propose a new efficient iteratively reweighted least square (IRLS) with thresholding algorithm, with nonexpansiveness of the group support set. Compared with the classical IRLS with smoothing algorithm, our algorithm performs better in both theoretical global convergence guarantee and numerical computation.

Low-Rank Hypergraph Hashing for Large-Scale Remote Sensing Image Retrieval

As remote sensing (RS) images increase dramatically, the demand for remote sensing image retrieval (RSIR) is growing, and has received more and more attention. The characteristics of RS images, e.g., large volume, diversity and high complexity, make RSIR more challenging in terms of speed and accuracy. To reduce the retrieval complexity of RSIR, a hashing technique has been widely used for RSIR, mapping high-dimensional data into a low-dimensional Hamming space while preserving the similarity structure of data. In order to improve hashing performance, we propose a new hash learning method, named low-rank hypergraph hashing (LHH), to accomplish for the large-scale RSIR task. First, LHH employs a l2-1 norm to constrain the projection matrix to reduce the noise and redundancy among features. In addition, low-rankness is also imposed on the projection matrix to exploit its global structure. Second, LHH uses hypergraphs to capture the high-order relationship among data, and is very suitable to explore the complex structure of RS images. Finally, an iterative algorithm is developed to generate high-quality hash codes and efficiently solve the proposed optimization problem with a theoretical convergence guarantee. Extensive experiments are conducted on three RS image datasets and one natural image dataset that are publicly available. The experimental results demonstrate that the proposed LHH outperforms the existing hashing learning in RSIR tasks.

Human-swarm Interactions for Formation Control Using Interpreters

In this work, we develop a novel Human-Swarm Interaction (HSI) framework for formation control using the notion of an interpreter, enabling the user to control a robotic swarm's shape and formation using abstraction. The user conveys their intended commands by drawing shapes through arm gestures and motions which are recorded by an off-the-shelf wearable device. We propose a novel interpreter system, which acts as an intermediary between the user and the swarm to simplify the roles of both. The interpreter takes in high level input in the form of shapes drawn by the user, and translates it into swarm control commands by planning in the shape space using novel shape morphing dynamics (SMD), which is also used for user feedback. The proposed interpreter employs machine learning, estimation and optimal control techniques to translate the users intention into swarm control parameters. The dynamics of the swarm are realized by means of a novel decentralized formation controller based on distributed linear iterations and dynamic average consensus. Theoretical guarantees of convergence along with convergence rate of the proposed swarm controller are given. The resulting shape morphing dynamics are illustrated and discussed through simulations. The entire framework is demonstrated theoretically as well as experimentally in a 2D environment, with a human controlling a swarm of simulated robots in real time with the help of a Graphical User Environemnt (GUI).

Learning-Based Proxy Collision Detection for Robot Motion Planning Applications

This article demonstrates that collision detection-intensive applications such as robotic motion planning may be accelerated by performing collision checks with a machine learning model. We propose Fastron, a learning-based algorithm, to model a robot's configuration space to be used as a proxy collision detector in place of standard geometric collision checkers. We demonstrate that leveraging the proxy collision detector results in up to an order of magnitude faster performance in robot simulation and planning than state-of-the-art collision detection libraries. Our results show that Fastron learns a model more than 100 times faster than a competing C-space modeling approach, while also providing theoretical guarantees of learning convergence. Using the open motion planning libraries (OMPLs), we were able to generate initial motion plans across all experiments with varying robot and environment complexities and workspace obstacle locations. With Fastron, we can repeatedly generate new motion plans at a 56 Hz rate, showing its application toward autonomous surgical assistance task in shared environments with human-controlled manipulators. All performance gains were achieved despite using only CPU-based calculations, suggesting further computational gains with a GPU approach that can parallelize tensor algebra. Code is available online.

Learning Latent Features With Pairwise Penalties in Low-Rank Matrix Completion

Low-rank matrix completion has achieved great success in many real-world data applications. A matrix factorization model that learns latent features is usually employed and, to improve prediction performance, the similarities between latent variables can be exploited by pairwise learning using the graph regularized matrix factorization (GRMF) method. However, existing GRMF approaches often use the squared loss to measure the pairwise differences, which may be overly influenced by dissimilar pairs and lead to inferior prediction. To fully empower pairwise learning for matrix completion, we propose a general optimization framework that allows a rich class of (non-)convex pairwise penalty functions. A new and efficient algorithm is developed to solve the proposed optimization problem, with a theoretical convergence guarantee under mild assumptions. In an important situation where the latent variables form a small number of subgroups, its statistical guarantee is also developed. In particular, we theoretically characterize the performance of the complexity-regularized maximum likelihood estimator, as a special case of our framework, which is shown to have smaller errors when compared to the standard matrix completion framework without pairwise penalties. We conduct extensive experiments on both synthetic and real datasets to demonstrate the superior performance of this general framework.

Domain Adaptation by Class Centroid Matching and Local Manifold Self-Learning.

Domain adaptation has been a fundamental technology for transferring knowledge from a source domain to a target domain. The key issue of domain adaptation is how to reduce the distribution discrepancy between two domains in a proper way such that they can be treated indifferently for learning. In this paper, we propose a novel domain adaptation approach, which can thoroughly explore the data distribution structure of target domain. Specifically, we regard the samples within the same cluster in target domain as a whole rather than individuals and assigns pseudo-labels to the target cluster by class centroid matching. Besides, to exploit the manifold structure information of target data more thoroughly, we further introduce a local manifold self-learning strategy into our proposal to adaptively capture the inherent local connectivity of target samples. An efficient iterative optimization algorithm is designed to solve the objective function of our proposal with theoretical convergence guarantee. In addition to unsupervised domain adaptation, we further extend our method to the semi-supervised scenario including both homogeneous and heterogeneous settings in a direct but elegant way. Extensive experiments on seven benchmark datasets validate the significant superiority of our proposal in both unsupervised and semi-supervised manners.

Variance-Reduced Stochastic Learning Under Random Reshuffling

Several useful variance-reduced stochastic gradient algorithms, such as SVRG, SAGA, Finito, and SAG, have been proposed to minimize empirical risks with linear convergence properties to the exact minimizer. The existing convergence results assume uniform data sampling with replacement. However, it has been observed in related works that random reshuffling can deliver superior performance over uniform sampling and, yet, no formal proofs or guarantees of exact convergence exist for variance-reduced algorithms under random reshuffling. This paper makes two contributions. First, it provides a theoretical guarantee of linear convergence under random reshuffling for SAGA in the mean-square sense; the argument is also adaptable to other variance-reduced algorithms. Second, under random reshuffling, the article proposes a new amortized variance-reduced gradient (AVRG) algorithm with constant storage requirements compared to SAGA and with balanced gradient computations compared to SVRG. AVRG is also shown analytically to converge linearly.

Empirical Convergence Analysis of Federated Averaging for Failure Prognosis

Data driven prognosis involves machine learning algorithms to learn from previous failures and generate its prediction model. However, often a single asset does not fail so frequently to have enough training data in the form of historical failures. This problem can be addressed by learning from failures across a cluster of similar other assets, but often working in different environments. The algorithm therefore must learn from a distributed dataset which might be heterogenous but with underlying similarities. Federated Learning is an emerging technique that has recently also been proposed as a fitting solution for prognosis of industrial assets. However, even the most commonly used Federated Learning algorithms lack theoretical convergence guarantees, and therefore their convergence must be analysed empirically. This paper empirically analyses the convergence of the Federated Averaging (FedAvg) algorithm for a fleet of simulated turbofan engines. Results demonstrate that while FedAvg is applicable for prognosis, it cannot acknowledge the differences in asset failure mechanisms. As a result, the prognosis framework needs to be modified such that similar failures are clustered together before FedAvg can be implemented.

Snapshot Compressed Sensing: Performance Bounds and Algorithms

Snapshot compressed sensing (CS) refers to compressive imaging systems in which multiple frames are mapped into a single measurement frame. Each pixel in the acquired frame is a noisy linear mapping of the corresponding pixels in the frames that are combined together. While the problem can be cast as a CS problem, due to the very special structure of the sensing matrix, standard CS theory cannot be employed to study such systems. In this paper, a compression-based framework is employed for theoretical analysis of snapshot CS systems. It is shown that this framework leads to two novel, computationally-efficient and theoretically-analyzable compression-based recovery algorithms. The proposed methods are iterative and employ compression codes to define and impose the structure of the desired signal. Theoretical convergence guarantees are derived for both algorithms. In the simulations, it is shown that, in the cases of both noise-free and noisy measurements, combining the proposed algorithms with a customized video compression code, designed to exploit nonlocal structures of video frames, significantly improves the state-of-the-art performance.

Effective Image Retrieval via Multilinear Multi-Index Fusion

Multi-index fusion has demonstrated impressive performances in the retrieval task by integrating different visual representations in a unified framework. However, previous works mainly consider propagating similarities via a neighbor structure, ignoring the high-order information among different visual representations. In this paper, we propose a new multi-index fusion scheme for image retrieval. By formulating this procedure as a multilinear-based optimization problem, the complementary information hidden in different indexes can be explored more thoroughly. Specifically, we first build our multiple indexes from various visual representations. Then, a so-called index-specific functional matrix, which aims to propagate similarities, is introduced to update the original index. The functional matrices are then optimized in a unified tensor space to achieve a refinement, such that the relevant images can be pushed closer. The optimization problem can be efficiently solved by the augmented Lagrangian method with a theoretical convergence guarantee. Unlike the traditional multi-index fusion scheme, our approach embeds the multi-index subspace structure into the new indexes with sparse constraint and, thus, it has little additional memory consumption in the online query stage. Experimental evaluation on three benchmark datasets reveals that the proposed approach achieves state-of-the-art performance, that is, N-score 3.94 on UKBench, mAP 94.1% on Holiday, and 62.39% on Market-1501.

Shrinking horizon parametrized predictive control with application to energy-efficient train operation

A nonlinear model predictive control approach is studied, for problems where a fixed terminal instant and corresponding terminal set to be reached are imposed. The new technique features a shrinking horizon, rather than the most common receding one, and an input parametrization strategy to reduce computational burden. The property of transferability of the parametrization strategy is introduced. Under this property, theoretical convergence guarantees in nominal conditions are obtained by construction. Two relaxed techniques are then proposed to retain recursive feasibility in presence of bounded additive input disturbance. A bound on the constraint violation achieved by these relaxed techniques as a function of the uncertainty bound is derived, too. The developed strategy is applied to the problem of energy-efficient operation of trains, in either a fully autonomous mode (with continuous input values) or a driver assistance mode (with discrete input values, resulting in a nonlinear integer program if no parametrization is used). Realistic simulation results in this context illustrate the effectiveness of the approach.

A game-theoretic approach for price-based coordination of flexible devices operating in integrated energy-reserve markets

This paper presents a novel distributed control strategy for large scale deployment of demand response. In the considered framework, large populations of storage devices and electric vehicles (EVs) participate to an integrated energy-reserve market. They react to prices and autonomously schedule their operation in order to optimize their own objective functions. The price signals are obtained through the resolution of an optimal power flow problem that explicitly takes into account the impact of demand response on the optimal power dispatch and reserve procurement of committed generation. Differently from previous approaches, the adopted game-theoretic framework provides rigorous theoretical guarantees of convergence and optimality of the proposed control scheme in a multi-price setup that includes ancillary services. The performance of the coordination scheme is also evaluated in simulation on the PJM 5-bus system, demonstrating its capability to flatten demand profiles and reduce the costs of generators and flexible devices.

End-to-End Distributed Flow Control for Networks with Nonconcave Utilities

This paper proposes near-optimal decentralized allocation for traffic generated by real-time applications in communication networks. The quality of experience perceived by users in practical applications cannot be accurately modeled using concave functions. Therefore, we tackle the problem of optimizing general nonconcave network utilities. The approach for solving the resulting nonconvex network utility maximization problem relies on designing a sequence of convex relaxations whose solutions converge to a point that characterizes an optimal solution of the original problem. Three different algorithms are designed for solving the proposed convex relaxation, and their theoretical convergence guarantees are studied. All proposed algorithms are distributed in nature, where each user independently controls its traffic in a way that drives the overall network traffic allocation to an optimal operating point subject to resource constraints. All computations required by the algorithms are performed independently and locally at each user using local information available to that user. We highlight the tradeoff between the convergence speed and the network overhead required by each algorithm. Furthermore, we demonstrate the robustness and scalability of these algorithms by showing that traffic is automatically rerouted in case of a link failure or having new users joining the network. Numerical results are presented to validate our findings.

Global Optimization with Orthogonality Constraints via Stochastic Diffusion on Manifold

Orthogonality constrained optimization is widely used in applications from science and engineering. Due to the nonconvex orthogonality constraints, many numerical algorithms often can hardly achieve the global optimality. We aim at establishing an efficient scheme for finding global minimizers under one or more orthogonality constraints. The main concept is based on noisy gradient flow constructed from stochastic differential equations (SDE) on the Stiefel manifold, the differential geometric characterization of orthogonality constraints. We derive an explicit representation of SDE on the Stiefel manifold endowed with a canonical metric and propose a numerically efficient scheme to simulate this SDE based on Cayley transformation with theoretical convergence guarantee. The convergence to global optimizers is proved under second-order continuity. The effectiveness and efficiency of the proposed algorithms are demonstrated on a variety of problems including homogeneous polynomial optimization, computation of stability number, and 3D structure determination from Common Lines in Cryo-EM.

Intelligent Initialization and Adaptive Thresholding for Iterative Matrix Completion: Some Statistical and Algorithmic Theory for Adaptive-Impute

Over the past decade, various matrix completion algorithms have been developed. Thresholded singular value decomposition (SVD) is a popular technique in implementing many of them. A sizable number of studies have shown its theoretical and empirical excellence, but choosing the right threshold level still remains as a key empirical difficulty. This article proposes a novel matrix completion algorithm which iterates thresholded SVD with theoretically justified and data-dependent values of thresholding parameters. The estimate of the proposed algorithm enjoys the minimax error rate and shows outstanding empirical performances. The thresholding scheme that we use can be viewed as a solution to a nonconvex optimization problem, understanding of whose theoretical convergence guarantee is known to be limited. We investigate this problem by introducing a simpler algorithm, generalized- softImpute, analyzing its convergence behavior, and connecting it to the proposed algorithm.

No-Regret Bayesian Optimization with Unknown Hyperparameters

Bayesian optimization (BO) based on Gaussian process models is a powerful paradigm to optimize black-box functions that are expensive to evaluate. While several BO algorithms provably converge to the global optimum of the unknown function, they assume that the hyperparameters of the kernel are known in advance. This is not the case in practice and misspecification often causes these algorithms to converge to poor local optima. In this paper, we present the first BO algorithm that is provably no-regret and converges to the optimum without knowledge of the hyperparameters. During optimization we slowly adapt the hyperparameters of stationary kernels and thereby expand the associated function class over time, so that the BO algorithm considers more complex function candidates. Based on the theoretical insights, we propose several practical algorithms that achieve the empirical sample efficiency of BO with online hyperparameter estimation, but retain theoretical convergence guarantees. We evaluate our method on several benchmark problems.

A Framework for Solving Non-Linear DSGE Models

We propose a framework to solve non-linear DSGE models combining approximation and estimation techniques. Instead of relying on a fixed grid, we use Monte Carlo methods to draw samples from the state space, which are used to estimate an approximation for the value or policy functions of interest. By using estimators from high-dimensional statistics we can attenuate the curse of dimensionality while maintaining flexibility, theoretical guarantees for convergence and upper bound for the errors. In particular, we propose two different methods: a regularized projection and a support vector machine algorithm. To illustrate these solution procedures, we apply the first algorithm to solve a standard growth model, which has a known linear solution, and show that it achieves good accuracy, correctly shrinking the coefficients of a polynomial basis. Moreover, we use the support vector machine algorithm to solve a New Keynesian model with a Zero Lower Bound (ZLB) and compare our results with the ones from the Smolyak Method, which is widely used in the literature. We show that the latter overestimate the impact of the ZLB in the economy, achieving a lower accuracy than the one from our solution.

Knowledge Reused Outlier Detection

Tremendous efforts have been invested in the unsupervised outlier detection research, which is conducted on unlabeled data set with abnormality assumptions. With abundant related labeled data available as auxiliary information, we consider transferring the knowledge from the labeled source data to facilitate the unsupervised outlier detection on target data set. To fully make use of the source knowledge, the source data and target data are put together for joint clustering and outlier detection using the source data cluster structure as a constraint. To achieve this, the categorical utility function is employed to regularize the partitions of target data to be consistent with source data labels. With an augmented matrix, the problem is completely solved by a K-means - a based method with the rigid mathematical formulation and theoretical convergence guarantee. We have used four real-world data sets and eight outlier detection methods of different kinds for extensive experiments and comparison. The results demonstrate the effectiveness and significant improvements of the proposed methods in terms of outlier detection and cluster validity metrics. Moreover, the parameter analysis is provided as a practical guide, and noisy source label analysis proves that the proposed method can handle real applications where source labels can be noisy.

Linearized ADMM for Nonconvex Nonsmooth Optimization With Convergence Analysis

Linearized alternating direction method of multipliers (ADMM) as an extension of ADMM has been widely used to solve linearly constrained problems in signal processing, machine leaning, communications, and many other fields. Despite its broad applications in nonconvex optimization, for a great number of nonconvex and nonsmooth objective functions, its theoretical convergence guarantee is still an open problem. In this paper, we propose a two-block linearized ADMM and a multi-block parallel linearized ADMM for problems with nonconvex and nonsmooth objectives. Mathematically, we present that the algorithms can converge for a broader class of objective functions under less strict assumptions compared with previous works. Furthermore, our proposed algorithm can update coupled variables in parallel and work for less restrictive nonconvex problems, where the traditional ADMM may have difficulties in solving subproblems.

Robust Multi-View Clustering With a Unified Weight Learning Paradigm

Multi-view clustering, which exploits multi-view information to improve the clustering performance has attracted much attention in recent years. However, existing methods seldom consider the diverse quality of data points in different views, and assign each data point with the same importance for clustering. This way degrades the clustering performance due to the interference of low quality data points on the learned clustering indicators. In this paper, a novel robust multi-view clustering method with a unified weight learning paradigm is proposed to address this issue. The unified weight learning paradigm adaptively learns the quality of data points and the clustering capability of each view. Specifically, the reconstruction error of each data point in each view is treated as a factor to depict the quality of data point in this view. Afterwards, the clustering capability of each view is captured from the diverse quality of data points in each view. The clustering capability of each view in turn improves the learning process of data quality. An alternating iterative optimization algorithm with theoretical convergence guarantee and complexity analysis is designed to optimize the objective function. Experimental results on real-world benchmark datasets demonstrate the superiority of the proposed method.