Convex Loss Function Research Articles

FedADMM-InSa: An inexact and self-adaptive ADMM for federated learning

Federated learning (FL) is a promising framework for learning from distributed data while maintaining privacy. The development of efficient FL algorithms encounters various challenges, including heterogeneous data and systems, limited communication capacities, and constrained local computational resources. Recently developed FedADMM methods show great resilience to both data and system heterogeneity. However, they still suffer from performance deterioration if the hyperparameters are not carefully tuned. To address this issue, we propose an inexact and self-adaptive FedADMM algorithm, termed FedADMM-InSa. First, we design an inexactness criterion for the clients’ local updates to eliminate the need for empirically setting the local training accuracy. This inexactness criterion can be assessed by each client independently based on its unique condition, thereby reducing the local computational cost and mitigating the undesirable straggle effect. The convergence of the resulting inexact ADMM is proved under the assumption of strongly convex loss functions. Additionally, we present a self-adaptive scheme that dynamically adjusts each client’s penalty parameter, enhancing algorithm robustness by mitigating the need for empirical penalty parameter choices for each client. Extensive numerical experiments on both synthetic and real-world datasets have been conducted. As validated by some tests, our FedADMM-InSa algorithm improves model accuracy by 7.8% while reducing clients’ local workloads by 55.7% compared to benchmark algorithms.

Group inference of high-dimensional single-index models

For the supervised and semi-supervised settings, a group inference method is proposed for regression parameters in high-dimensional semi-parametric single-index models with an unknown random link function. The inference procedure is based on least squares, which can be extended to other general convex loss functions. The proposed test statistics are weighted quadratic forms of the regression parameter estimates, in which the weight could be a non-random matrix or the sample covariance matrix of the covariates. The proposed method could detect dense but weak signals and deal with high correlation of covariates inside the group. A ‘contaminated test statistic’ is established in the semi-supervised regime to decrease the variance. The asymptotic properties of the resulting estimators are established. The finite-sample behaviour of the proposed method is evaluated through extensive simulation studies. Applications to two genomic datasets are provided.

Large-Scale Non-convex Stochastic Constrained Distributionally Robust Optimization

Distributionally robust optimization (DRO) is a powerful framework for training robust models against data distribution shifts. This paper focuses on constrained DRO, which has an explicit characterization of the robustness level. Existing studies on constrained DRO mostly focus on convex loss function, and exclude the practical and challenging case with non-convex loss function, e.g., neural network. This paper develops a stochastic algorithm and its performance analysis for non-convex constrained DRO. The computational complexity of our stochastic algorithm at each iteration is independent of the overall dataset size, and thus is suitable for large-scale applications. We focus on the general Cressie-Read family divergence defined uncertainty set which includes chi^2-divergences as a special case. We prove that our algorithm finds an epsilon-stationary point with an improved computational complexity than existing methods. Our method also applies to the smoothed conditional value at risk (CVaR) DRO.

Linearized alternating direction method of multipliers for elastic-net support vector machines

In many high-dimensional datasets, the phenomenon that features are relevant often occurs. Elastic-net regularization is widely used in support vector machines (SVMs) because it can automatically perform feature selection and encourage highly correlated features to be selected or removed together. Recently, some effective algorithms have been proposed to solve the elastic-net SVMs with different convex loss functions, such as hinge, squared hinge, huberized hinge, pinball and huberized pinball. In this paper, we develop a linearized alternating direction method of multipliers (LADMM) algorithm to solve above elastic-net SVMs. In addition, our algorithm can be applied to solve some new elastic-net SVMs such as elastic-net least squares SVM. Compared with some existing algorithms, our algorithm has comparable or better performances in terms of computational cost and accuracy. Under mild conditions, we prove the convergence and derive convergence rate of our algorithm. Furthermore, numerical experiments on synthetic and real datasets demonstrate the feasibility and validity of the proposed algorithm.

From big data to smart data: a sample gradient descent approach for machine learning

This research paper presents an innovative approach to gradient descent known as ‘‘Sample Gradient Descent’’. This method is a modification of the conventional batch gradient descent algorithm, which is often associated with space and time complexity issues. The proposed approach involves the selection of a representative sample of data, which is subsequently subjected to batch gradient descent. The selection of this sample is a crucial task, as it must accurately represent the entire dataset. To achieve this, the study employs the use of Principle Component Analysis (PCA), which is applied to the training data, with a condition that only those rows and columns of data that explain 90% of the overall variance are retained. This approach results in a convex loss function, where a global minimum can be readily attained. Our results indicate that the proposed method offers faster convergence rates, with reduced computation times, when compared to the conventional batch gradient descent algorithm. These findings demonstrate the potential utility of the ‘‘Sample Gradient Descent’’ technique in various domains, ranging from machine learning to optimization problems. In our experiments, both approaches were run for 30 epochs, with each epoch taking approximately 3.41 s. Notably, our ‘‘Sample Gradient Descent’’ approach exhibited remarkable performance, converging in just 8 epochs, while the conventional batch gradient descent algorithm required 20 epochs to achieve convergence. This substantial difference in convergence rates, along with reduced computation times, highlights the superior efficiency of our proposed method. These findings underscore the potential utility of the ‘‘Sample Gradient Descent’’ technique across diverse domains, ranging from machine learning to optimization problems. The significant improvements in convergence rates and computation times make our algorithm particularly appealing to practitioners and researchers seeking enhanced efficiency in gradient descent optimization.

Gradient complexity and non-stationary views of differentially private empirical risk minimization

In this paper, we study the Differentially Private Empirical Risk Minimization (DP-ERM) problem, considering both convex and non-convex loss functions. For cases where DP-ERM involves smooth (strongly) convex loss functions with or without non-smooth regularization, we propose several novel methods. These methods achieve (near) optimal expected excess risks (i.e., utility bounds) while reducing the gradient complexity compared to existing approaches. When dealing with DP-ERM and smooth convex loss functions in high dimensions, we introduce an algorithm that achieves a superior upper bound with lower gradient complexity than previous solutions.In the second part of the paper, for DP-ERM with non-convex loss functions, we explore both low and high dimensional spaces. In the low dimensional case with a non-smooth regularizer, we extend an existing approach by measuring the utility using the ℓ2 norm of the projected gradient. Furthermore, we introduce a novel error bound measurement, transitioning from empirical risk to population risk by employing the expected ℓ2 norm of the gradient. For the high dimensional case, we demonstrate that by measuring utility with the Frank-Wolfe gap, we can bound the utility using the Gaussian Width of the constraint set, instead of the dimensionality (p) of the underlying space. We also show that the advantages of this approach can be achieved by measuring the ℓ2 norm of the projected gradient. Finally, we reveal that the utility of certain special non-convex loss functions can be reduced to a level (depending only on log⁡p) similar to that of convex loss functions.

Generalization Analysis of CNNs for Classification on Spheres.

Deep learning based on deep convolutional neural networks (CNNs) is extremely efficient in solving classification problems in speech recognition, computer vision, and many other fields. But there is no enough theoretical understanding about this topic, especially the generalization ability of the induced CNN algorithms. In this article, we develop some generalization analysis of a deep CNN algorithm for binary classification with data on spheres. An essential property of the classification problem is the lack of continuity or high smoothness of the target function associated with a convex loss function such as the hinge loss. This motivates us to consider the approximation of functions in the Lp space with 1≤ p ≤ ∞. We provide rates of Lp -approximation when the approximated function lies in a Sobolev space and then present generalization bounds and learning rates for the excess misclassification error of the deep CNN classification algorithm. Our novel analysis is based on efficient cubature formulae on spheres and other tools from spherical analysis and approximation theory.

Recurrent Neural Network Training With Convex Loss and Regularization Functions by Extended Kalman Filtering

This paper investigates the use of extended Kalman filtering to train recurrent neural networks with rather general convex loss functions and regularization terms on the network parameters, including <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\ell _{1}$</tex-math></inline-formula> -regularization. We show that the learning method is competitive with respect to stochastic gradient descent in a nonlinear system identification benchmark and in training a linear system with binary outputs. We also explore the use of the algorithm in data-driven nonlinear model predictive control and its relation with disturbance models for offset-free closed-loop tracking.

Online distributed detection of sensor networks with delayed information

Distributed decision on sensor network is affected by the bandwidth, network congestion, and computation capability of sensors, which cause the communication delays and feedback delays during the signal processing process. In this paper, the effects of state constraints, directed graph, communication delays and feedback delays on the online distributed parameter estimation problem for multi-sensor networks are considered comprehensively. To this end, we first propose an Online Decentralized Gradient Descent Algorithm (ODGDA) to overcome the effects of these factors and give the regret upper bounds for convex loss function and strongly convex loss function, respectively. Further, considering that the loss function may not be fully disclosed in the complex environment, we extend ODGDA to the scenario of bandit feedback and propose an Online Decentralized Bandit Feedback Algorithm (ODBFA), which is updated using historical state information stored by sensors. The analysis shows that, compared with the ODGDA which can directly use the gradient information for state update, the ODBFA utilizing two-point bandit feedback information does not lead to regret degradation in the order sense. Finally, the effectiveness of the proposed algorithm is verified by the multi-sensor network.

Asymptotic linear convergence of fully-corrective generalized conditional gradient methods

AbstractWe proposea fully-corrective generalized conditional gradient method(FC-GCG) for the minimization of the sum of a smooth, convex loss function and a convex one-homogeneous regularizer over a Banach space. The algorithm relies on the mutual update of a finite set $$\mathcal {A}_k$$Akof extremal points of the unit ball of the regularizer and of an iterate $$u_k \in {\text {cone}}(\mathcal {A}_k)$$uk∈cone(Ak). Each iteration requires the solution of one linear problem to update $$\mathcal {A}_k$$Akand of one finite dimensional convex minimization problem to update the iterate. Under standard hypotheses on the minimization problem we show that the algorithm converges sublinearly to a solution. Subsequently, imposing additional assumptions on the associated dual variables, this is improved to a linear rate of convergence. The proof of both results relies on two key observations: First, we prove the equivalence of the considered problem to the minimization of a lifted functional over a particular space of Radon measures using Choquet’s theorem. Second, the FC-GCG algorithm is connected to aPrimal-Dual-Active-Point methodon the lifted problem for which we finally derive the desired convergence rates.

Meta-Inductive Probability Aggregation

There is a plurality of formal constraints for aggregating probabilities of a group of individuals. Different constraints characterise different families of aggregation rules. In this paper, we focus on the families of linear and geometric opinion pooling rules which consist in linear, respectively, geometric weighted averaging of the individuals’ probabilities. For these families, it is debated which weights exactly are to be chosen. By applying the results of the theory of meta-induction, we want to provide a general rationale, namely, optimality, for choosing the weights in a success-based way by scoring rules. A major argument put forward against weighting by scoring is that these weights heavily depend on the chosen scoring rule. However, as we will show, the main condition for the optimality of meta-inductive weights is so general that it holds under most standard scoring rules, more precisely under all scoring rules that are based on a convex loss function. Therefore, whereas the exact choice of a scoring rule for weighted probability aggregation might depend on the respective purpose of such an aggregation, the epistemic rationale behind such a choice is generally valid.

Regret and Cumulative Constraint Violation Analysis for Distributed Online Constrained Convex Optimization

This paper considers the distributed online convex optimization problem with time-varying constraints over a network of agents. This is a sequential decision making problem with two sequences of arbitrarily varying convex loss and constraint functions. At each round, each agent selects a decision from the decision set, and then only a portion of the loss function and a coordinate block of the constraint function at this round are privately revealed to this agent. The goal of the network is to minimize the network-wide loss accumulated over time. Two distributed online algorithms with full-information and bandit feedback are proposed. Both dynamic and static network regret bounds are analyzed for the proposed algorithms, and network cumulative constraint violation is used to measure constraint violation, which excludes the situation that strictly feasible constraints can compensate the effects of violated constraints. In particular, we show that the proposed algorithms achieve $\mathcal{O}(T^{\max\{\kappa,1-\kappa\}})$ static network regret and $\mathcal{O}(T^{1-\kappa/2})$ network cumulative constraint violation, where $T$ is the time horizon and $\kappa\in(0,1)$ is a user-defined trade-off parameter. Moreover, if the loss functions are strongly convex, then the static network regret bound can be reduced to $\mathcal{O}(T^{\kappa})$. Finally, numerical simulations are provided to illustrate the effectiveness of the theoretical results.

Clustered Federated Learning Based on Momentum Gradient Descent for Heterogeneous Data

Data heterogeneity may significantly deteriorate the performance of federated learning since the client’s data distribution is divergent. To mitigate this issue, an effective method is to partition these clients into suitable clusters. However, existing clustered federated learning is only based on the gradient descent method, which leads to poor convergence performance. To accelerate the convergence rate, this paper proposes clustered federated learning based on momentum gradient descent (CFL-MGD) by integrating momentum and cluster techniques. In CFL-MGD, scattered clients are partitioned into the same cluster when they have the same learning tasks. Meanwhile, each client in the same cluster utilizes their own private data to update local model parameters through the momentum gradient descent. Moreover, we present gradient averaging and model averaging for global aggregation, respectively. To understand the proposed algorithm, we also prove that CFL-MGD converges at an exponential rate for smooth and strongly convex loss functions. Finally, we validate the effectiveness of CFL-MGD on CIFAR-10 and MNIST datasets.

Robust and optimal epsilon-insensitive Kernel-based regression for general noise models

Sparse representation of kernel based regression (KBR) has received considerable attention in recent years. Studies on sparse KBR can be divided into two distinct groups, namely (i) pruning-based methods that remove the training samples with the least training errors and retrain the remaining training samples, and (ii) direct methods that begin with a full-dense solution and delete training data according to objective criteria. Pruning-based methods give rise to a high computation time, while direct methods may lead to non-optimal solutions and thus a poor approximation. In addition, most current KBR models assume that the error distribution is Gaussian. However, observations in many practical applications indicate that the noise models do not satisfy the Gaussian error distribution. In such cases, current KBR models are not optimal. To address the above-mentioned problems, this study proposes a new sparse KBR framework for general noise distributions, including the epsilon-insensitive noise family. Compared with other sparse algorithms, sparsity is directly imposed by epsilon-insensitive convex loss functions derived from the theoretical framework of the Bayesian approach within the scope of regularization networks, and then handles the optimization problem in Lagrangian form. Experiments on artificial and real-life benchmark datasets demonstrate that the proposed epsilon-insensitive KBR models are more effective and efficient than pruning-based approaches.

On the Convergence of Hybrid Server-Clients Collaborative Training

Modern distributed machine learning (ML) paradigms, such as federated learning (FL), utilize data distributed at different clients to train a global model. In such paradigm, local datasets never leave the clients for better privacy protection, and the parameter server (PS) only performs simple aggregation. In practice, however, there is often some amount of data available at the PS, and its computation capability is strong enough to carry out more demanding tasks than simple model aggregation. The focus of this paper is to analyze the model convergence of a new hybrid learning architecture, which leverages the PS dataset and its computation power for collaborative model training with clients. Different from FL where stochastic gradient descent (SGD) is always computed in parallel across clients, the new architecture has both parallel SGD at clients and sequential SGD at PS. We analyze the convergence rate upper bounds of this <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">aggregate-then-advance</i> design for both strongly convex and non-convex loss functions. We show that when the local SGD has an <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mathcal {O}(1/t)$ </tex-math></inline-formula> stepsize, the server SGD needs to scale its stepsize to no slower than <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\mathcal {O}(1/t^{2})$ </tex-math></inline-formula> in order to strictly outperform local SGD with strongly convex loss functions. The theoretical findings are corroborated by numerical experiments, where advantages in terms of both accuracy and convergence speed over clients-only (local SGD and FED AVG) and server-only training are demonstrated.

The optimal dynamic regret for smoothed online convex optimization with squared [formula omitted] norm switching costs

Online convex optimization (OCO) with switching costs is a key enabler for cloud resource provision, online portfolio optimization, and many other applications. Surprisingly, very little theoretical understanding is known. In this study, we investigate OCO with squared l2 norm switching cost (OCOl2SC) for three kinds of loss functions: (a) generally convex, (b) convex and smooth, and (c) strongly convex and smooth. We design customized gradient descent algorithms for OCOl2SC in these three cases: specifically, SOGD (smoothed online gradient descent) for generally convex loss functions, OOMD (online optimistic mirror descent) for convex and smooth functions, and OMGD (online multiple gradient descent) for strongly convex and smooth loss functions. We theoretically analyze the three algorithms’ dynamic regrets and their upper bounds. By showing that the dynamic regrets match their lower bounds, we conclude that the three algorithms achieve the order optimal or near-optimal dynamic regret bounds in the corresponding case. Numerical studies further verify the remarkable performance of the three algorithms.

Online Learning of Time-Varying Unbalanced Networks in Non-Convex Environments: A Multi-Armed Bandit Approach

This study discusses how agents in a time-varying distributed network can converge to the global minimizer of a time-varying graph network. Each agent knows only the local loss of its observation and must cooperate constructively with other agents to find the global minimizer of the network. Unlike most existing works in the literature that consider a convex loss function, this study assumes a generalized local Lipschitz loss function for each agent, which can be convex or non-convex. We propose a multi-armed bandit algorithm CD EXP3 where each agent does not know its loss function but only observes its losses. Through simulations using two different time-varying graph topologies, we show that the algorithm helps all agents converge to the minimizer of the network. In addition, we discuss the effects of the two different topologies and various simulation parameters on convergence. We obtain an upper bound on the expected regret and compare it with the sublinearity of the regret bounds of well-known online distributed algorithms.

Federated learning for minimizing nonsmooth convex loss functions

Federated learning possesses a distributed learning framework for protecting privacy where local clients need to collaboratively train a shared model via a central server. Many existing methods in the literature of federated learning are analyzed under some smoothness conditions of loss functions and need access to gradient information to perform local optimization algorithms such as stochastic gradient descent or dual average. Some methods even need a strong convexity of the loss function. However, in many situations of real world applications such as readability of texts, first-order gradient information is difficult to obtain, and the strong smoothness and strong convexity of loss functions are not satisfied. We consider methods to overcome such situations. This paper aims at providing an understanding of federated learning in the situation that the loss functions are nonsmooth and gradient information is unavailable, also a strong convexity condition is not needed. Based on Nesterov's zeroth-order (gradient-free) techniques, we propose a zeroth-order stochastic federated learning method. Constant and decreasing step size strategies are considered. Moreover, a new type of approximating sequence is proposed in federated learning for strictly decreasing step sizes. Expected error bounds for the proposed approximating sequence and learning rates of the proposed method are derived under some selection rules of the step sizes and smoothing parameters.

Large-scale machine learning with fast and stable stochastic conjugate gradient

In deterministic optimization, conjugate gradient (CG) type approaches are preferred with a superior convergence rate than the ordinary gradient approaches. The requirement of solving large-scale data, growing exponentially, makes recent works study the effectiveness of the CG-type approaches with stochastic approximation, especially for large-scale machine learning problems. However, it is challenging that how to incorporate the noisy gradients into CG-type approaches. In this paper, we develop a class of fast and robust stochastic conjugate gradient (SCG) type approach via using the stochastic recursive gradient algorithm (SARAH) and the hyper-gradient descent (HD) technique in the mini-batching setting. That the use of the SARAH gradient estimator makes the proposed approaches enjoy the low variance accelerates the convergence rate and saves the gradient complexity of the conventional SCG-type approach. In addition, using HD to determine the learning rate for the SCG-type approach greatly saves the computational burden, comparing with the existing literature that usually works with the line search technique in practice. We rigorously prove that the proposed approach attains a linear convergence rate for strongly convex loss functions and show that its complexity matches modern stochastic optimization approaches. Various experimental results on machine learning problems are provided to demonstrate the property and the effectiveness of the proposed approaches respectively.

High-accuracy low-cost privacy-preserving federated learning in IoT systems via adaptive perturbation

With the rapid development of the Internet of Things (IoT), federated learning (FL) has been widely used to obtain insights from collected data while preserving data privacy. Differential privacy (DP) is an additive noise scheme that has been widely studied as a privacy-preserving approach on FL. However, privacy protection under DP usually comes at the cost of model accuracy for the underlying FL process. In this paper, we propose a novel low-cost (for both communication and computational overhead) adaptive noise perturbation/masking scheme to protect FL clients’ privacy without degrading the global model accuracy. In particular, we set the magnitude of the additive noise to adaptively change with the magnitude of the local model updates. Then, a direction-based filtering scheme is used to accelerate the convergence of the FL model. A maximum tolerable noise bound for local clients is derived using the central limit theorem (CLT). The designed noise maximizes privacy protection for clients while preserving the accuracy and convergence rate of the FL model , as a result of the noise cancelling out and forming a more concentrated distribution after the aggregation operation on the server. We theoretically prove that FL with the proposed noise perturbation scheme retains the same accuracy and convergence rate (O(1/T) for convex loss functions and O(1/T) for non-convex loss functions) as that of non-private FL with SGD. We also evaluate the performance of the proposed scheme in terms of convergence behavior, computational efficiency, and privacy protection against state-of-the-art privacy inference attacks on real-world datasets. Experimental results show that FL with our proposed perturbation scheme outperforms DP in the accuracy and convergence rate of the FL model in both client dropout and non-client dropout scenarios. Compared with DP, our proposed scheme does not incur additional computational and communication overhead. Our approach provides DP-comparable or better effectiveness in defending against privacy attacks under the same global model accuracy.