Accelerated Proximal Gradient Research Articles

Enhancing Label Correlations in multi-label classification through global-local label specific feature learning to Fill Missing labels

In multi-label classification, challenges arise from missing labels due to subjective analysis or label ambiguity. This makes it difficult to accurately capture label correlations and enhance classifier performance. Previous research has primarily focused on obtaining label correlations with sparsity assumptions from incomplete label matrices to fill in these missing labels. However, this approach leads to inaccurate label correlations since it relies on incomplete label matrix information. To overcome this issue and improve accuracy, this paper proposes a new method called LCFM, which effectively learns label correlations using a combination of global and local label-specific features to recover missing labels. Firstly, we rely on the sparsity assumption of Lasso regression coefficients to select label-specific features from the training samples. Secondly, we introduce l2-norm regularization to learn global label correlations through global label-specific features. simultaneously, we utilize similar label-specific features to describe local label correlations. Finally, we minimize the difference between predicted results and true labels using the binary cross-entropy loss function and employ the Accelerated Proximal Gradient (APG) algorithm for model optimization. Experimental results comparing LCFM with five benchmark indicators on nine datasets demonstrate that our proposed method outperforms other advanced methods, showing strong competitiveness.

Dynamic High-Resolution Network for Semantic Segmentation in Remote-Sensing Images

Semantic segmentation of remote-sensing (RS) images is one of the most fundamental tasks in the understanding of a remote-sensing scene. However, high-resolution RS images contain plentiful detailed information about ground objects, which scatter everywhere spatially and have variable sizes, styles, and visual appearances. Due to the high similarity between classes and diversity within classes, it is challenging to obtain satisfactory and accurate semantic segmentation results. This paper proposes a Dynamic High-Resolution Network (DyHRNet) to solve this problem. Our proposed network takes HRNet as a super-architecture, aiming to leverage the important connections and channels by further investigating the parallel streams at different resolution representations of the original HRNet. The learning task is conducted under the framework of a neural architecture search (NAS) and channel-wise attention module. Specifically, the Accelerated Proximal Gradient (APG) algorithm is introduced to iteratively solve the sparse regularization subproblem from the perspective of neural architecture search. In this way, valuable connections are selected for cross-resolution feature fusion. In addition, a channel-wise attention module is designed to weight the channel contributions for feature aggregation. Finally, DyHRNet fully realizes the dynamic advantages of data adaptability by combining the APG algorithm and channel-wise attention module simultaneously. Compared with nine classical or state-of-the-art models (FCN, UNet, PSPNet, DeepLabV3+, OCRNet, SETR, SegFormer, HRNet+FCN, and HRNet+OCR), DyHRNet has shown high performance on three public challenging RS image datasets (Vaihingen, Potsdam, and LoveDA). Furthermore, the visual segmentation results, the learned structures, the iteration process analysis, and the ablation study all demonstrate the effectiveness of our proposed model.

Reducing the Complexity of Two Classes of Optimization Problems by Inexact Accelerated Proximal Gradient Method

We propose a double-loop inexact accelerated proximal gradient (APG) method for a strongly convex composite optimization problem with two smooth components of different smoothness constants and computational costs. Compared to APG, the inexact APG can reduce the time complexity for finding a near-stationary point when one smooth component has higher computational cost but a smaller smoothness constant than the other. The strongly convex composite optimization problem with this property arises from subproblems of a regularized augmented Lagrangian method for affine-constrained composite convex optimization and also from the smooth approximation for bilinear saddle-point structured nonsmooth convex optimization. We show that the inexact APG method can be applied to these two problems and reduce the time complexity for finding a near-stationary solution. Numerical experiments demonstrate significantly higher efficiency of our methods over an optimal primal-dual first-order method by Hamedani and Aybat [SIAM J. Optim., 31 (2021), pp. 1299–1329] and the gradient sliding method by Lan, Ouyang, and Zhou [arXiv2101.00143, 2021].

Subspace screening rule for multi-label estimator with sparsity-inducing regularization

Multi-label learning has drawn wide attention for the last decade. To exploit the correlation between labels, a multi-label classifier based on the nuclear norm has been proposed recently, which joints Ranking support vector machine (RankSVM) and Binary Relevance (BR) with robust Low-rank learning (RBRL). Therefore, it has satisfactory classification outcomes in the application. Nonetheless, tackling the large-scale problem still remains a challenge for RBRL. Motivated by this, a Subspace Screening Rule (SSR) for RBRL is proposed to accelerate the solving process. Its primary strategy is to reduce the size of the matrix variable to be estimated, based on the fact that a low-rank matrix can be represented by a few subspaces. Specifically, at each iteration we delete a majority of subspaces with zero coefficients in the optimal solution by matrix decomposition and optimality condition. Then, we solve the small-scale reduced problem rather than the initial large-scale matrix problem. An excellent acceleration effect is obtained. To further improve the solving speed, Approximate Singular Value Decomposition (ASVD) and Accelerated Proximal Gradient (APG) are employed in the different stages. Extensive experiments on seven benchmark datasets as well as an artificial dataset demonstrate the efficiency of SSR.

Exploiting task relationships for Alzheimer’s disease cognitive score prediction via multi-task learning

Alzheimer’s disease (AD) is highly prevalent and a significant cause of dementia and death in elderly individuals. Motivated by breakthroughs of multi-task learning (MTL), efforts have been made to extend MTL to improve the Alzheimer’s disease cognitive score prediction by exploiting structure correlation. Though important and well-studied, three key aspects are yet to be fully handled in an unified framework: (i) appropriately modeling the inherent task relationship; (ii) fully exploiting the task relatedness by considering the underlying feature structure. (iii) automatically determining the weight of each task. To this end, we present the Bi-Graph guided self-Paced Multi-Task Feature Learning (BGP-MTFL) framework for exploring the relationship among multiple tasks to improve overall learning performance of cognitive score prediction. The framework consists of the two correlation regularization for features and tasks, ℓ2,1 regularization and self-paced learning scheme. Moreover, we design an efficient optimization method to solve the non-smooth objective function of our approach based on the Alternating Direction Method of Multipliers (ADMM) combined with accelerated proximal gradient (APG). The proposed model is comprehensively evaluated on the Alzheimer’s disease neuroimaging initiative (ADNI) datasets. Overall, the proposed algorithm achieves an nMSE (normalized Mean Squared Error) of 3.923 and an wR (weighted R-value) of 0.416 for predicting eighteen cognitive scores, respectively. The empirical study demonstrates that the proposed BGP-MTFL model outperforms the state-of-the-art AD prediction approaches and enables identifying more stable biomarkers.

Robust Multivariate Lasso Regression with Covariance Estimation

Multivariate regression with covariance estimation (MRCE) is a method that performs sparse estimation of multivariate regression coefficients, while taking account the covariance structure of the response variables. MRCE uses a penalized likelihood approach to simultaneously estimate the regression coefficients and the inverse covariance matrix so that prediction accuracy can be significantly improved. However, traditional likelihood-based methods such as MRCE can produce very misleading results in the presence of outliers. In this work, we propose an extension of MRCE, namely, a robust multivariate lasso regression with covariance estimation (RMLC) to handle potential outliers within the data. By using Huber’s loss or Tukey’s biweight loss, RMLC can be resistant to outliers in the responses or in both the responses and the covariates. A novel optimization algorithm that incorporates a 2-fold accelerated proximal gradient (APG) algorithm is developed to solve RMLC efficiently. We also demonstrate that our proposed RMLC enjoys the oracle property. Our simulation study shows that RMLC produces very reliable results for both the regression coefficients and the correlation structure of the responses, even if the data are contaminated. A real analysis on hyperspectral data further demonstrates the utility of RMLC. Supplementary materials for this article are available online.

Robust support vector machine classifier with truncated loss function by gradient algorithm

The support vector machine (SVM) is an increasingly important tool in machine learning. Despite its popularity, the SVM classifier can be adversely affected under the presence of noise in the training dataset. The SVM can be fit in the regularization framework of Loss ＋ Penalty. The loss function plays an essential role which is used to keep the fidelity of the resulting model to the data. Most SVMs use convex losses, however, they often suffer from the negative impact of points far away from their own classes. This paper proposes a new nonconvex differentiable loss, namely huberied truncated pinball loss, which can be able to reduce the effects of noise in the training sample. The SVM classifier with the huberied truncated pinball loss (HTPSVM) is proposed. The HTPSVM combines the elastic net penalty and the nonconvex huberied truncated pinball loss. It inherits the benefits of both ℓ1 and ℓ2 norm regularizers. The HTPSVM involves nonconvex minimization, the accelerated proximal gradient (APG) algorithm was used to solve the corresponding optimization. To evaluate the performance of classifiers, classification accuracy and area under ROC curve (AUC) were employed as the accuracy indicators. The numerical results show that our new classifier is effective. Friedman and Nemenyi post hoc tests of the experimental results indicate that the proposed HTPSVM is shown to be more robust to noise than HSVM, PSVM and HHSVM.

Sparse and risk diversification portfolio selection.

Portfolio risk management has become more important since some unpredictable factors, such as the 2008 financial crisis and the recent COVID-19 crisis. Although the risk can be actively managed by risk diversification, the high transaction cost and managerial concerns ensue by over diversifying portfolio risk. In this paper, we jointly integrate risk diversification and sparse asset selection into mean-variance portfolio framework, and propose an optimal portfolio selection model labeled as JMV. The weighted piecewise quadratic approximation is considered as a penalty promoting sparsity for the asset selection. The variance associated with the marginal risk regard as another penalty term to diversify the risk. By exposing the feature of JMV, we prove that the KKT point of JMV is the local minimizer if the regularization parameter satisfies a mild condition. To solve this model, we introduce the accelerated proximal gradient (APG) algorithm [Wen in SIAM J. Optim 27:124–145, 2017], which is one of the most efficient first-order large-scale algorithm. Meanwhile, the APG algorithm is linearly convergent to a local minimizer of the JMV model. Furthermore, empirical analysis consistently demonstrate the theoretical results and the superiority of the JMV model.

Cross-spectral matrix denoising for beamforming based on Schatten-p norm

Acoustic imaging is often severely affected by incoherent noise. However, whether it is Diagonal Removal (DR) or Diagonal Denoising (DD), these methods only remove the noise concentrated on the diagonal of the Cross-Spectral Matrix (CSM). In this work, a Robust Principal Component Analysis (RPCA) based on the non-convex Schatten-p (0<p<1) norm is proposed for CSM denoising. Non-convex accelerated proximal gradient (APG) and accelerated proximal gradient line-search-like (APGL) are combined to form a non-convex APGL algorithm to solve CSM denoising based on the Schatten-p norm. This problem is also solved under the framework of the alternate direction method of multipliers (ADMM) algorithms. According to simulation data, the Schatten-p norm has a more accurate reconstruction ability than the nuclear norm at low SNR. The specific performance is that the reconstruction error is smaller, and the rank estimation of the CSM is more accurate. The Schatten-p norm can effectively remove the influence of incoherent noise. Moreover, the Schatten-p norm is more beneficial in reducing the sensitivity of the regularization parameter than the nuclear norm. Experimental results show that the proposed method is an accurate CSM denoising algorithm, which can effectively remove incoherent noise pollution under low SNR conditions.

Twin robust matrix machine for intelligent fault identification of outlier samples in roller bearing

In the industrial processes, the intelligent fault diagnosis related to signal analysis and pattern recognition is an important step to ensure the health of mechanical equipment. A popular intelligent monitor method as it has been, support matrix machine (SMM) enables to use the two-dimensional features extracted from vibration signals to build model. The core of the SMM is to extract structure information within matrix by minimizing nuclear norm to approximate the rank of the matrix. However, the nuclear norm has limited performance to eliminate the noise contained in structure information. Furthermore, features extracted from vibration signals often become outliers, and SMM is sensitive to outliers in training data. Therefore, a novel nonparallel classifier called twin robust matrix machine (TRMM) is proposed and applied to roller bearing fault diagnosis. TRMM can not only fully leverage the low-rank structure information, but also has the following novelties. First, TRMM uses the truncated nuclear norm as the low-rank constraint, to pay more attention to the large singular values related to the main structure information. Further, the ramp loss is used in TRMM as the loss function, which reduces the loss penalty for outlier sample and make TRMM insensitive to outlier samples. Finally, the accelerated proximal gradient (APG) is devised to solve the resulting optimization problem. Experimental results show that the proposed method has excellent fault diagnosis performance, especially in the case of existing outlier samples.

Region proposal patch-image model for infrared small target detection

ABSTRACT The infrared patch-image (IPI) model has been proved to be very successful in infrared small target detection. However, the IPI model suffers from two major issues: abundant of residuals in the recovered target image and low detection speed caused by the large-size low-rank decomposition matrix. To mitigate these challenges, a novel infrared small target detection method called the region proposal patch-image (RPPI) model is proposed. The method contains two stages, where the first one is to generate region proposals (RPs) based on the contrast mechanism and the second one is to recover targets based on matrix decomposition. In the first stage, RPs containing interest targets are extracted in order to suppress background clutters and reduce the size of the decomposed matrix. In RP extraction, the contrast mechanism instead of the original convolutional neural network (CNN) is adapted due to lack of texture features for infrared small target and high computational complexity with CNN. In the second stage, considering the weakness of the correlation between RPs, a weighted nuclear norm (WNN) is utilized for avoiding relaxation of the sparsity too much, which can preserve targets and suppress the background simultaneously. Finally, accelerated proximal gradient (APG)-combined WNN is applied to solve this model. Experimental results on eight image groups with various kinds of targets and backgrounds demonstrate that the proposed method can effectively detect the target with faster speed, higher detection rate, and lower false alarm rate compared with other state-of-the-art methods.

Total Variation-Based Interframe Infrared Patch-Image Model for Small Target Detection

Infrared (IR) small target detection is one of the most fundamental techniques in the infrared search and track (IRST) system. Due to the interferences caused by background clutter and image noise, conventional IR small target detection algorithms always suffer from a high false alarm rate and are unable to achieve robust performance in complex scenes. To accurately distinguish IR small target from the background, we propose a total variation (TV)-based interframe infrared patch-image model that regards the long-distance IR small target detection task as an optimization problem. First, the input IR image is converted to a patch-image that consists of a sparse target matrix and a low-rank background matrix. Then, the interframe similarity of target appearance is utilized to impose a temporal consistency constraint on the target matrix. Next, a TV regularization term is proposed to further alleviate the false alarms generated by noise. Finally, an alternating optimization algorithm using singular value decomposition (SVD) and accelerated proximal gradient (APG) is designed to mathematically solve the proposed model. Both qualitative and quantitative experiments implemented on real IR sequences demonstrate that our model outperforms other traditional IR small target methods in terms of the signal-to-clutter ratio gain (SCRG) and the background suppression factor (BSF).

Phase Shift Optimization Algorithm for Achievable Rate Maximization in Reconfigurable Intelligent Surface-Assisted THz Communications

Terahertz (THz) band communications are considered a crucial technology to support future applications, such as ultra-high bit rate wireless local area networks, in the next generation of wireless communication systems. In this work, we consider an ultra-massive multiple-input multiple-output (UM-MIMO) THz communication system operating in a typical indoor scenario where the direct link between the transmitter and receiver is obstructed due to surrounding obstacles. To help establish communication, we assume the aid of a nearby reconfigurable intelligent surface (RIS) whose phase shifts can be adjusted. To configure the individual phase shifts of the RIS elements, we formulate the problem as a constrained achievable rate maximization. Due to the typical large dimensions of this optimization problem, we apply the accelerated proximal gradient (APG) method, which results in a low complexity algorithm that copes with the non-convex phase shift constraint through simple element-wise normalization. Our numerical results demonstrate the effectiveness of the proposed algorithm even when considering realistic discrete phase shifts’ quantization and imperfect channel knowledge. Furthermore, comparison against existing alternatives reveals improvements between 30% and 120% in terms of range, for a reference rate of 100 Gbps when using the proposed approach with only 81 RIS elements.

Energy-efficient power allocation in cell-free massive MIMO with zero-forcing: First order methods

In this paper, the downlink of cell-free massive multiple-input multiple-output (MIMO) with zero-forcing processing is considered. To maximize the system energy efficiency (EE), we design power allocation algorithms taking into account imperfect channel state information, hardware, and backhaul power consumption. The total EE optimization problem is nonconvex, which traditionally is solved by the successive convex approximation framework which involves second order cone programs (SOCPs). As such methods have high complexity, the run time is extremely long, especially in large-scale systems with thousands of access points (APs) and users. To overcome this problem, in this paper, we propose to apply two computationally efficient methods, namely proximal gradient (PG) method and accelerated proximal gradient (APG) method to solve the considered problem. Numerical results show that, compared to the conventional SOCPs approximation methods, our proposed methods achieve the same performance while the run time is much smaller.

A symmetric alternating minimization algorithm for total variation minimization

In this paper, we propose a novel symmetric alternating minimization algorithm to solve a broad class of total variation (TV) regularization problems. Unlike the usual zk → xk Gauss-Seidel cycle, the proposed algorithm performs the special x¯k→zk→xk cycle. The main idea for our setting is the recent symmetric Gauss-Seidel (sGS) technique which is developed for solving the multi-block convex composite problem. This idea also enables us to build the equivalence between the proposed method and the well-known accelerated proximal gradient (APG) method. The faster convergence rate of the proposed algorithm can be directly obtained from the APG framework and numerical results including image denoising, image deblurring, and analysis sparse recovery problem demonstrate the effectiveness of the new algorithm.

Ptychographic phase retrieval by proximal algorithms

We derive a set of ptychography phase-retrieval iterative engines based on proximal algorithms originally developed in convex optimization theory, and discuss their connections with existing ones. The use of proximal operator creates a simple frame work that allows us to incorporate the effect of noise from a maximum-likelihood (ML) principle. We focus on three particular algorithms, namely proximal minimization, alternating direction method of multiplier and accelerated proximal gradient (APG). We benchmark their performance with numerical simulations, and discuss their optimal conditions for convergence and accuracy. An experimental dataset is used to demonstrate their effectiveness as well, in which case an array of cubic Au nanoparticles with a size of 50 nm is imaged. We show that with the presence of Poisson noise, a dataset with photon counts up to 104 at one detector pixel already requires ML-based methods to achieve a stable convergence. Among the three algorithms derived in this work, APG method is reported first time for its application in ptychographic reconstruction and shows superior performance in terms of both accuracy and convergence rate with a noisy dataset.

Range-Angle Decoupling and Estimation for FDA-MIMO Radar via Atomic Norm Minimization and Accelerated Proximal Gradient

Frequency diverse array multiple-input multiple-output (FDA-MIMO) radar can offer the capability of range-angle-dependent beampattern and enjoy the advantage of effectively resolving the targets closely spaced in the same angle cell but different range cells. However, traditional subspace methods for range-angle estimation fail to efficiently work in a limited number of snapshots and coherent targets. In addition, the coupling between range and angle may lead to the degradation of estimation accuracy with ambiguity problem. In this letter, a gridless compressed sensing-based algorithm is proposed to joint estimate range-angle for FDA-MIMO radar. First, a decoupling model is presented to separate the range and angle from each other. Then, a 2D atomic norm minimization (ANM) problem for range-angle estimation is formulated and transformed into a semi-definite programming (SDP) problem with convex relaxation. Finally, a computationally efficient estimation algorithm via accelerated proximal gradient (APG) is developed to solve the SDP problem. Numerical simulations are conducted to illustrate the superior performance of the proposed algorithm.

Point Cloud Denoising via Feature Graph Laplacian Regularization.

Point cloud is a collection of 3D coordinates that are discrete geometric samples of an object's 2D surfaces. Imperfection in the acquisition process means that point clouds are often corrupted with noise. Building on recent advances in graph signal processing, we design local algorithms for 3D point cloud denoising. Specifically, we design a signal-dependent feature graph Laplacian regularizer (SDFGLR) that assumes surface normals computed from point coordinates are piecewise smooth with respect to a signal-dependent graph Laplacian matrix. Using SDFGLR as a signal prior, we formulate an optimization problem with a general 'p-norm fidelity term that can explicitly remove only two types of additive noise: small but non-sparse noise like Gaussian (using '2 fidelity term) and large but sparser noise like Laplacian (using '1 fidelity term). To establish a linear relationship between normals and 3D point coordinates, we first perform bipartite graph approximation to divide the point cloud into two disjoint node sets (red and blue). We then optimize the red and blue nodes' coordinates alternately. For '2-norm fidelity term, we iteratively solve an unconstrained quadratic programming (QP) problem, efficiently computed using conjugate gradient with a bounded condition number to ensure numerical stability. For '1-norm fidelity term, we iteratively minimize an '1-'2 cost function using accelerated proximal gradient (APG), where a good step size is chosen via Lipschitz continuity analysis. Finally, we propose simple mean and median filters for flat patches of a given point cloud to estimate the noise variance given the noise type, which in turn is used to compute a weight parameter trading off the fidelity term and signal prior in the problem formulation. Extensive experiments show state-of-the-art denoising performance among local methods using our proposed algorithms.

A robust recovery algorithm with smoothing strategies

This paper addresses the robust sparse recovery problem in the presence of impulsive measurement noise. In order to overcome the poor performance of ℓ2-norm loss function with the outliers under the impulsive noise, we employ the ℓ1-norm as the loss function for the residual error, which is less sensitive to outliers in the measurements than the popular ℓ2-loss. To rise to the challenges introduced by the non-smooth problem, we first employ two smoothing strategies to approximate the ℓ1-norm loss function: one introduces a relaxation factor in the ℓ1-norm and the other uses the infimal convolution smoothing technique to transform it into a smooth counterpart. Both of them can approximate the ℓ1-norm with arbitrary degree of accuracy and provide a Lipschitz continuous gradient loss function. Then, we employ the accelerated proximal gradient (APG) and monotone APG (mAPG) frameworks for the convex and non-convex regularization functions, respectively. The convergence performance is discussed for generalized regularization penalty. The simulation result demonstrates our conclusions and indicates that the algorithm proposed in this paper can improve the reconstruction quality.

Image Recovery with Data Missing in the Presence of Salt-and-Pepper Noise

In this paper, an image recovery problem under the case of salt-and-pepper noise and data missing that degrade image quality is addressed if they are not effectively handled, where the salt-and-pepper noise as the impulsive noise is remodeled as a sparse signal due to its impulsiveness and the data missing pattern, denoted by a sparse vector, contains only zeros and ones to formulate the data missing. In particular, the salt-and-pepper noise and data missing are reformatted by their sparsity, respectively. The wavelet and framelet domains are explored to sparsely represent the image in order to accurately reconstruct the clean image. From the reformulations conducted and to recover the image, under one optimization framework, a joint estimation is developed to perform the image recovery, the salt-and-pepper noise suppression, and the missing patter estimation. To solve the optimization problem, two efficient solvers are developed to obtain the joint estimation solution, and they are based on the alternating direction method of multipliers (ADMM) and accelerated proximal gradient (APG). Finally, numerical studies verify that the joint estimation algorithm outperforms the state-of-the-art approaches in terms of both objective and subjective evaluation standards.