Majorization-minimization Algorithm Research Articles

Large-scale robust regression with truncated loss via majorization-minimization algorithm

The utilization of regression methods employing truncated loss functions is widely praised for its robustness in handling outliers and representing the solution in the sparse form of the samples. However, due to the non-convexity of the truncated loss, the commonly used algorithms such as difference of convex algorithm (DCA) fail to maintain sparsity when dealing with non-convex loss functions, and adapting DCA for efficient optimization also incurs additional development costs. To address these challenges, we propose a novel approach called truncated loss regression via majorization-minimization algorithm (TLRM). TLRM employs a surrogate function to approximate the original truncated loss regression and offers several desirable properties: (i) Eliminating outliers before the training process and encapsulating general convex loss regression within its structure as iterative subproblems, (ii) Solving the convex loss problem iteratively thereby facilitating the use of a well-established toolbox for convex optimization. (iii) Converging to a truncated loss regression and providing a solution with sample sparsity. Extensive experiments demonstrate that TLRM achieves superior sparsity without sacrificing robustness, and it can be several tens of thousands of times faster than traditional DCA on large-scale problems. Moreover, TLRM is also applicable to datasets with millions of samples, making it a practical choice for real-world scenarios. The codebase for methods with truncated loss functions is accessible at https://i-do-lab.github.io/optimal-group.org/Resources/Code/TLRM.html.

Multinomial Restricted Unfolding

For supervised classification we propose to use restricted multidimensional unfolding in a multinomial logistic framework. Where previous research proposed similar models based on squared distances, we propose to use usual (i.e., not squared) Euclidean distances. This change in functional form results in several interpretational advantages of the resulting biplot, a graphical representation of the classification model. First, the conditional probability of any class peaks at the location of the class in the Euclidean space. Second, the interpretation of the biplot is in terms of distances towards the class points, whereas in the squared distance model the interpretation is in terms of the distance towards the decision boundary. Third, the distance between two class points represents an upper bound for the estimated log-odds of choosing one of these classes over the other. For our multinomial restricted unfolding, we develop and test a Majorization Minimization algorithm that monotonically decreases the negative log-likelihood. With two empirical applications we point out the advantages of the distance model and show how to apply multinomial restricted unfolding in practice, including model selection.

A Majorization-Minimization Algorithm for Neuroimage Registration.

Intensity-based image registration is critical for neuroimaging tasks, such as 3D reconstruction, times-series alignment, and common coordinate mapping. The gradient-based optimization methods commonly used to solve this problem require a careful selection of step-length. This limitation imposes substantial time and computational costs. Here we propose a gradient-independent rigid-motion registration algorithm based on the majorization-minimization (MM) principle. Each iteration of our intensity-based MM algorithm reduces to a simple point-set rigid registration problem with a closed form solution that avoids the step-length issue altogether. The details of the algorithm are presented, and an error bound for its more practical truncated form is derived. The performance of the MM algorithm is shown to be more effective than gradient descent on simulated images and Nissl stained coronal slices of mouse brain. We also compare and contrast the similarities and differences between the MM algorithm and another gradient-free registration algorithm called the block-matching method. Finally, extensions of this algorithm to more complex problems are discussed.

An inexact proximal majorization-minimization algorithm for remote sensing image stripe noise removal

An inexact proximal majorization-minimization algorithm for remote sensing image stripe noise removal

Compressed sensing with log-sum heuristic recover for seismic denoising

The compressed sensing (CS) method, commonly utilized for restructuring sparse signals, has been extensively used to attenuate the random noise in seismic data. An important basis of CS-based methods is the sparsity of sparse coefficients. In this method, the sparse coefficient vector is acquired by minimizing the l1 norm as a substitute for the l0 norm. Many efforts have been made to minimize the lp norm (0 < p < 1) to obtain a more desirable sparse coefficient representation. Despite the improved performance that is achieved by minimizing the lp norm with 0 < p < 1, the related sparse coefficient vector is still suboptimal since the parameter p is greater than 0 rather than infinitely approaching 0 p→0+. Therefore, the CS method with the limit p→0+ is proposed to enhance the sparse performance and thus generate better denoised results in this paper. Our proposed method is referred to as the CS-LHR method because the solving process for minimizing p→0+ is the log-sum heuristic recovery (LHR). Furthermore, to improve the computational efficiency, we incorporate the majorization-minimization (MM) algorithm in this CS-LHR method. Experimental results of synthetic and real seismic records demonstrate the remarkable performance of CS-LHR in random noise suppression.

On the Convergence of Block Majorization-Minimization Algorithms on the Grassmann Manifold

On the Convergence of Block Majorization-Minimization Algorithms on the Grassmann Manifold

Reconfigurable Intelligent Surfaces-Enhanced Uplink User-Centric Networks on Energy Efficiency Optimization

User-centric network (UCN) has been considered as a promising technology to improve the throughput of users, especially users at the cell edge, where each user is connected to and served by a group of access points. With the rapid growth of user’s uplink traffic demands, energy consumption has become an important issue in uplink UCN as user’s battery capacity is limited. To address this issue, in this research, we develop a reconfigurable intelligent surface (RIS)-enhanced uplink UCN, where RIS is exploited to improve the signal quality of uplink transmission for users with virtually no extra energy consumption. To approach an optimal energy efficiency, we jointly perform reflect beamforming at RISs and uplink power control at users. This is formulated into a non-convex and intractable energy efficiency maximization problem. To solve it effectively, we decompose it into two subproblems and carry out the optimization alternately. That is, we design the reflect beamforming matrices at RISs through fractional programming and an uplink power control at users through constructing a surrogate function via majorization-minimization algorithm. Numerical results demonstrate that the proposed algorithm could achieve significant gain both on energy efficiency and spectral efficiency compared to the benchmark algorithm.

A fusion non-convex group sparsity difference method and its application in rolling bearing fault diagnosis

Bearing fault is a common factor leading to machine failures. How to extract the periodic transient signal due to bearing faults submerged in strong noise is a challenging problem for bearing fault diagnosis. Total variation denoising is a method used to remove noise and extract features. However, it solely relies on the sparsity of the first-order difference of the signal, resulting in the loss of important features and underestimation of amplitude. Additionally, it fails to capture the periodicity of the signal. To address these issues, a novel approach called fusion non-convex group sparsity difference (FNC-GSD) method is proposed for bearing fault diagnosis. Firstly, it is recognized that noise does not exhibit sparsity in the difference domain, while transient signal exhibits group sparsity in the difference domain. This grouping property enhances the feature selection ability of sparse model. Inspired by this, the group sparsity of the transient signal in the difference domain is used to preserve the fault features as much as possible. Additionally, in order to promote sparsity of the signal itself in the time domain to preserve the potential impulse component, a l 1-norm regularization term is introduced. Furthermore, a non-convex sparsity-inducing penalty function strategy is employed to prevent amplitude underestimation. The proposed sparse model considers both the group sparsity in the difference domain and the sparsity in the time domain of the transient signal, and its solution is derived according to the majorization–minimization algorithm. And the particle swarm optimization algorithm is used to adaptively search the regularization parameters of FNC-GSD. Finally, multiple bearing fault diagnosis experiments are conducted to demonstrate the performance of the FNC-GSD. The results show that it has advantages in fault feature extraction compared with some other methods.

Link prediction in bipartite networks via effective integration of explicit and implicit relations

Link prediction in bipartite networks aims to identify or predict possible links between nodes of different types based on known network information. However, most existing studies predominantly focus on monopartite networks, neglecting the intrinsic properties unique to bipartite networks, such as the intricate high-order relationships between nodes. Both explicit relations (representing low-order information) and implicit relations (representing high-order information) play essential roles in predicting the evolution of bipartite networks, and they are indispensable and mutually reinforce each other. To fully leverage their potential in addressing the link prediction problem, we propose a novel framework from the perspective of network representation. This framework not only effectively integrates explicit relations and implicit relations, but also preserves the local and global structure of bipartite networks. Specifically, the probability of a link between two nodes of different types is determined by the linear sum of the contribution values of the mutually connected nodes in their respective common neighbors. Implicit relations are then used to preserve the local structure during the network representation. Furthermore, we implement optimization using a relaxed majorization-minimization algorithm, offering the advantage of uncovering high-quality local minima. Our proposed framework has undergone extensive testing on eight real-world datasets, and the results unequivocally demonstrate its significant superiority over state-of-the-art methods.

Gmcoda: Graphical model for multiple compositional vectors in microbiome studies.

Microbes are essential components in the ecosystem and participate in most biological procedures in environments. The high-throughput sequencing technologies help researchers directly quantify the abundance of microbes in a natural environment. Microbiome studies explore the construction, stability, and function of microbial communities with the aid of sequencing technology. However, sequencing technologies only provide relative abundances of microbes, and this kind of data is called compositional data in statistics. The constraint of the constant-sum requires flexible statistical methods for analyzing microbiome data. Current statistical analysis of compositional data mainly focuses on one compositional vector such as bacterial communities. The fungi are also an important component in microbial communities and are always measured by sequencing internal transcribed spacer instead of 16S rRNA genes for bacteria. The different sequencing methods between fungi and bacteria bring two compositional vectors in microbiome studies. We propose a novel statistical method, called gmcoda, based on an additive logistic normal distribution for estimating the partial correlation matrix for cross-domain interactions. A majorization-minimization algorithm is proposed to solve the optimization problem involved in gmcoda. Through simulation studies, gmcoda is demonstrated to work well in estimating partial correlations between two compositional vectors. Gmcoda is also applied to infer cross-domain interactions in a real microbiome dataset and finds potential interactions between bacteria and fungi. Gmcoda is open source and freely available from https://github.com/huayingfang/gmcoda under GNU LGPL v3.

Parsimony and parameter estimation for mixtures of multivariate leptokurtic-normal distributions

AbstractMixtures of multivariate leptokurtic-normal distributions have been recently introduced in the clustering literature based on mixtures of elliptical heavy-tailed distributions. They have the advantage of having parameters directly related to the moments of practical interest. We derive two estimation procedures for these mixtures. The first one is based on the majorization-minimization algorithm, while the second is based on a fixed point approximation. Moreover, we introduce parsimonious forms of the considered mixtures and we use the illustrated estimation procedures to fit them. We use simulated and real data sets to investigate various aspects of the proposed models and algorithms.

Convergence analysis of stochastic higher-order majorization–minimization algorithms

Majorization–minimization schemes are a broad class of iterative methods targeting general optimization problems, including nonconvex, nonsmooth and stochastic. These algorithms minimize successively a sequence of upper bounds of the objective function so that along the iterations the objective value decreases. We present a stochastic higher-order algorithmic framework for minimizing the average of a very large number of sufficiently smooth functions. Our stochastic framework is based on the notion of stochastic higher-order upper bound approximations of the finite-sum objective function and minibatching. We derive convergence results for nonconvex and convex optimization problems when the higher-order approximation of the objective function yields an error that is p times differentiable and has Lipschitz continuous p derivative. More precisely, for general nonconvex problems we present asymptotic stationary point guarantees and under Kurdyka–Lojasiewicz property we derive local convergence rates ranging from sublinear to linear. For convex problems with uniformly convex objective function, we derive local (super)linear convergence results for our algorithm. Compared to existing stochastic (first-order) methods, our algorithm adapts to the problem's curvature and allows using any batch size. Preliminary numerical tests support the effectiveness of our algorithmic framework.

Laplacian-based Cluster-Contractive t-SNE for High-Dimensional Data Visualization

Dimensionality reduction techniques aim at representing high-dimensional data in low-dimensional spaces to extract hidden and useful information or facilitate visual understanding and interpretation of the data. However, few of them take into consideration the potential cluster information contained implicitly in the high-dimensional data. In this article, we propose Lap tSNE, a new graph-layout nonlinear dimensionality reduction method based on t-SNE, one of the best techniques for visualizing high-dimensional data as 2D scatter plots. Specifically, Lap tSNE leverages the eigenvalue information of the graph Laplacian to shrink the potential clusters in the low-dimensional embedding when learning to preserve the local and global structure from high-dimensional space to low-dimensional space. It is nontrivial to solve the proposed model because the eigenvalues of normalized symmetric Laplacian are functions of the decision variable. We provide a majorization-minimization algorithm with convergence guarantee to solve the optimization problem of Lap tSNE and show how to calculate the gradient analytically, which may be of broad interest when considering optimization with Laplacian-composited objective. We evaluate our method by a formal comparison with state-of-the-art methods on seven benchmark datasets, both visually and via established quantitative measurements. The results demonstrate the superiority of our method over baselines such as t-SNE and UMAP. We also provide out-of-sample extension, large-scale extension, and mini-batch extension for our Lap tSNE to facilitate dimensionality reduction in various scenarios.

Multi-Beam Design for Near-Field Extremely Large-Scale RIS-Aided Wireless Communications

As the energy-saving array composed of passive elements, reconfigurable intelligent surface (RIS) will evolve to the extremely large-scale RIS (XL-RIS) to overcome serious path loss. This change leads to the near-field propagation becoming dominant. There are some works to explore the near-field beam design via beam training. Unfortunately, due to the constant modulus constraint for XL-RIS, most of works in the near-field scenario focus on single-beam design. For massive connectivity requirement scenario, these works will face a serious loss of beam gains, resulting in a decrease in transmission rate. To solve this problem, we propose a block coordinate descent-based scheme with majorization-minimization (MM) algorithm for multi-beam design. The proposed scheme handles constant modulus constraint from two aspects. Firstly, under this constraint, the multi-beam design is an intractable non-convex quadratic programming problem. We utilize MM algorithm to solve this problem as several iterative sub-problems which are easily to be sloved. Secondly, the solution space for multi-beam optimization is confined to a limited space due to this constraint, so we introduce the phases for beam gains as an extra optimizable variable to enrich the degree of freedom for optimization. Simulation results show that the proposed scheme could achieve a superior rate 50% higher than the existing schemes.

Fast and Robust Non-Rigid Registration Using Accelerated Majorization-Minimization

Non-rigid 3D registration, which deforms a source 3D shape in a non-rigid way to align with a target 3D shape, is a classical problem in computer vision. Such problems can be challenging because of imperfect data (noise, outliers and partial overlap) and high degrees of freedom. Existing methods typically adopt the lp type robust norm to measure the alignment error and regularize the smoothness of deformation, and use a proximal algorithm to solve the resulting non-smooth optimization problem. However, the slow convergence of such algorithms limits their wide applications. In this paper, we propose a formulation for robust non-rigid registration based on a globally smooth robust norm for alignment and regularization, which can effectively handle outliers and partial overlaps. The problem is solved using the majorization-minimization algorithm, which reduces each iteration to a convex quadratic problem with a closed-form solution. We further apply Anderson acceleration to speed up the convergence of the solver, enabling the solver to run efficiently on devices with limited compute capability. Extensive experiments demonstrate the effectiveness of our method for non-rigid alignment between two shapes with outliers and partial overlaps, with quantitative evaluation showing that it outperforms state-of-the-art methods in terms of registration accuracy and computational speed. The source code is available at https://github.com/yaoyx689/AMM_NRR.

STAR-RIS Assisted Wireless Powered IoT Networks

The paper proposes a novel design of simultaneously transmitting and reflecting reconfigurable intelligent surfaces (STAR-RIS) in a wireless powered Internet of Things (IoT) network, where two sensor node groups (SNGs) harvest energy from a power station (PS) and transmit their message to an access point (AP) with the harvested energy. The STAR-RIS, which is deployed in the middle of the SNGs and adopts the time splitting (TS) working mode, can help the energy transfer in the wireless energy transfer (WET) phase and the information transfer in the wireless information transfer (WIT) phase. The paper aims to maximize the sum throughput from the two SNGs to the AP by jointly designing the phase shifts of the STAR-RIS and the working time allocated to the two SNGs in the WET and WIT phases, respectively. To solve the formulated non-convex optimization problem, we propose a low-complexity algorithm where we first derive the optimal phase shifts of the STAR-RIS in the WIT phase. Then, we adopt the Lagrange dual method to simplify the optimization problem and optimize the phase shifts of the STAR-RIS in the WET phase by the Majorization-Minimization (MM) algorithm and the complex circle manifold (CCM) algorithm. Next, a two-layer iterative algorithm is used to obtain the optimal values of time allocated to the two SNGs. Finally, we evaluate the improvement of the proposed scheme by the simulation results compared with other benchmark schemes.

A new algorithm and a discussion about visualization for logistic reduced rank regression

Logistic reduced rank regression is a useful data analysis tool when we have multiple binary response variables and a set of predictors. In this paper, we describe logistic reduced rank regression and present a new majorization minimization algorithm for the estimation of model parameters. Furthermore, we discuss Type I and Type D triplots for visualizing the results of a logistic reduced rank regression model, compare them, and then develop a hybrid triplot using elements of both types. Two empirical data sets are analyzed. This analysis is used to (1) compare the new algorithm to an existing one in terms of speed; and (2) to show the hybrid triplot and its interpretation.

Dynamic data-free knowledge distillation by easy-to-hard learning strategy

Data-free knowledge distillation (DFKD) is a widely-used strategy for Knowledge Distillation (KD) whose training data is not available. It trains a lightweight student model with the aid of a large pretrained teacher model without any access to training data. However, existing DFKD methods suffer from inadequate and unstable training process, as they do not adjust the generation target dynamically based on the status of the student model during learning. To address this limitation, we propose a novel DFKD method called CuDFKD. It teaches students by a dynamic strategy that gradually generates easy-to-hard pseudo samples, mirroring how humans learn. Besides, CuDFKD adapts the generation target dynamically according to the status of student model. Moreover, we provide a theoretical analysis of the majorization minimization (MM) algorithm and explain the convergence of CuDFKD. To measure the robustness and fidelity of DFKD methods, we propose two more metrics, and experiments shows CuDFKD has comparable performance to state-of-the-art (SOTA) DFKD methods on all datasets. Experiments also present that our CuDFKD has the fastest convergence and best robustness over other SOTA DFKD methods.

A Non-Convex Hybrid Overlapping Group Sparsity Model with Hyper-Laplacian Prior for Multiplicative Noise

Multiplicative noise removal is a quite challenging problem in image denoising. In recent years, hyper-Laplacian prior information has been successfully introduced in the image denoising problem and significant denoising effects have been achieved. In this paper, we propose a new hybrid regularizer model for removing multiplicative noise. The proposed model consists of the non-convex higher-order total variation and overlapping group sparsity on a hyper-Laplacian prior regularizer. It combines the advantages of the non-convex regularization and the hybrid regularization, which may simultaneously preserve the fine-edge information and reduce the staircase effect at the same time. We develop an effective alternating minimization method for the proposed nonconvex model via an alternating direction method of multipliers framework, where the majorization–minimization algorithm and the iteratively reweighted algorithm are adopted to solve the corresponding subproblems. Numerical experiments show that the proposed model outperforms the most advanced model in terms of visual quality and certain image quality measurements.

Convex and Nonconvex Risk-Based Linear Regression at Scale

The value at risk (VaR) and the conditional value at risk (CVaR) are two popular risk measures to hedge against the uncertainty of data. In this paper, we provide a computational toolbox for solving high-dimensional sparse linear regression problems under either VaR or CVaR measures, the former being nonconvex and the latter convex. Unlike the empirical risk (neutral) minimization models in which the overall losses are decomposable across data, the aforementioned risk-sensitive models have nonseparable objective functions so that typical first order algorithms are not easy to scale. We address this scaling issue by adopting a semismooth Newton-based proximal augmented Lagrangian method of the convex CVaR linear regression problem. The matrix structures of the Newton systems are carefully explored to reduce the computational cost per iteration. The method is further embedded in a majorization–minimization algorithm as a subroutine to tackle the nonconvex VaR-based regression problem. We also discuss an adaptive sieving strategy to iteratively guess and adjust the effective problem dimension, which is particularly useful when a solution path associated with a sequence of tuning parameters is needed. Extensive numerical experiments on both synthetic and real data demonstrate the effectiveness of our proposed methods. In particular, they are about 53 times faster than the commercial package Gurobi for the CVaR-based sparse linear regression with 4,265,669 features and 16,087 observations. History: Accepted by Antonio Frangioni, Area Editor for Design & Analysis of Algorithms–Continuous. Funding: This work was supported in part by the NSF, the Division of Computing and Communication Foundations [Grant 2153352], the National Natural Science Foundation of China [Grant 12271187], and the Hong Kong Research Grant Council [Grant 15304019]. Supplemental Material: The software that supports the findings of this study is available within the paper and its Supplemental Information ( https://pubsonline.informs.org/doi/suppl/10.1287/ijoc.2023.1282 ) as well as from the IJOC GitHub software repository ( https://github.com/INFORMSJoC/2022.0012 ) at ( http://dx.doi.org/10.5281/zenodo.7483279 ).