Proximal Forward-backward Splitting Research Articles

A proximal forward-backward splitting based algorithmic framework for Wasserstein logistic regression using heavy ball strategy

ABSTRACT In this paper, a forward-backward splitting based algorithmic framework incorporating the heavy ball strategy is proposed so as to efficiently solve the Wasserstein logistic regression problem. The proposed algorithmic framework consists two phases: the first phase involves a gradient descent step extension method, whilst the second phase involves a problem of instantaneous optimisation which balances the minimisation of a regularisation term while maintaining close proximity to the interim state given in the first phase. Then, it proves that the proposed algorithmic framework converges to the optimal solution of the Wasserstein logistic regression problem. Finally, numerical experiments are conducted, which illustrate the efficient implementation for high-dimensional sparsity data. The numerical results demonstrate that the proposed algorithmic framework outperforms not only the off-the-shelf solvers, but also some existing first-order algorithms.

Enhanced total generalized variation method based on moreau envelope

Image restoration with total generalized variation (TGV) regularization has been proved to be a creditable method and used broadly, but this method is imperfect in retaining image details. This paper proposes a non-convex and non-separable regularization term based on TGV enhancement, which can maintain the strict convexity of the total cost function and avoid the underestimation of the TGV method. The new regularization is obtained by subtracting the Moreau envelope of TGV from the TGV term, in which the former is defined by infimal convolution. To minimize the cost function, the proximal forward-backward splitting (FBS) algorithm is applied, and the alternating direction method of multipliers with adaptive parameter estimation (APE-ADMM), which avoids the drawbacks of FBS algorithm, is proposed by applying the variable splitting and the augment Lagrangian method (ALM). This proposed algorithm can update the regularization parameter adaptively. Experiments varify that the proposed method is more accurate in solution estimation than the other promoted ones, and the new ADMM algorithm with new regularization shows improved effects of image restoration.

Proximal Normalized Subband Adaptive Filtering for Acoustic Echo Cancellation

In this paper, we propose a novel normalized subband adaptive filter algorithm suited for sparse scenarios, which combines the proportionate and sparsity-aware mechanisms. The proposed algorithm is derived based on the proximal forward-backward splitting and the soft-thresholding methods. We analyze the mean and mean square behaviors of the algorithm, which is supported by simulations. In addition, an adaptive approach for the choice of the thresholding parameter in the proximal step is also proposed based on the minimization of the mean square deviation. Simulations in the contexts of system identification and acoustic echo cancellation verify the superiority of the proposed algorithm over its counterparts.

4D-AirNet: a temporally-resolved CBCT slice reconstruction method synergizing analytical and iterative method with deep learning

Four-dimensional (4D) cone-beam CT (CBCT) reconstructs temporally-resolved phases of 3D volumes often with the same amount of projection data that are meant for reconstructing a single 3D volume. 4D CBCT is a sparse-data problem that is very challenging for high-quality 4D CBCT image reconstruction. Here we develop a new method, namely 4D-AirNet, that synergizes analytical and iterative method with deep learning for high-quality temporally-resolved CBCT slice reconstruction. 4D-AirNet is an unrolling method using the optimization framework of fused analytical and iterative reconstruction (AIR), which is based on proximal forward-backward splitting (PFBS). Three different strategies are developed for 4D-AirNet: random-phase (RP), prior-guided (PG), and all-phase (AP). RP-AirNet and PG-AirNet utilize phase-by-phase training and reconstruction, while PG-AirNet also uses a prior image reconstructed with all-phase projection data. Dense connectivity is built into 4D-AirNet networks for improved reconstruction quality. In contrast, AP-AirNet trains and reconstructs all phases simultaneously. In addition, the joint regularization method of DL and conventional spatiotemporal total variation (TV) is investigated. 4D-AirNet methods were evaluated in comparison with conventional iterative (TV) and deep learning (LEARN) methods, using simulated 2D-t CBCT scans from a lung dataset with various sparse-data levels. The reconstruction results suggest 4D-AirNet methods outperform TV and LEARN, and AP-AirNet provides the best reconstruction quality overall.

Low-dose CT with deep learning regularization via proximal forward–backward splitting

Low-dose x-ray computed tomography (LDCT) is desirable for reduced patient dose. This work develops new image reconstruction methods with deep learning (DL) regularization for LDCT. Our methods are based on the unrolling of a proximal forward–backward splitting (PFBS) framework with data-driven image regularization via deep neural networks. In contrast to PFBS-IR, which utilizes standard data fidelity updates via an iterative reconstruction (IR) method, PFBS-AIR involves preconditioned data fidelity updates that fuse the analytical reconstruction (AR) and IR methods in a synergistic way, i.e. fused analytical and iterative reconstruction (AIR). The results suggest that the DL-regularized methods (PFBS-IR and PFBS-AIR) provide better reconstruction quality compared to conventional methods (AR or IR). In addition, owing to the AIR, PFBS-AIR noticeably outperformed PFBS-IR and another DL-based postprocessing method, FBPConvNet.

AirNet: Fused analytical and iterative reconstruction with deep neural network regularization for sparse-data CT.

Sparse-data computed tomography (CT) frequently occurs, such as breast tomosynthesis, C-arm CT, on-board four-dimensional cone-beam CT (4D CBCT), and industrial CT. However, sparse-data image reconstruction remains challenging due to highly undersampled data. This work develops a data-driven image reconstruction method for sparse-data CT using deep neural networks (DNN). The new method so-called AirNet is designed to incorporate the benefits from analytical reconstruction method (AR), iterative reconstruction method (IR), and DNN. It is built upon fused analytical and iterative reconstruction (AIR) that synergizes AR and IR via the optimization framework of modified proximal forward-backward splitting (PFBS). By unrolling PFBS into IR updates of CT data fidelity and DNN regularization with residual learning, AirNet utilizes AR such as FBP during the data fidelity, introduces dense connectivity into DNN regularization, and learns PFBS coefficients and DNN parameters that minimize the loss function during the training stage; and then AirNet with trained parameters can be used for end-to-end image reconstruction. A CT atlas of 100 prostate scans was used to validate the AirNet in comparison with state-of-art DNN-based postprocessing and image reconstruction methods. The validation loss in AirNet had the fastest decreasing rate, owing to inherited fast convergence from AIR. AirNet was robust to noise in projection data and content differences between the training set and the images to be reconstructed. The impact of image quality on radiotherapy treatment planning was evaluated for both photon and proton therapy, and AirNet achieved the best treatment plan quality, especially for proton therapy. For example, with limited-angle data, the maximal target dose for AirNet was 109.5% in comparison with the ground truth 109.1%, while it was significantly elevated to 115.1% and 128.1% for FBPConvNet and LEARN, respectively. A new image reconstruction AirNet is developed for sparse-data CT image reconstruction. AirNet achieved the best image reconstruction quality both visually and quantitatively among all methods under comparison for all sparse-data scenarios (sparse-view and limited-angle), and provided the best photon and proton treatment plan quality based on sparse-data CT.

Projection-Based Regularized Dual Averaging for Stochastic Optimization

We propose a novel stochastic-optimization framework based on the regularized dual averaging (RDA) method. The proposed approach differs from the previous studies of RDA in three major aspects. First, the squared-distance loss function to a “random” closed convex set is employed for stability. Second, a sparsity-promoting metric (used implicitly by a certain proportionate-type adaptive filtering algorithm) and a quadratically-weighted $\ell _1$ regularizer are used simultaneously. Third, the step size and regularization parameters are both constant due to the smoothness of the loss function. These three differences yield an excellent sparsity-seeking property, high estimation accuracy, and insensitivity to the choice of the regularization parameter. Numerical examples show the remarkable advantages of the proposed method over the existing methods (including AdaGrad and the adaptive proximal forward–backward splitting method) in applications to regression and classification with real/synthetic data.

A link prediction algorithm based on low-rank matrix completion

Link prediction is an essential research area in network analysis. Based on the technique of matrix completion, an algorithm for link prediction in networks is proposed. We propose a new model to describe matrix completion. In addition to the observed data, the model takes the noise matrix into account, which is important for detecting missing links. We propose an alternative iteration algorithm to solve matrix completion. The algorithm uses the proximal forward-backward splitting to minimize the nuclear and L2,1 norm simultaneously. A random projected shrinkage operator on the singular values is defined, and an algorithm for implementing the projected shrinkage operator is presented. Using this operator, the time complexity of our algorithm is reduced greatly and reaches the lower bound of the time complexity for a similarity-based link prediction method. The empirical results of real-world networks show that the proposed algorithm can achieve higher quality prediction results than other algorithms.

Iterative reconstruction for low dose dual energy CT using information-divergence constrained spectral redundancy information.

Dual energy computed tomography (DECT) can improve the capability of differentiating different materials compared with conventional CT. However, due to non-negligible radiation exposure to patients, dose reduction has recently become a critical concern in CT imaging field. In this work, to reduce noise at the same time maintain DECT images quality, we present an iterative reconstruction algorithm for low-dose DECT images where in the objective function of the algorithm consists of a data-fidelity term and a regularization term. The former term is based on alpha-divergence to describe the statistical distribution of the DE sinogram data. And the latter term is based on the redundant information to reflect the prior information of the desired DECT images. For simplicity, the presented algorithm is termed as "AlphaD-aviNLM". To minimize the associative objective function, a modified proximal forward-backward splitting algorithm is proposed. Digital phantom, physical phantom, and patient data were utilized to validate and evaluate the presented AlphaD-aviNLM algorithm. The experimental results characterize the performance of the presented AlphaD-aviNLM algorithm. Speficically, in the digital phantom study, the presented AlphaD-aviNLM algorithm performs better than the PWLS-TV, PWLS-aviNLM, and AlphaD-TV with more than 49%, 34%, and 40% gains for the RMSE metric, 1.3%, 0.4%, and 0.7% gains for the FSIM metric and 13%, 8%, and 11% gains for the PSNR metric. In the physical phantom study, the presented AlphaD-aviNLM algorithm performs better than the PWLS-TV, PWLS-aviNLM, and AlphaD-TV with more than 0.55%, 0.07%, and 0.16% gains for the FSIM metric.

Supervised nonnegative matrix factorization via minimization of regularized Moreau-envelope of divergence function with application to music transcription

We propose a convex-analytic approach to supervised nonnegative matrix factorization (NMF), using the Moreau envelope, a smooth approximation, of the β-divergence as a loss function.The supervised NMF problem is cast as minimization of the loss function penalized by four terms: (i) a time-continuity enhancing regularizer, (ii) the indicator function enforcing the nonnegativity, (iii) a basis-vector selector (a block ℓ1 norm), and (iv) a sparsity-promoting regularizer. We derive a closed-form expression of the proximity operator of the sum of the three non-differentiable penalty terms (ii)–(iv). The optimization problem can thus be solved numerically by the proximal forward–backward splitting method, which requires no auxiliary variable and is therefore free from extra errors. The source number is automatically attained as an outcome of optimization. The simulation results show the efficacy of the proposed method in an application to polyphonic music transcription.

Proximal forward-backward splitting method for zeros of sum accretive operators for a fixed point set and inverse problems

Proximal forward-backward splitting method for zeros of sum accretive operators for a fixed point set and inverse problems

Global sparse gradient guided variational Retinex model for image enhancement

In this paper, we propose a global sparse gradient guided variational Retinex model (GSG-VR) for image enhancement. Based on the Retinex theory, a new variational Retinex model is proposed to decompose an image into illumination layer and reflectance layer. The gradient of illumination layer is expected to approximate a guided gradient field which is estimated by a global sparse gradient model (GSG). To estimate the guided gradient at each pixel, GSG makes use of pixels within its neighborhood (even global image). And a sparse regularization is imposed on the whole gradient field. These two models, the new variational Retinex and GSG model, compose a complete system GSG-VR. To solve it, a proximal forward–backward splitting algorithm and an alternating minimization algorithm are developed. A few numerical examples are presented to illustrate the effectiveness of the proposed models and algorithms.

Iterative reconstruction for sparse-view x-ray CT using alpha-divergence constrained total generalized variation minimization.

Accurate statistical model of the measured projection data is essential for computed tomography (CT) image reconstruction. The transmission data can be described by a compound Poisson distribution upon an electronic noise background. However, such a statistical distribution is numerically intractable for image reconstruction. Although the sinogram data is easily manipulated, it lacks a statistical description for image reconstruction. To address this problem, we present an alpha-divergence constrained total generalized variation (AD-TGV) method for sparse-view x-ray CT image reconstruction. The AD-TGV method is formulated as an optimization problem, which balances the alpha-divergence (AD) fidelity and total generalized variation (TGV) regularization in one framework. The alpha-divergence is used to measure the discrepancy between the measured and estimated projection data. The TGV regularization can effectively eliminate the staircase and patchy artifacts which is often observed in total variation (TV) regularization. A modified proximal forward-backward splitting algorithm was proposed to minimize the associated objective function. Qualitative and quantitative evaluations were carried out on both phantom and patient data. Compared with the original TV-based method, the evaluations clearly demonstrate that the AD-TGV method achieves higher accuracy and lower noise, while preserving structural details. The experimental results show that the presented AD-TGV method can achieve more gains over the AD-TV method in preserving structural details and suppressing image noise and undesired patchy artifacts. The authors can draw the conclusion that the presented AD-TGV method is potential for radiation dose reduction by lowering the milliampere-seconds (mAs) and/or reducing the number of projection views.

Consistent learning by composite proximal thresholding

We investigate the modeling and the numerical solution of machine learning problems with prediction functions which are linear combinations of elements of a possibly infinite dictionary of functions. We propose a novel flexible composite regularization model, which makes it possible to incorporate various priors on the coefficients of the prediction function, including sparsity and hard constraints. We show that the estimators obtained by minimizing the regularized empirical risk are consistent in a statistical sense, and we design an error-tolerant composite proximal thresholding algorithm for computing such estimators. New results on the asymptotic behavior of the proximal forward–backward splitting method are derived and exploited to establish the convergence properties of the proposed algorithm. In particular, our method features a o(1 / m) convergence rate in objective values.

Fused analytical and iterative reconstruction (AIR) via modified proximal forward–backward splitting: a FDK-based iterative image reconstruction example for CBCT

This work is to develop a general framework, namely analytical iterative reconstruction (AIR) method, to incorporate analytical reconstruction (AR) method into iterative reconstruction (IR) method, for enhanced CT image quality and reconstruction efficiency. Specifically, AIR is established based on the modified proximal forward–backward splitting (PFBS) algorithm, and its connection to the filtered data fidelity with sparsity regularization is discussed.As a result, AIR decouples data fidelity and image regularization with a two-step iterative scheme, during which an AR-projection step updates the filtered data fidelity term, while a denoising solver updates the sparsity regularization term. During the AR-projection step, the image is projected to the data domain to form the data residual, and then reconstructed by certain AR to a residual image which is then weighted together with previous image iterate to form next image iterate. Intuitively since the eigenvalues of AR-projection operator are close to the unity, PFBS based AIR has a fast convergence. Such an advantage is rigorously established through convergence analysis and numerical computation of convergence rate.The proposed AIR method is validated in the setting of circular cone-beam CT with AR being FDK and total-variation sparsity regularization, and has improved image quality from both AR and IR. For example, AIR has improved visual assessment and quantitative measurement in terms of both contrast and resolution, and reduced axial and half-fan artifacts.

MO-DE-207A-07: Filtered Iterative Reconstruction (FIR) Via Proximal Forward-Backward Splitting: A Synergy of Analytical and Iterative Reconstruction Method for CT

Purpose: This work is to develop a general framework, namely filtered iterative reconstruction (FIR) method, to incorporate analytical reconstruction (AR) method into iterative reconstruction (IR) method, for enhanced CT image quality. Methods: FIR is formulated as a combination of filtered data fidelity and sparsity regularization, and then solved by proximal forward-backward splitting (PFBS) algorithm. As a result, the image reconstruction decouples data fidelity and image regularization with a two-step iterative scheme, during which an AR-projection step updates the filtered data fidelity term, while a denoising solver updates the sparsity regularization term. During the AR-projection step, the image is projected to the data domain to form the data residual, and then reconstructed by certain AR to a residual image which is in turn weighted together with previous image iterate to form next image iterate. Since the eigenvalues of AR-projection operator are close to the unity, PFBS based FIR has a fast convergence. Results: The proposed FIR method is validated in the setting of circular cone-beam CT with AR being FDK and total-variation sparsity regularization, and has improved image quality from both AR and IR. For example, AIR has improved visual assessment and quantitative measurement in terms of both contrast and resolution, and reduced axial and half-fan artifacts. Conclusion: FIR is proposed to incorporate AR into IR, with an efficient image reconstruction algorithm based on PFBS. The CBCT results suggest that FIR synergizes AR and IR with improved image quality and reduced axial and half-fan artifacts. The authors was partially supported by the NSFC (#11405105), the 973 Program (#2015CB856000), and the Shanghai Pujiang Talent Program (#14PJ1404500).

On Proximal Subgradient Splitting Method for Minimizing the sum of two Nonsmooth Convex Functions

In this paper we present a variant of the proximal forward-backward splitting iteration for solving nonsmooth optimization problems in Hilbert spaces, when the objective function is the sum of two nondifferentiable convex functions. The proposed iteration, which will be called Proximal Subgradient Splitting Method, extends the classical subgradient iteration for important classes of problems, exploiting the additive structure of the objective function. The weak convergence of the generated sequence was established using different stepsizes and under suitable assumptions. Moreover, we analyze the complexity of the iterates.

A primal–dual fixed point algorithm for convex separable minimization with applications to image restoration

Recently, the minimization of a sum of two convex functions has received considerable interest in a variational image restoration model. In this paper, we propose a general algorithmic framework for solving a separable convex minimization problem from the point of view of fixed point algorithms based on proximity operators (Moreau 1962 C. R. Acad. Sci., Paris I 255 2897–99). Motivated by proximal forward–backward splitting proposed in Combettes and Wajs (2005 Multiscale Model. Simul. 4 1168–200) and fixed point algorithms based on the proximity operator (FP2O) for image denoising (Micchelli et al 2011 Inverse Problems 27 45009–38), we design a primal–dual fixed point algorithm based on the proximity operator (PDFP2Oκ for κ ∈ [0, 1)) and obtain a scheme with a closed-form solution for each iteration. Using the firmly nonexpansive properties of the proximity operator and with the help of a special norm over a product space, we achieve the convergence of the proposed PDFP2Oκ algorithm. Moreover, under some stronger assumptions, we can prove the global linear convergence of the proposed algorithm. We also give the connection of the proposed algorithm with other existing first-order methods. Finally, we illustrate the efficiency of PDFP2Oκ through some numerical examples on image supper-resolution, computerized tomographic reconstruction and parallel magnetic resonance imaging. Generally speaking, our method PDFP2O (κ = 0) is comparable with other state-of-the-art methods in numerical performance, while it has some advantages on parameter selection in real applications.

Multikernel Adaptive Filtering

This paper exemplifies that the use of multiple kernels leads to efficient adaptive filtering for nonlinear systems. Two types of multikernel adaptive filtering algorithms are proposed. One is a simple generalization of the kernel normalized least mean square (KNLMS) algorithm [2], adopting a coherence criterion for dictionary designing. The other is derived by applying the adaptive proximal forward-backward splitting method to a certain squared distance function plus a weighted block l1 norm penalty, encouraging the sparsity of an adaptive filter at the block level for efficiency. The proposed multikernel approach enjoys a higher degree of freedom than those approaches which design a kernel as a convex combination of multiple kernels. Numerical examples show that the proposed approach achieves significant gains particularly for nonstationary data as well as insensitivity to the choice of some design-parameters.

An Accelerated Proximal Gradient Algorithm for Frame-Based Image Restoration via the Balanced Approach

Frame-based image restoration by using the balanced approach has been developed over the last decade. Many recently developed algorithms for image restoration can be viewed as an acceleration of the proximal forward-backward splitting algorithm. Accelerated proximal gradient (APG) algorithms studied by Nesterov, Nemirovski, and others have been demonstrated to be efficient in solving various regularized convex optimization problems arising in compressed sensing, machine learning, and control. In this paper, we adapt the APG algorithm to solve the $\ell_1$-regularized linear least squares problem in the balanced approach in frame-based image restoration. This algorithm terminates in $O(1/\sqrt{\epsilon})$ iterations with an $\epsilon$-optimal solution, and we demonstrate that this single algorithmic framework can universally handle several image restoration problems, such as image deblurring, denoising, inpainting, and cartoon-texture decomposition. Our numerical results suggest that the APG algorithms are efficient and robust in solving large-scale image restoration problems. The algorithms we implemented are able to restore $512\times512$ images in various image restoration problems in less than 50 seconds on a modest PC. We also compare the numerical performance of our proposed algorithms applied to image restoration problems by using one frame-based system with that by using cartoon and texture systems for image deblurring, denoising, and inpainting.