Projected Gradient Descent Method Research Articles

Projected gradient descent method for cardinality-constrained portfolio optimization

Cardinality-constrained portfolio optimization aims at determining the investment weights on given assets using the historical data. This problem typically requires three constraints, namely, capital budget, long–only, and sparsity. The sparsity restraint allows investment managers to select a small number of stocks from the given assets. Most existing approaches exploit the penalty technique to handle the sparsity constraint. Therefore, they require tweaking the associated regularization parameter to obtain the desired cardinality level, which is time-consuming. This paper formulates the sparse portfolio design as a cardinality-constrained nonconvex optimization problem, where the sparsity constraint is modeled as a bounded ℓ0-norm. The projected gradient descent (PGD) method is then utilized to deal with the resultant problem. Different from existing algorithms, the suggested approach, called ℓ0-PGD, can explicitly control the cardinality level. In addition, its convergence is established. Specifically, the ℓ0-PGD guarantees that the objective function value converges, and the variable sequences converges to a local minimum. To remedy the weaknesses of gradient descent, the momentum technique is exploited to enhance the performance of the ℓ0-PGD, yielding ℓ0-PMGD. Numerical results on four real-world datasets, viz. NASDAQ 100, S&P 500, Russell 1000, and Russell 2000 exhibit the superiority of the ℓ0-PGD and ℓ0-PMGD over existing algorithms in terms of mean return and Sharpe ratio.

Optimal Control of Temporal Networks With Variable Input and Node-Source Connection.

Many networked systems built upon real-life physical or social interactions have time-varying connections among individual units, where the temporal changes in connectivity and/or interaction strength lead to complicated dynamics. The temporal network model was proposed in the form of controlled linear dynamical systems acting in an ordered sequence of time intervals. One of the core challenges in network science is the control of networks and the optimization of the control strategy. However, most canonical frameworks for solving optimal control problems were established for static networks featuring constant topology. New theories and techniques are yet to be developed for the temporal networks, with an important case being that the input and the source-node connection are both variables. In this work, by formulating a quadratic energy cost without solving the Riccati differential equation, we show that the control effort can be reduced substantially by improving either the system trajectories or the input matrices. The two approaches are further combined in a coordinate descent framework, integrating linearly constrained quadratic programming, and a projected gradient descent method. Taken together, the results underline the potential of temporal networks as energy-efficient control systems and present strategies to improve the control input. Moreover, the proposed algorithms can serve as a starting point for future engineering of real-world temporal networks.

Evaluating the Robustness of Deep Learning Models against Adversarial Attacks: An Analysis with FGSM, PGD and CW

This study evaluated the generation of adversarial examples and the subsequent robustness of an image classification model. The attacks were performed using the Fast Gradient Sign method, the Projected Gradient Descent method, and the Carlini and Wagner attack to perturb the original images and analyze their impact on the model’s classification accuracy. Additionally, image manipulation techniques were investigated as defensive measures against adversarial attacks. The results highlighted the model’s vulnerability to conflicting examples: the Fast Gradient Signed Method effectively altered the original classifications, while the Carlini and Wagner method proved less effective. Promising approaches such as noise reduction, image compression, and Gaussian blurring were presented as effective countermeasures. These findings underscore the importance of addressing the vulnerability of machine learning models and the need to develop robust defenses against adversarial examples. This article emphasizes the urgency of addressing the threat posed by harmful standards in machine learning models, highlighting the relevance of implementing effective countermeasures and image manipulation techniques to mitigate the effects of adversarial attacks. These efforts are crucial to safeguarding model integrity and trust in an environment marked by constantly evolving hostile threats. An average 25% decrease in accuracy was observed for the VGG16 model when exposed to the Fast Gradient Signed Method and Projected Gradient Descent attacks, and an even more significant 35% decrease with the Carlini and Wagner method.

VAASI: Crafting valid and abnormal adversarial samples for anomaly detection systems in industrial scenarios

In the realm of industrial anomaly detection, machine and deep learning models face a critical vulnerability to adversarial attacks. In this context, existing attack methodologies primarily target continuous features, often in the context of images, making them unsuitable for the categorical or discrete features prevalent in industrial systems. To fortify the cybersecurity of industrial environments, this paper introduces a groundbreaking adversarial attack approach tailored to the unique demands of these settings. Our novel technique enables the creation of targeted adversarial samples that are valid within the framework of supervised cyberattack detection models in industrial scenarios, preserving the consistency of discrete values and correcting cases where an adversarial sample transitions into a normal one. Our approach leverages the SHAP interpretability method to identify the most salient features for each sample. Subsequently, the Projected Gradient Descent technique is employed to perturb continuous features, ensuring adversarial sample generation. To handle categorical features for a specific adversarial sample, our method scrutinizes the closest sample within the normal training dataset and replicates its categorical feature values. Additionally, Decision Trees trained within a Random Forest are utilized to ensure that the resulting adversarial samples maintain the essential abnormal behavior required for detection. The validation of our proposal was conducted using the WADI dataset obtained from a water distribution plant, providing a realistic industrial context. During validation, we assessed the mean error and the total number of adversarial samples generated by our approach, comparing it with the original Projected Gradient Descent method and the Carlini & Wagner attack across various parameter configurations. Remarkably, our proposal consistently achieved the best trade-off between mean error and the number of generated adversarial samples, showcasing its superiority in safeguarding industrial systems.

Apodizer Design to Efficiently Couple Light into a Fiber Bragg Grating

We provide an optimal control framework for efficiently coupling light in a bare fiber into Bragg gratings with cubic nonlinearity. The light-grating interaction excites gap solitons, a type of localized nonlinear coherent state which propagates with a central frequency in the forbidden band gap, resulting in a dramatically slower group velocity. Due to the nature of the band gap, a substantial amount of light is back-reflected by the grating’s strong reflective properties. We optimize, via a projected gradient descent method, the transmission efficiency of previously designed nonuniform grating structures in order to couple more slow light into the grating. We further explore the space of possible grating designs, using genetic algorithms, along with a previously unexplored design parameter: the grating chirp. Through these methods, we find structures that couple a greater fraction of light into the grating with the added bonus of creating slower pulses.

Orientation estimation of cryo-EM images using projected gradient descent method

Orientation estimation is an important task in three-dimensional cryo-EM image reconstruction. By applying the common line method, the orientation estimation task can be formulated as a least squares (LS) problem or a least un-squared deviation (LUD) problem with orthogonality constraint. However, the non-convexity of the orthogonality constraint introduces numerical difficulties to the orientation estimation. The conventional approach is to reformulate the orthogonality constrained minimization problem into a semi-definite programming problem using convex relaxation strategies. In this paper, we consider a direct way to solve the constrained minimization problem without relaxation. We focus on the weighted LS problem because the LUD problem can be reformulated into a sequence of weighted LS problems using the iteratively re-weighted LS approach. As a classical approach, the projected gradient descent (PGD) method has been successfully applied to solve the convex constrained minimization problem. We apply the PGD method to the minimization problem with orthogonality constraint and show that the constraint set is a generalized prox-regular set, and it satisfies the norm compatibility condition. We also demonstrate that the objective function of the minimization problem satisfies the restricted strong convexity and the restricted strong smoothness over a constraint set. Therefore, the sequence generated by the PGD method converges when the initial conditions are satisfied. Experimental results show that the PGD method significantly outperforms the semi-definite relaxation methods from a computation standpoint, and the mean square error is almost the same or smaller.

Learning matrix factorization with scalable distance metric and regularizer

Matrix factorization has always been an encouraging field, which attempts to extract discriminative features from high-dimensional data. However, it suffers from negative generalization ability and high computational complexity when handling large-scale data. In this paper, we propose a learnable deep matrix factorization via the projected gradient descent method, which learns multi-layer low-rank factors from scalable metric distances and flexible regularizers. Accordingly, solving a constrained matrix factorization problem is equivalently transformed into training a neural network with an appropriate activation function induced from the projection onto a feasible set. Distinct from other neural networks, the proposed method activates the connected weights not just the hidden layers. As a result, it is proved that the proposed method can learn several existing well-known matrix factorizations, including singular value decomposition, convex, nonnegative and semi-nonnegative matrix factorizations. Finally, comprehensive experiments demonstrate the superiority of the proposed method against other state-of-the-arts.

Numerical and Experimental Study of Colored Magnetic Particle Mapping via Magnetoelectric Sensors

Colored imaging of magnetic nanoparticles (MNP) is a promising noninvasive method for medical applications such as therapy and diagnosis. This study investigates the capability of the magnetoelectric sensor and projected gradient descent (PGD) algorithm for colored particle detection. In the first step, the required circumstances for image reconstruction are studied via a simulation approach for different signal-to-noise ratios (SNR). The spatial accuracy of the reconstructed image is evaluated based on the correlation coefficient (CC) factor. The inverse problem is solved using the PGD method, which is adapted according to a nonnegativity constraint in the complex domain. The MNP characterizations are assessed through a magnetic particle spectrometer (MPS) for different types. In the experimental investigation, the real and imaginary parts of the MNP's response are used to detect the spatial distribution and particle type, respectively. The experimental results indicate that the average phase difference for CT100 and ARA100 particles is 14 degrees, which is consistent with the MPS results and could satisfy the system requirements for colored imaging. The experimental evaluation showed that the magnetoelectric sensor and the proposed approach could be potential candidates for color bio-imaging applications.

Perception-Based Sampled-Data Optimization of Dynamical Systems

Motivated by perception-based and sensing-based control problems in autonomous systems, this paper addresses the problem of developing feedback controllers to regulate the inputs and the states of a dynamical system to optimal solutions of an optimization problem when one has no access to exact measurements of the system states. In particular, we consider the case where the states need to be estimated from high-dimensional sensory data received only at some time instants. We develop a sampled-data feedback controller that is based on adaptations of a projected gradient descent method and includes neural networks as integral components to estimate the state of the system from perceptual information. We derive sufficient conditions to guarantee (local) input-to-state stability of the control loop. Moreover, we show that the interconnected system tracks the solution trajectory of the underlying optimization problem up to an error that depends on the approximation errors of the neural network and on the time-variability of the optimization problem; the latter originates from time-varying safety and performance objectives, input constraints, and unknown disturbances.

Downlink Power Allocation in Massive MIMO via Deep Learning: Adversarial Attacks and Training

The successful emergence of deep learning (DL) in wireless system applications has raised concerns about new security-related challenges. One such security challenge is adversarial attacks. Although there has been much work demonstrating the susceptibility of DL-based classification tasks to adversarial attacks, regression-based problems in the context of a wireless system have not been studied so far from an attack perspective. The aim of this paper is twofold: (i) we consider a regression problem in a wireless setting and show that adversarial attacks can break the DL-based approach and (ii) we analyze the effectiveness of adversarial training as a defensive technique in adversarial settings and show that the robustness of DL-based wireless system against attacks improves significantly. Specifically, the wireless application considered in this paper is the DL-based power allocation in the downlink of a multicell massive multi-input-multi-output system, where the goal of the attack is to yield an infeasible solution by the DL model. We extend the gradient-based adversarial attacks: fast gradient sign method (FGSM), momentum iterative FGSM, and projected gradient descent method to analyze the susceptibility of the considered wireless application with and without adversarial training. We analyze the deep neural network (DNN) models performance against these attacks, where the adversarial perturbations are crafted using both the white-box and black-box attacks.

Gradient-descent optimization of fermion nodes in the diffusion Monte Carlo technique

We present a method for optimizing the location of the fermion ground-state nodes using a combination of diffusion Monte Carlo (DMC) and projected gradient descent (PGD). A PGD iteration shifts the parameters of an arbitrary node-fixing trial function in the opposite direction of the DMC energy gradient, while maintaining the cusp condition for atomic electrons. The energy gradient is calculated from DMC walker distributions by one of three methods we derive from an exact analytical expression. We combine our energy gradient calculation methods with different gradient descent algorithms and a projection operator that maintains the cusp condition. We apply this stochastic PGD method to trial functions of Be, Li$_2$, and Ne, all consisting of a single Slater determinant with randomized parameters, and find that the nodes dramatically improve to the same DMC energy as nodes optimized by variational Monte Carlo. Our method, therefore, departs from the standard procedure of optimizing the nodes with a non-DMC scheme such as variational Monte Carlo, Density function theory, or configuration interaction based calculation, which do not directly minimize the DMC energy.

Minimum-monitor-unit optimization via a stochastic coordinate descent method

Objective. Deliverable proton spots are subject to the minimum monitor-unit (MMU) constraint. The MMU optimization problem with relatively large MMU threshold remains mathematically challenging due to its strong nonconvexity. However, the MMU optimization is fundamental to proton radiotherapy (RT), including efficient IMPT and proton arc delivery (ARC). This work aims to develop a new optimization algorithm that is effective in solving the MMU problem. Approach. Our new algorithm is primarily based on stochastic coordinate decent (SCD) method. It involves three major steps: first to decouple the determination of active sets for dose-volume-histogram (DVH) planning constraints from the MMU problem via iterative convex relaxation method; second to handle the nonconvexity of the MMU constraint via SCD to localize the index set of nonzero spots; third to solve convex subproblems projected to this convex set of nonzero spots via projected gradient descent method. Main results. Our new method SCD is validated and compared with alternating direction method of multipliers (ADMM) for IMPT and ARC. The results suggest SCD had better plan quality than ADMM, e.g. the improvement of conformal index (CI) from 0.56 to 0.69 during IMPT, and from 0.28 to 0.80 during ARC for the lung case. Moreover, SCD successfully handled the nonconvexity from large MMU threshold that ADMM failed to handle, in the sense that (1) the plan quality from ARC was worse than IMPT (e.g. CI was 0.28 with IMPT and 0.56 with ARC for the lung case), when ADMM was used; (2) in contrast, with SCD, ARC achieved better plan quality than IMPT (e.g. CI was 0.69 with IMPT and 0.80 with ARC for the lung case), which is compatible with more optimization degrees of freedom from ARC compared to IMPT. Significance. To the best of our knowledge, our new MMU optimization method via SCD can effectively handle the nonconvexity from large MMU threshold that none of the current methods can solve. Therefore, we have developed a unique MMU optimization algorithm via SCD that can be used for efficient IMPT, proton ARC, and other particle RT applications where large MMU threshold is desirable (e.g. for the delivery of high dose rates or/and a large number of spots).

Blind Super-Resolution of Point Sources via Projected Gradient Descent

Blind super-resolution can be cast as a low rank matrix recovery problem by exploiting the inherent simplicity of the signal and the low dimensional structure of point spread functions. In this paper, we develop a simple yet efficient non-convex projected gradient descent method for this problem based on the low rank structure of the vectorized Hankel matrix associated with the target matrix. Theoretical analysis indicates that the proposed method exactly converges to the target matrix with a linear convergence rate under the similar conditions as convex approaches. Numerical results show that our approach is competitive with existing convex approaches in terms of recovery ability and efficiency.

Iterative Deblending of Simultaneous-Source Seismic Data via a Robust Singular Spectrum Analysis Filter

We solve the simultaneous source separation problem by adopting the projected gradient descent (PGD) method to iteratively estimate the data one would acquire via a conventional seismic acquisition. The projection operator is a windowed robust singular spectrum analysis (SSA) filter that suppresses source interferences in the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$f-x$ </tex-math></inline-formula> (frequency-space) domain. We reformulate the SSA filter as a robust optimization problem solved via a bifactored gradient descent (BFGD) algorithm. Robustness becomes achievable by adopting Tukey’s biweight loss function for the design of the robust SSA filter. The SSA filter requires breaking down common-receiver gathers or common offset gathers into small overlapping windows. The traditional SSA method needs the filter rank as an input parameter, which can vary from window to window. The latter has been a shortcoming for the application of classical SSA filtering to complex seismic data processing. The proposed robust SSA filter is less sensitive to rank-selection, making it appealing for deblending applications that require windowing. Additionally, the robust SSA projection provides an effective attenuation of random source interferences during the initial iterations of the PGD method. Comparing classical and robust SSA filters, we also report an acceleration of the PGD method convergence when we adopt the robust SSA filter. Finally, we provide synthetic and real data examples, and discuss heuristic strategies for parameter selection.

Generating facial expression adversarial examples based on saliency map

The security demands in humanization should be considered as important, because artificial intelligence is developing rapidly in this area. Recent studies have shown the vulnerability of many deep learning models to adversarial examples; however, only a few studies on facial expression adversarial examples have been conducted. Thus, in this paper, we propose a novel method for generating facial expression adversarial examples using facial saliency maps and facial masking maps. Extensive numerical experiments demonstrate the outstanding performance of our method in terms of attack accuracy, structural similarity index measure score, and computational time, compared with other leading methods, such as the fast gradient sign method, projected gradient descent method, and Carlini–Wagner attacks.

Project Gradient Descent Adversarial Attack against Multisource Remote Sensing Image Scene Classification

Deep learning technology (a deeper and optimized network structure) and remote sensing imaging (i.e., the more multisource and the more multicategory remote sensing data) have developed rapidly. Although the deep convolutional neural network (CNN) has achieved state-of-the-art performance on remote sensing image (RSI) scene classification, the existence of adversarial attacks poses a potential security threat to the RSI scene classification task based on CNN. The corresponding adversarial samples can be generated by adding a small perturbation to the original images. Feeding the CNN-based classifier with the adversarial samples leads to the classifier misclassify with high confidence. To achieve a higher attack success rate against scene classification based on CNN, we introduce the projected gradient descent method to generate adversarial remote sensing images. Then, we select several mainstream CNN-based classifiers as the attacked models to demonstrate the effectiveness of our method. The experimental results show that our proposed method can dramatically reduce the classification accuracy under untargeted and targeted attacks. Furthermore, we also evaluate the quality of the generated adversarial images by visual and quantitative comparisons. The results show that our method can generate the imperceptible adversarial samples and has a stronger attack ability for the RSI scene classification.

Physarum Powered Differentiable Linear Programming Layers and Applications

Consider a learning algorithm, which involves an internal call to an optimization routine such as a generalized eigenvalue problem, a cone programming problem or even sorting. Integrating such a method as layers within a trainable deep network in a numerically stable way is not simple – for instance, only recently, strategies have emerged for eigendecomposition and differentiable sorting. We propose an efficient and differentiable solver for general linear programming problems which can be used in a plug and play manner within deep neural networks as a layer. Our development is inspired by a fascinating but not widely used link between dynamics of slime mold (physarum) and mathematical optimization schemes such as steepest descent. We describe our development and demonstrate the use of our solver in a video object segmentation task and meta-learning for few-shot learning. We review the relevant known results and provide a technical analysis describing its applicability for our use cases. Our solver performs comparably with a customized projected gradient descent method on the first task and outperforms the very recently proposed differentiable CVXPY solver on the second task. Experiments show that our solver converges quickly without the need for a feasible initial point. Interestingly, our scheme is easy to implement and can easily serve as layers whenever a learning procedure needs a fast approximate solution to a LP, within a larger network.

Sparse Analysis Model Based Dictionary Learning for Signal Declipping

Clipping is a common type of distortion in which the amplitude of a signal is truncated if it exceeds a certain threshold. Sparse representation has underpinned several algorithms developed recently for reconstruction of the original signal from clipped observations. However, these declipping algorithms are often built on a synthesis model, where the signal is represented by a dictionary weighted by sparse coding coefficients. In contrast to these works, we propose a sparse analysis-model-based declipping (SAD) method, where the declipping model is formulated on an analysis (i.e. transform) dictionary, and additional constraints characterizing the clipping process. The analysis dictionary is updated using the Analysis SimCO algorithm, and the signal is recovered by using a least-squares based method or a projected gradient descent method, incorporating the observable signal set. Numerical experiments on speech and music are used to demonstrate improved performance in signal to distortion ratio (SDR) compared to recent state-of-the-art methods including A-SPADE and ConsDL.

On the Adversarial Robustness of LASSO Based Feature Selection

In this paper, we investigate the adversarial robustness of feature selection based on the $\ell_1$ regularized linear regression model, namely LASSO. In the considered model, there is a malicious adversary who can observe the whole dataset, and then will carefully modify the response values or the feature matrix in order to manipulate the selected features. We formulate the modification strategy of the adversary as a bi-level optimization problem. Due to the difficulty of the non-differentiability of the $\ell_1$ norm at the zero point, we reformulate the $\ell_1$ norm regularizer as linear inequality constraints. We employ the interior-point method to solve this reformulated LASSO problem and obtain the gradient information. Then we use the projected gradient descent method to design the modification strategy. In addition, We demonstrate that this method can be extended to other $\ell_1$ based feature selection methods, such as group LASSO and sparse group LASSO. Numerical examples with synthetic and real data illustrate that our method is efficient and effective.

The stability of neural networks under condition of adversarial attacks to biomedical image classification

Recently, the majority of research and development teams working in the field deep learning are concentrated on the improvement of the classification accuracy and related measures of the quality of image classification whereas the problem of adversarial attacks to deep neural networks attracts much less attention. This article is dedicated to an experimental study of the influence of various factors on the stability of convolutional neural networks under the condition of adversarial attacks to biomedical image classification. On a very extensive dataset consisted of more than 1.45 million of radiological as well as histological images we assess the efficiency of attacks performed using the projected gradient descent (PGD), DeepFool and Carlini – Wagner (CW) methods. We analyze the results of both white and black box attacks to the commonly used neural architectures such as InceptionV3, Densenet121, ResNet50, MobileNet and Xception. The basic conclusion of this study is that in the field of biomedical image classification the problem of adversarial attack stays sharp because the methods of attacks being tested are successfully attacking the above-mentioned networks so that depending on the specific task their original classification accuracy falls down from 83–97 % down to the accuracy score of 15 %. Also, it was found that under similar conditions the PGD method is less successful in adversarial attacks comparing to the DeepFool and CW methods. When the original images and adversarial examples are compared using the L2-norm, the DeepFool and CW methods generate the adversarial examples of similar maliciousness. In addition, in three out of four of black-box attacks, the PGD method has demonstrated lower attacking efficiency.