Gradient Descent Direction Research Articles

Contrastive Unfolding Deraining Network.

Due to the fact that the degradation of image quality caused by rain usually affects outdoor vision tasks, image deraining becomes more and more important. Focusing on the single image deraining (SID) task, in this article, we propose a novel Contrastive Unfolding DEraining Network (CUDEN), which combines the traditional iterative algorithm and deep network, exhibiting excellent performance and nice interpretability. CUDEN transforms the challenge of locating rain streaks into discovering rain features and defines the relationship between the image and feature domains in terms of mapping pairs. To obtain the mapping pairs efficiently, we propose a dynamic multidomain translation (DMT) module for decomposing the original mapping into sub-mappings. To enhance the feature extraction capability of networks, we also propose a new serial multireceptive field fusion (SMF) block, which extracts complex and variable rain features with convolution kernels of different receptive fields. Moreover, we are the first to introduce contrastive learning to the SID task and combine it with perceptual loss to propose a new contrastive perceptual loss (CPL), which is quite generalized and greatly helpful in identifying the appropriate gradient descent direction during training. Extensive experiments on synthetic and real-world datasets demonstrate that our proposed CUDEN outperforms the state-of-the-art (SOTA) deraining networks.

QLABGrad: A Hyperparameter-Free and Convergence-Guaranteed Scheme for Deep Learning

The learning rate is a critical hyperparameter for deep learning tasks since it determines the extent to which the model parameters are adjusted during the learning course. However, the choice of learning rates typically depends on empirical judgment, which may not result in satisfactory outcomes without intensive try-and-error experiments. In this study, we propose a novel learning rate adaptation scheme called QLABGrad. Without any user-specified hyperparameter, QLABGrad automatically determines the learning rate by optimizing the quadratic loss approximation-based (QLAB) function for a given gradient descent direction, where only one extra forward propagation is required. We theoretically prove the convergence of QLABGrad under the smooth Lipschitz condition on the loss function. Experiment results on multiple architectures, including MLP, CNN, and ResNet, on MNIST, CIFAR10, and ImageNet datasets, demonstrate that QLABGrad outperforms widely adopted schemes for deep learning.

Unrolled Optimization via Physics-Assisted Convolutional Neural Network for MR-Based Electrical Properties Tomography: A Numerical Investigation.

Magnetic Resonance imaging based Electrical Properties Tomography (MR-EPT) is a non-invasive technique that measures the electrical properties (EPs) of biological tissues. In this work, we present and numerically investigate the performance of an unrolled, physics-assisted method for 2D MR-EPT reconstructions, where a cascade of Convolutional Neural Networks is used to compute the contrast update. Each network takes in input the EPs and the gradient descent direction (encoding the physics underlying the adopted scattering model) and returns as output the updated contrast function. The network is trained and tested in silico using 2D slices of realistic brain models at 128 MHz. Results show the capability of the proposed procedure to reconstruct EPs maps with quality comparable to that of the popular Contrast Source Inversion-EPT, while significantly reducing the computational time.

Green non-fluorinated gradient anisotropic microcolumn array: Enables droplet transition to a low viscosity state on highly viscous substrates

Superhydrophobic surfaces with low adhesion and self-cleaning properties have broad application prospected in various fields. However, the current methods for preparing low-adhesion superhydrophobic surfaces typically require toxic chemicals or fluorination modifications, which are prone to environmental pollution and health hazards, and still face challenges for applications in industries such as food and medical care. This paper aims to prepare anisotropic gradient array surfaces (AGAS) with low adhesion on stainless steel substrates by WEDM, without the need for modification with low-surface-energy materials. AGAS can achieve a stable Cassie state with a low adhesion performance by controlling the dimensional parameters and structural inclination of columnar arrays. Experiments and numerical simulations show that owing to the anisotropic gradient structure, pinning and de-pinning occur asymmetrically in time at the contact lines of the leading and trailing edges of droplets impacting the AGAS. On AGAS, droplets can jump and roll in the direction of gradient descent during the impact process to complete self-cleaning. Due to the discontinuity of the solid–liquid–gas three-phase contact line and the stepped arrangement of the column array, the prepared surface can achieve low adhesion and self-cleaning properties for many common liquid foods in daily life, including milk and juice.

Improvement accuracy in deep learning: An increasing neurons distance approach with the penalty term of loss function

The increasing use of neural networks for solving complex tasks has emphasized the need to optimize their performance. In recent years, the development of neural networks has enabled them to tackle increasingly challenging problems for applications in various fields. Existing methods, such as L1 and L2 penalty terms, improve the neural network performance by reducing the neurons' weight and removing some neurons' output. However, these algorithms reduce the number of active neurons and the theoretical maximum capacity of the neural network. A loss function penalty term can improve the network performance by guiding the gradient descent direction of neurons during the training process. Moreover, it does not need to alter the network structure, increase the data, or change the training process. In this study, a novel algorithm is proposed to improve the performance of the network by adding a new loss function penalty term. The algorithm proposed in this study enhances the accuracy and anti-interference capabilities of neural networks by increasing the number of active neurons based on the finding that identical neurons extract identical features from the data, while distinct neurons extract unique features. The neurons are treated as hyperspace points and the distance between them is converted into repulsive forces to increase the distance between these points. To verify the validity of the proposed algorithm, we tested it on image classification datasets (CIFAR-100 and CALTECH-256), text datasets (SST-2 and THUCNews), and an image detection dataset (PASCAL VOC). The results show that the proposed algorithm can improve the network performance for a variety of neural network structures, data types, and task types. This algorithm has demonstrated its effectiveness in robustness against data noise and compatibility with pre-trained models such as Faster R-CNN, YOLO, and RetinaNet.

Dual-scale similarity-guided cycle generative adversarial network for unsupervised low-dose CT denoising

Removing the noise in low-dose CT (LDCT) is crucial to improving the diagnostic quality. Previously, many supervised or unsupervised deep learning-based LDCT denoising algorithms have been proposed. Unsupervised LDCT denoising algorithms are more practical than supervised ones since they do not need paired samples. However, unsupervised LDCT denoising algorithms are rarely used clinically due to their unsatisfactory denoising ability. In unsupervised LDCT denoising, the lack of paired samples makes the direction of gradient descent full of uncertainty. On the contrary, paired samples used in supervised denoising allow the parameters of networks to have a clear direction of gradient descent. To bridge the gap in performance between unsupervised and supervised LDCT denoising, we propose dual-scale similarity-guided cycle generative adversarial network (DSC-GAN). DSC-GAN uses similarity-based pseudo-pairing to better accomplish unsupervised LDCT denoising. We design a Vision Transformer-based global similarity descriptor and a residual neural network-based local similarity descriptor for DSC-GAN to effectively describe the similarity between two samples. During training, pseudo-pairs, i.e., similar LDCT samples and normal-dose CT (NDCT) samples, dominate parameter updates. Thus, the training can achieve equivalent effect as training with paired samples. Experiments on two datasets demonstrate that DSC-GAN beats the state-of-the-art unsupervised algorithms and reaches a level close to supervised LDCT denoising algorithms.

Service life of RC seawall under chloride invasion: A theoretical model incorporating convection-diffusion effect

This paper presents a theoretical approach, which comprises the chloride transportation model and cover cracking model, to predict the service life of RC seawall under chloride invasion. The chloride transportation model combined the Fick's second law and Darcy's law to model the convection-diffusion of chloride ions in the RC seawall. The governing equation for chloride migration in the RC seawall is solved with the separation of variables technique. The proposed model is validated by comparing with a corresponding finite difference solution and the experimental results. The mechanical model that represents the cracking of the concrete cover induced by the expansion pressure of the corrosion products is finally developed to predict the service life of the RC seawall under chloride invasion. Parametric studies are performed to illustrate the effects of convection-diffusion and other key variables on the chloride ions transportation and service life of the RC seawall. The results reveal that the convection effect substantially accelerates the chloride penetration process in the chloride gradient descent direction while inhibits the penetration in the opposite direction. The proposed analytical solution in conjunction with the mechanical model provides a theoretical approach for predicting the service life of RC seawall, which can be used to aid durability design of coastal structures.

Differential Privacy-Enabled Multi-Party Learning with Dynamic Privacy Budget Allocating Strategy

As one of the promising paradigms of decentralized machine learning, multi-party learning has attracted increasing attention, owing to its capability of preventing the privacy of participants from being directly exposed to adversaries. Multi-party learning enables participants to train their model locally without uploading private data to a server. However, recent studies have shown that adversaries may launch a series of attacks on learning models and extract private information about participants by analyzing the shared parameters. Moreover, existing privacy-preserving multi-party learning approaches consume higher total privacy budgets, which poses a considerable challenge to the compromise between privacy guarantees and model utility. To address this issue, this paper explores an adaptive differentially private multi-party learning framework, which incorporates zero-concentrated differential privacy technique into multi-party learning to get rid of privacy threats, and offers sharper quantitative results. We further design a dynamic privacy budget allocating strategy to alleviate the high accumulation of total privacy budgets and provide better privacy guarantees, without compromising the model’s utility. We inject more noise into model parameters in the early stages of model training and gradually reduce the volume of noise as the direction of gradient descent becomes more accurate. Theoretical analysis and extensive experiments on benchmark datasets validated that our approach could effectively improve the model’s performance with less privacy loss.

Construction method of hybrid quantum long-short term memory neural network for image classification

Long-short term memory (LSTM) neural network solves the problems of long-term dependence, gradient disappearance and gradient explosion by introducing memory units, and is widely used in time series analysis and prediction. Combining quantum computing with LSTM neural network will help to improve its computational efficiency and reduce the number of model parameters, thus significantly improving the performance of traditional LSTM neural network. This paper proposes a hybrid quantum LSTM (hybrid quantum long-short term memory, HQLSTM) network model that can be used to realize the image classification. It uses variable quantum circuits to replace the nerve cells in the classical LSTM network to realize the memory function of the quantum network. At the same time, it introduces Choquet integral operator to enhance the degree of aggregation between data. The memory cells in the HQLSTM network are composed of multiple variation quantum circuits (VQC) that can realize different functions. Each VQC consists of three parts: the coding layer, which uses angle coding to reduce the complexity of network model design; the variation layer, which is designed with quantum natural gradient optimization algorithm, so that the gradient descent direction does not target specific parameters, thereby optimizing the parameter update process and improving the generalization and convergence speed of the network model; the measurement layer, which uses the Pauli Z gate to measure, and the expected value of the measurement result is input to the next layer to extract useful information from the quantum circuit. The experimental results on the MNIST, FASHION-MNIST and CIFAR datasets show that the HQLSTM model achieves higher image classification accuracy and lower loss value than the classical LSTM model and quantum LSTM model. At the same time, the network space complexity of HQLSTM and quantum LSTM are significantly reduced compared with the classical LSTM network.

A gradient descent direction based-cumulants method for probabilistic energy flow analysis of individual-based integrated energy systems

This study presents an individual-based model of heterogeneous integrated energy systems, which is composed of sub-energy system individuals and coupling individuals. Different evolution rules are depicted within one individual to form an accurate model of a complex system. Based on the model, a gradient descent direction iterative method (GDDM) is proposed for energy flow calculation of integrated energy systems to improve its convergence performance. Furthermore, a GDDM-based cumulants method (GDDM-CM) is presented to analyse the probabilistic energy flow distribution throughout the system. To verify the effectiveness of the proposed method, simulation studies have been undertaken in a modified complex integrated energy system. The results show that GDDM performs better than non-gradient descent method (NGDM). It converges faster and is insensitive to the initial point. Besides, GDDM-CM can obtain similar results as monte-carlo sampling method does and its calculation accuracy is better than that of the point estimate method. Moreover, the proposed GDDM-CM can greatly reduce the calculation time compared with the other two methods. Utilizing the GDDM-CM, the coupling effects of different individuals are investigated and the weak points in each individual can be identified.

Mean wind flow reconstruction of a high-rise building based on variational data assimilation using sparse pressure measurements

The paper investigates the applicability of the variational data assimilation approach to reconstruct three-dimensional wind flows around a high-rise building model, and provides guidelines towards an efficient reconstruction. The objective is to determine the best distributed control parameters based on the Reynolds-averaged-Navier–Stokes (RANS) model coupled with pressure data. The strategy considers a distributed additive forcing control parameter, acting on the momentum equation and/or the transport equation of turbulent dissipations, where the closure is performed. To avoid unphysical solutions and accelerate the optimization convergence a Sobolev gradient descent regularization is proposed and compared with penalty techniques. The proposed data assimilation framework is applied to a realistic wind engineering estimation problem combining sparse wall pressure measurements with a routinely used industrial three-dimensional RANS numerical code. Results demonstrate that wall-pressure data is a meaningful enough piece of information to recover accurately the wake flow extension. A hybrid control parameter on the transport of the dissipation and the momentum equations leads to very precise results, whereas the Sobolev gradient descent direction gives efficient regularization and a fast convergence. These findings can be useful for future application of such a data assimilation technique to wind engineering problems.

On the Sparse Gradient Denoising Optimization of Neural Network Models for Rolling Bearing Fault Diagnosis Illustrated by a Ship Propulsion System

The drive rolling bearing is an important part of a ship’s system; the detection of the drive rolling bearing is an important component in ship-fault diagnosis, and machine learning methods are now widely used in the fault diagnosis of rolling bearings. However, training methods based on small batches have a disadvantage in that the samples which best represent the gradient descent direction can be disturbed by either other samples in the opposite direction or anomalies. Aiming at this problem, a sparse denoising gradient descent (SDGD) optimization algorithm, based on the impact values of network nodes, was proposed to improve the updating method of the batch gradient. First, the network is made sparse by using the node weight method based on the mean impact value. Second, the batch gradients are clustered via a distribution-density-based clustering method. Finally, the network parameters are updated using the gradient values after clustering. The experimental results show the efficiency and feasibility of the proposed method. The SDGD model can achieve up to a 2.35% improvement in diagnostic accuracy compared to the traditional network diagnosis model. The training convergence speed of the SDGD model improves by 2.16%, up to 17.68%. The SDGD model can effectively solve the problem of falling into the local optimum point while training a network.

Gradient Descent-Based Objective Function Reformulation for Finite Control Set Model Predictive Current Control With Extended Horizon

This article proposes a novel objective function formulation based on gradient descent (GD) for finite control set predictive current control (FCS-PCC) with extended horizon. FCS-PCC has become increasingly attractive for electrical drive applications owing to its short settling time, lower switching frequency, capability to handle multiple conflicting targets, and feasible inclusion of constraints. However, it still suffers from high-torque ripple and poor current quality at the steady state. To tackle the aforementioned issue, a GD-based objective function reformulation is employed in the FCS-PCC with extended horizon. First, the optimization problem underlying FCS-PCC is formulated as a constrained quadratic programming problem with proved convexity from a geometric perspective. Based on the above, the tracking error of the control objective is minimized more efficiently by searching along the direction of GD. Consequently, the objective function is reconstructed as the deviation between the normalized GD and derivative, combined with the extension of feasible set. The abovementioned procedures are iteratively learned in every prediction horizon. The effectiveness of the proposed algorithm is verified on a 2.2-kW induction machine platform with a prediction horizon of <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$N$</tex-math></inline-formula> = 3. It is confirmed that the proposed algorithm outperforms the conventional and multistep FCS-PCC in steady state and transient state.

A hybrid‐driven continuous‐time filter for manoeuvering target tracking

This paper considers the problem of target tracking in complex manoeuvering scenarios with a lack of relevant prior knowledge. This is a challenge for classical model-based manoeuvering target tracking algorithms because they rely heavily on accurate domain and prior knowledge of target motion. To address this problem, we propose a hybrid-driven continuous-time filter algorithm in this paper, which combines the advantages of the model-driven and data-driven. We use the stochastic differential equation (SDE) with the acceleration model as the basic framework of the proposed algorithm. In order to deal with unpredictable manoeuvres and unknown perturbations, we adopt neural networks as data-driven to estimate target accelerations and compensations of composite perturbations in real time using historical and current measurements. And the direction of gradient descent of the neural network is constrained by the motion model, thus the learning efficiency of the network and the interpretability of the proposed algorithm is improved. The proposed algorithm can simultaneously utilise historical trajectory information and domain knowledge as hybrid-driven to achieve complex manoeuvering target tracking with little prior information. Experimental results demonstrate the superiority of our algorithm in tracking accuracy, robustness and environmental adaptability.

Active Neuron Least Squares: A Training Method for Multivariate Rectified Neural Networks

We propose a novel training method for multivariate two-layer rectified neural networks. The core mechanism is the augmentation of standard gradient descent direction by the inclusion of search vectors which are chosen to explicitly adjust the activation patterns of the neurons. Active neuron least squares (ANLS) proceeds iteratively with each iteration consisting of three steps: (a) generation of search vectors (including ones designed to change activation patterns), (b) identification of the candidate weights, and (c) a decision on which candidate weights are selected for update. We develop stable and efficient procedures for implementing the method. Numerical examples are provided that demonstrate the effectiveness of ANLS compared with existing popular approaches on various learning tasks ranging from function approximation to solving PDEs.

RMOCG: A Riemannian Manifold Optimization-Based Conjugate Gradient Method for Phase-Only Beamforming Synthesis

The phase-only null beamforming synthesis is widely used in communication, radar, and sonar. Due to the phase-only constraint (constant modulus constraint), the problem is nonconvex and NP-hard. The existing methods mainly solve the problem in an indirect way, by relaxing the problem to a more tractable form. Nevertheless, these methods either need huge computational cost or degrade the performance. To break the tradeoffs between the performance and computational complexity, a direct method is proposed without relaxation. In the proposed method, the problem is first transformed into an unconstrained problem on a complex circle manifold. Then, by deriving the gradient descent direction and the step size, a Riemannian manifold optimization-based conjugate gradient algorithm is derived to obtain the direct solution for the transformed problem. Compared with the existing methods, the proposed method can achieve 4 dB null gain reduction and 1 magnitude complexity decreasing.

Information geometry of physics-informed statistical manifolds and its use in data assimilation

The data-aware method of distributions (DAMD) is a low-dimensional data assimilation procedure to forecast the behavior of dynamical systems described by differential equations. The core of DAMD is the minimization of a distance between an observation and a prediction in distributional terms, with prior and posterior distributions constrained to a statistical manifold defined by the method of distributions (MD). We leverage the information-geometric properties of the statistical manifold to reduce predictive uncertainty via data assimilation. Specifically, we exploit the information-geometric structures induced by two discrepancy metrics, the Kullback-Leibler divergence and the Wasserstein distance, which explicitly yield natural gradient descent. The use of a deep neural network as a surrogate model for MD enables automatic differentiation, further accelerating optimization. The manifold's geometry is quantified without sampling, yielding an accurate approximation of the gradient descent direction. Our numerical experiments demonstrate that accounting for the manifold's geometry significantly reduces the computational cost of data assimilation by both facilitating the calculation of gradients and reducing the number of required iterations.

Stability analysis of proportional delayed projection neural network for quadratic programming problem

At present, projection neural network (PNN) with bounded time delay has been widely used for solving convex quadratic programming problem (QPP). However, there is little research concerning PNN with unbounded time delay. In this paper, we propose the proportional delayed PNN to solve QPP with equality constraints. By utilizing homeomorphism mapping principle, we prove the proportional delayed PNN exists with unique equilibrium point which is the optimal solution of QPP. Simultaneously, delay-dependent criteria about global exponential stability (GES) and global polynomial stability (GPS) are also acquired by applying the method of variation of constants and inequality techniques. On the other hand, when proportional delay factor [Formula: see text] is equal to 1, the proportional delayed PNN becomes the one without time delay which still can be utilized for solving QPP. But in most situations, [Formula: see text] is not equal to 1, and time delay is unpredictable and may be unbounded in the actual neural network, which causes instability of system. Therefore, it is necessary to consider proportional delayed PNN. A numerical example demonstrates that, compared with the proportional delayed Lagrange neural network, the proportional delayed PNN is faster in terms of convergence rate. The possible reason is that appropriate parameters make the model converge to the equilibrium point along the direction of gradient descent.

A Superpixel-Guided Unsupervised Fast Semantic Segmentation Method of Remote Sensing Images

Semantic segmentation is one of the fundamental tasks of pixel-level remote sensing image analysis. Currently, most high-performance semantic segmentation methods are trained in a supervised learning manner. These methods require a large number of image labels as support, but manual annotations are difficult to obtain. To address the problem, we propose an efficient unsupervised remote sensing image segmentation method based on superpixel segmentation and fully convolutional networks (FCNs) in this letter. Our method can achieve pixel-level images segmentation of various scales rapidly without any manual labels or prior knowledge. We use the superpixel segmentation results as synthetic ground truth to guide the gradient descent direction during FCN training. In experiments, our method achieved high performance compared to current unsupervised image segmentation methods on three public datasets. Specifically, our method achieves an Adjusted Mutual Information (AMI) score of 0.2955 on the Gaofen Image Dataset (GID) dataset, while processing each image of size 7200 &#x00D7; 6800 pixels in just 30 seconds.

Blade shape optimization and internal-flow characteristics of the backward non-volute centrifugal fan

The work proposed the double parameter optimization method of the non-volute centrifugal fan’s blade profile based on the steepest descent method. Total-pressure efficiency improvement at the high-flow area was taken as an optimization objective. A method of applying the steepest descent method to modify the blade profile of backward centrifugal fan is proposed in this paper. The gradient descent direction was analyzed to design the blade profile and obtain the optimal blade profile at a high-flow rate. Besides, numerical simulations were carried out to analyze the aerodynamic performance and the internal flow characteristics of the centrifugal fan by the computational fluid dynamics method. Numerical results showed that the blade profile along the gradient descent was optimized to effectively increase the total pressure and the total pressure efficiency of the original model at the high-flow rate. The steepest descent method for local optimization could improve the fan blade design.