Fast Gradient Method Research Articles

DPNN-ac4C: A Dual-Path Neural Network with self-attention mechanism for identification of N4‑acetylcytidine (ac4C) in mRNA.

The modification of N4-acetylcytidine (ac4C) in RNA is a conserved epigenetic mark that plays a crucial role in post-transcriptional regulation, mRNA stability, and translation efficiency. Traditional methods for detecting ac4C modifications are laborious and costly, necessitating the development of efficient computational approaches for accurate identification of ac4C sites in mRNA. We present DPNN-ac4C, a dual-path neural network with a self-attention mechanism for the identification of ac4C sites in mRNA. Our model integrates embedding modules, bidirectional GRU networks, convolutional neural networks, and self-attention to capture both local and global features of RNA sequences. Extensive evaluations demonstrate that DPNN-ac4C outperforms existing models, achieving an AUROC of 91.03%, accuracy of 82.78%, MCC of 65.78%, and specificity of 84.78% on an independent test set. Moreover, DPNN-ac4C exhibits robustness under the Fast Gradient Method (FGM) attack, maintaining a high level of accuracy in practical applications. The model code and dataset are publicly available on GitHub (https://github.com/shock1ng/DPNN-ac4C).

Named Entity Recognition for Crop Diseases and Pests Based on Gated Fusion Unit and Manhattan Attention

Named entity recognition (NER) is a crucial step in building knowledge graphs for crop diseases and pests. To enhance NER accuracy, we propose a new NER model—GatedMan—based on the gated fusion unit and Manhattan attention. GatedMan utilizes RoBERTa as a pre-trained model and enhances it using bidirectional long short-term memory (BiLSTM) to extract features from the context. It uses a gated unit to perform weighted fusion between the outputs of RoBERTa and BiLSTM, thereby enriching the information flow. The fused output is then fed into a novel Manhattan attention mechanism to capture the long-range dependencies. The global optimum tagging sequence is obtained using the conditional random fields layer. To enhance the model’s robustness, we incorporate adversarial training using the fast gradient method. This introduces adversarial examples, allowing the model to learn more disturbance-resistant feature representations, thereby improving its performance against unknown inputs. GatedMan achieved F1 scores of 93.73%, 94.13%, 93.98%, and 96.52% on the AgCNER, Peoples_daily, MSRA, and Resume datasets, respectively, thereby outperforming the other models. Experimental results demonstrate that GatedMan accurately identifies entities related to crop diseases and pests and exhibits high generalizability in other domains.

Hybrid two-level protection system for preserving pre-trained DNN models ownership

AbstractRecent advancements in deep neural networks (DNNs) have made them indispensable for numerous commercial applications. These include healthcare systems and self-driving cars. Training DNN models typically demands substantial time, vast datasets and high computational costs. However, these valuable models face significant risks. Attackers can steal and sell pre-trained DNN models for profit. Unauthorised sharing of these models poses a serious threat. Once sold, they can be easily copied and redistributed. Therefore, a well-built pre-trained DNN model is a valuable asset that requires protection. This paper introduces a robust hybrid two-level protection system for safeguarding the ownership of pre-trained DNN models. The first-level employs zero-bit watermarking. The second-level incorporates an adversarial attack as a watermark by using a perturbation technique to embed the watermark. The robustness of the proposed system is evaluated against seven types of attacks. These are Fast Gradient Method Attack, Auto Projected Gradient Descent Attack, Auto Conjugate Gradient Attack, Basic Iterative Method Attack, Momentum Iterative Method Attack, Square Attack and Auto Attack. The proposed two-level protection system withstands all seven attack types. It maintains accuracy and surpasses current state-of-the-art methods.

Combining permuted language model and adversarial training for Chinese machine reading comprehension

Due to the polysemy and complexity of the Chinese language, Chinese machine reading comprehension has always been a challenging task. To improve the semantic understanding and robustness of Chinese machine reading comprehension models, we propose a model that utilizes adversarial training algorithms and Permuted Language Model (PERT). Firstly, we employ the PERT pre-training model to embed paragraphs and questions into vector space to obtain corresponding sequential representations. Secondly, we use a multi-head self-attention mechanism to extract key textual information from the sequence and employ a Bi-GRU network to semantically fuse the output feature vectors, aiming to learn deep semantic representations in the text. Finally, we introduce perturbations into the model training process. We achieve this by utilizing adversarial training algorithms such as Fast Gradient Method (FGM) and Projected Gradient Descent (PGD). These algorithms generate adversarial samples to enhance the model’s robustness and stability when facing diverse inputs. We conducted comparative experiments on the publicly available Chinese reading comprehension datasets CMRC2018 and DRCD. The experimental results show that our proposed model has achieved significant improvements in both EM and F1-Score compared to the baseline model. To validate the model’s generalization and robustness, we utilized ChatGPT to construct a scientific dataset that includes a large number of domain-specific terms, sentences with mixed Chinese and English, and complex comprehension tasks. Our model also performed remarkably well on the self-built dataset. In conclusion, the proposed model not only effectively enhances the understanding of semantic information in Chinese text but also demonstrates a certain level of generalization capability.

Friendly Attacks to Improve Channel Coding Reliability

This paper introduces a novel approach called "friendly attack" aimed at enhancing the performance of error correction channel codes. Inspired by the concept of adversarial attacks, our method leverages the idea of introducing slight perturbations to the neural network input, resulting in a substantial impact on the network's performance. By introducing small perturbations to fixed-point modulated codewords before transmission, we effectively improve the decoder's performance without violating the input power constraint. The perturbation design is accomplished by a modified iterative fast gradient method. This study investigates various decoder architectures suitable for computing gradients to obtain the desired perturbations. Specifically, we consider belief propagation (BP) for LDPC codes; the error correcting code transformer, BP and neural BP (NBP) for polar codes, and neural BCJR for convolutional codes. We demonstrate that the proposed friendly attack method can improve the reliability across different channels, modulations, codes, and decoders. This method allows us to increase the reliability of communication with a legacy receiver by simply modifying the transmitted codeword appropriately.

A 1-bit DACs precoding for MU-MIMO based on binary equilibrium constraint: An alternating direction method

High power consumption is one of the main problems in the practical application of massive multi-user multi-input-multi-output (MU-MIMO) systems, and using 1-bit digital-to-analog converters (DACs) instead of high-resolution DACs is an effective way to reduce the power consumption of massive MU-MIMO systems. How to design efficient precoding schemes to improve the performance of 1-bit MU-MIMO systems has become a hot research field. This paper mainly studies the precoding problem of 1-bit DACs based on minimum mean squared error (MMSE). We first transform the non-convex constraints of the traditional optimization problem using binary equilibrium constraints. Then, to solve the transformed objective problem, we establish a framework based on the alternating direction method (ADM), use the fast projection gradient (FPG) method, and propose an alternating maximum minimum (AMM) algorithm to solve the optimization variables alternately. We have verified the convergence of the proposed algorithm and analyzed the accuracy of the proposed ADM framework and the computational complexity of the proposed algorithm in detail. In the simulation section, we analyze the uncoded bit error rate (BER) performance of the proposed algorithm in phase-shift keying (PSK) modulation and quadrature amplitude modulation (QAM) modes. The simulation results show that compared with existing advanced algorithms, the proposed algorithm can achieve performance advantages of about 1 dB and 2 dB in QPSK modulation mode and 16QAM modulation mode, respectively.

Robust stabilization of LTI negative imaginary systems using the nearest negative imaginary controller

AbstractThis paper considers the problem of robust stabilization of linear time‐invariant systems with respect to unmodelled dynamics and structure uncertainties. To that end, a methodology to find the nearest negative imaginary system for a given non‐negative imaginary system is presented first. Then, this result is employed to construct a near optimal linear quadratic Gaussian controller achieving desired performance measures. The problem is formulated using port‐Hamiltonian method and the required conditions are defined in terms of linear matrix inequalities. The technique is presented using the fast gradient method to solve the problem systematically. The designed controller satisfies a negative imaginary property and guarantees a robust feedback loop. The effectiveness of the approach is demonstrated by a simulation on a numerical example.

Adversarial Defensive Framework for State-of-Health Prediction of Lithium Batteries

Neural Networks are subject to malicious data poisoning attacks affecting the ability of the model to make accurate predictions. The attacks are generated using adversarial techniques imperceptible to the human eye since they use minimal noise to alter features which end up affecting boundary decisions of the prediction Model. Predicting lithium-ion batteries' State of Health (SOH) in a adversarial context becomes a challenging task especially if the model is expected to always predict at a very high accuracy level. Our article presents three novel contributions. The first contribution is a SOH prediction model that shows one of the best accuracy rates in the literature ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$R2=99.82\%$</tex-math></inline-formula> ) and yet uses the simplest LSTM model configuration compared to literature. The second contribution is the implementation of three state-of-the-art adversarial data poisoning attacks at decision time, knowingly Fast Gradient Method (FGSM), Momentum Iterative Method (MIM) and Basic Iterative Method (BIM) and the assessment of their impact on the original prediction accuracy. Most of literature use the attacks in a classification context while we are applying it to a time series prediction context. The third and most important contribution of this article is presenting a generic defense strategy combined with a feature engineering method that can be genrelized to prevent any potential adversarial attack attempt on any prediction model in a time-series prediction context using the same suggested feature engineering approach as suggested in this paper. The accuracy of the model is assessed using error estimators Root Mean Square Error (RMSE), Mean Absolute Error (MAE) and the R square (R2).The results show that adversarial data poisoning attacks are lethal to a time-series prediction model and our proposed defense strategy is able to detect and flag the existence of malicious data using a Simple Vector Machine (SVM) classifier with a very high confidence rate ( <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$area\,uder\,curve=0.996$</tex-math></inline-formula> ) which allow our model to defend against any potential unseen adversarial attack.

A fast proximal gradient method and convergence analysis for dynamic mean field planning

In this paper, we propose an efficient and flexible algorithm to solve dynamic mean-field planning problems based on an accelerated proximal gradient method. Besides an easy-to-implement gradient descent step in this algorithm, a crucial projection step becomes solving an elliptic equation whose solution can be obtained by conventional methods efficiently. By induction on iterations used in the algorithm, we theoretically show that the proposed discrete solution converges to the underlying continuous solution as the grid becomes finer. Furthermore, we generalize our algorithm to mean-field game problems and accelerate it using multilevel and multigrid strategies. We conduct comprehensive numerical experiments to confirm the convergence analysis of the proposed algorithm, to show its efficiency and mass preservation property by comparing it with state-of-the-art methods, and to illustrate its flexibility for handling various mean-field variational problems.

Model Predictive Control for Six-Step Operation of PMSM Based on Adapted Fast Gradient Method

This paper proposes model predictive control (MPC) of permanent magnet synchronous machines covering both the space vector modulation and the six-step operation. According to the estimated stator flux and its reference trajectory corresponding to modulation methods, the optimization problem is formulated and solved through the adapted fast gradient method (FGM). The proposed FGM performs the projection operation through the offset voltage injection method and calculates gradients by dynamic programming to reduce the computational cost. Based on the proposed FGM, the MPC is devised with the system state compensation under the six-step operation. The validity of the proposed work is verified by simulation and experiments. By the proposed MPC, the flux regulation reveals optimized dynamic performance under the continuous transition between the space vector modulation and the six-step operation. Moreover, the computation time of the gradient calculation and projection operation are respectively reduced by 52% and 9.1%, compared to the conventional method.

Chinese Event Extraction Method Based on Roformer Model

Event extraction is an important research direction in the field of natural language processing. The current Chinese event extraction field still suffers from errors in the pretraining and fine-tuning stages, inability to directly handle texts with more than 512 tokens, and inaccurate event extraction due to insufficient semantic sample diversity. In this paper, we propose a Chinese event extraction method RoformerFC (Roformer model with FGM and CRF) based on the Roformer model to address the above problems. Firstly, our method utilizes the Roformer model based on rotary position embedding, which both moderates the errors in the pretraining and fine-tuning phases and allows the model to directly handle texts with more than 512 tokens; then, the adversarial networks based on FGM (fast gradient method) are realized to increase the diversity of semantic feature samples; finally, the classical CRF (conditional random fields) model is used to decode and identify the event element entity and its corresponding event role and event type. On the short text DuEE dataset, the microP, microR, and microF of our method improved 1.26%, 4.01%, and 2.68%, respectively, over the classical Chinese event extraction method BERT-CRF. On the long text JsEE dataset, the microP, microR, and microF of our method improved 2.26%, 5.03%, and 3.72%, respectively, over the classical Chinese event extraction method BERT-CRF.

Small Target Detection Method Based on Low-Rank Sparse Matrix Factorization for Side-Scan Sonar Images

Target detection in side-scan sonar images plays a significant role in ocean engineering. However, the target images are usually severely interfered by the complex background and strong environmental noise, which makes it difficult to extract robust features from small targets and makes the target detection task quite challenging. In this paper, a novel small target detection method in sonar images is proposed based on the low-rank sparse matrix factorization. Initially, the side-scan sonar images are preprocessed so as to highlight the individual differences of the target. Then, the problems of target feature extraction and noise removal are characterized as the problem of matrix decomposition. An improved Robust Principal Component Analysis algorithm is used to extract target information, and the fast proximal gradient method is used to optimize the solution. The original sonar image is reconstructed into the low-rank background matrix, the sparse target matrix, and the noise matrix. Eventually, a morphological operation is used to filter out the noise and refine the target edges in the target matrix for improving the accuracy of target detection. Experimental results show that the proposed method not only achieves better detection performance in comparison to the conventional baseline algorithms but also performs robustly in various signal-to-clutter ratio conditions.

Boosting Adversarial Attacks with Nadam Optimizer

Deep neural networks are extremely vulnerable to attacks and threats from adversarial examples. These adversarial examples deliberately crafted by attackers can easily fool classification models by adding imperceptibly tiny perturbations on clean images. This brings a great challenge to image security for deep learning. Therefore, studying and designing attack algorithms for generating adversarial examples is essential for building robust models. Moreover, adversarial examples are transferable in that they can mislead multiple different classifiers across models. This makes black-box attacks feasible for practical applications. However, most attack methods have low success rates and weak transferability against black-box models. This is because they often overfit the model during the production of adversarial examples. To address this issue, we propose a Nadam iterative fast gradient method (NAI-FGM), which combines an improved Nadam optimizer with gradient-based iterative attacks. Specifically, we introduce the look-ahead momentum vector and the adaptive learning rate component based on the Momentum Iterative Fast Gradient Sign Method (MI-FGSM). The look-ahead momentum vector is dedicated to making the loss function converge faster and get rid of the poor local maximum. Additionally, the adaptive learning rate component is used to help the adversarial example to converge to a better extreme point by obtaining adaptive update directions according to the current parameters. Furthermore, we also carry out different input transformations to further enhance the attack performance before using NAI-FGM for attack. Finally, we consider attacking the ensemble model. Extensive experiments show that the NAI-FGM has stronger transferability and black-box attack capability than advanced momentum-based iterative attacks. In particular, when using the adversarial examples produced by way of ensemble attack to test the adversarially trained models, the NAI-FGM improves the success rate by 8% to 11% over the other attack methods. Last but not least, the NAI-DI-TI-SI-FGM combined with the input transformation achieves a success rate of 91.3% on average.

User Sentiment Analysis of COVID-19 via Adversarial Training Based on the BERT-FGM-BiGRU Model

With the rapid development of social network platforms, Sina Weibo has become the main carrier for modern netizens to express public views and emotions. How to obtain the tendency of public opinion and analyze the text’s emotion more accurately and reasonably has become one of the main challenges for the government to monitor public opinion in the future. Due to the sparseness of Weibo text data and the complex semantics of Chinese, this paper proposes an emotion analysis model based on the Bidirectional Encoder Representation from Transformers pre-training model (BERT), Fast Gradient Method (FGM) and the bidirectional Gated Recurrent Unit (BiGRU), namely BERT-FGM-BiGRU model. Aiming to solve the problem of text polysemy and improve the extraction effect and classification ability of text features, this paper adopts the BERT pre-training model for word vector representation and BiGRU for text feature extraction. In order to improve the generalization ability of the model, this paper uses the FGM adversarial training algorithm to perturb the data. Therefore, a BERT-FGM-BiGRU model is constructed with the goal of sentiment analysis. This paper takes the Chinese text data from the Sina Weibo platform during COVID-19 as the research object. By comparing the BERT-FGM-BiGRU model with the traditional model, and combining the temporal and spatial characteristics, it further studies the changing trend of user sentiment. Finally, the results show that the BERT-FGM-BiGRU model has the best classification effect and the highest accuracy compared with other models, which provides a scientific method for government departments to supervise public opinion. Based on the classification results of this model and combined with the temporal and spatial characteristics, it can be found that public sentiment is spatially closely related to the severity of the pandemic. Due to the imbalance of information sources, the public showed negative emotions of fear and worry in the early and middle stages, while in the later stage, the public sentiment gradually changed from negative to positive and hopeful with the improvement of the epidemic situation.

AIDTF: Adversarial training framework for network intrusion detection

Network Intrusion Detection Systems (IDS) have achieved high accuracy by widely applying Machine Learning (ML) models. However, most current ML-based IDSs can not cope with targeted attacks from adversaries because they are commonly trained and tested using fixed data sets. In this paper, we propose an Adversarial Intrusion Detection Training Framework (AIDTF) to improve the robustness of IDSs, which consists of an attacker model a-model, a defender model d-model, and a black-box trainer t-module. Both the a-model and d-model are multilayer perceptrons, and the t-module is the module used to train IDSs. AIDTF improves the accuracy of IDS by using an adversarial training method, which is different from traditional training methods. Taking the distribution of normal samples in the dataset as the distribution that the a-model and d-model need to learn, the goal of the a-model is to generate samples that deceive the d-model, while the goal of the d-model is to determine whether the input samples are real samples, so there is an adversarial relationship between a-model and d-model. Different types of IDSs can be trained by the t-module using the samples generated from the confrontation between the a-model and the d-model, and we call this kind of IDS the Adversarial Training Intrusion Detection System (ATIDS). The main contribution of this paper is to propose a training method that is used to obtain an IDS with high accuracy not only for known test sets but also to identify unknown disguised attack samples. We tested different types of ATIDSs using the current mainstream attack methods, which include the Fast Gradient Method, Fast Gradient Sign Method, Projected Gradient Descent, and Jacobs Saliency Map Algorithm. The experimental results prove that AIDTF outperforms other adversarial training methods with not only higher accuracy for the test set but also up to a 99% recognition rate for the attack samples.

Attention-Based Adversarial Robust Distillation in Radio Signal Classifications for Low-Power IoT Devices

Due to great success of transformers in many applications, such as natural language processing and computer vision, transformers have been successfully applied in automatic modulation classification. We have shown that transformer-based radio signal classification is vulnerable to imperceptible and carefully crafted attacks called adversarial examples. Therefore, we propose a defense system against adversarial examples in transformer-based modulation classifications. Considering the need for computationally efficient architecture particularly for Internet of Things (IoT)-based applications or operation of devices in an environment where power supply is limited, we propose a compact transformer for modulation classification. The advantages of robust training such as adversarial training in transformers may not be attainable in compact transformers. By demonstrating this, we propose a novel compact transformer that can enhance robustness in the presence of adversarial attacks. The new method is aimed at transferring the adversarial attention map from the robustly trained large transformer to a compact transformer. The proposed method outperforms the state-of-the-art techniques for the considered white-box scenarios, including the fast gradient method and projected gradient descent attacks. We have provided reasoning of the underlying working mechanisms and investigated the transferability of the adversarial examples between different architectures. The proposed method has the potential to protect the transformer from the transferability of adversarial examples.

Horizon-Independent Preconditioner Design for Linear Predictive Control

First-order optimizationsolvers, such as the fast gradient method (FGM), are increasingly being used to solve model predictive control problems in resource-constrained environments. Unfortunately, the convergence rate of these solvers is significantly affected by the conditioning of the problem data, with ill-conditioned problems requiring a large number of iterations. To reduce the number of iterations required, we present a simple method for computing a horizon-independent preconditioning matrix for the Hessian of the condensed problem. The preconditioner is based on the block Toeplitz structure of the Hessian. Horizon independence allows one to use only the predicted system and cost matrices to compute the preconditioner, instead of the full Hessian. The proposed preconditioner has equivalent performance to an optimal preconditioner in numerical examples, producing speedups between 2x and 9x for the FGM. Additionally, we derive horizon-independent spectral bounds for the Hessian in terms of the transfer function of the predicted system, and show how these can be used to compute a novel horizon-independent bound on the condition number for the preconditioned Hessian.

Restart of Accelerated First-Order Methods With Linear Convergence Under a Quadratic Functional Growth Condition

Accelerated first-order methods, also called fast gradient methods, are popular optimization methods in the field of convex optimization. However, they are prone to suffer from oscillatory behavior that slows their convergence when medium to high accuracy is desired. In order to address this, restart schemes have been proposed in the literature, which seek to improve the practical convergence by suppressing the oscillatory behavior. This article presents a restart scheme for accelerated first-order methods, for which we show linear convergence under the satisfaction of a quadratic functional growth condition, thus encompassing a broad class of non-necessarily strongly convex optimization problems. Moreover, the worst-case convergence rate is comparable to the one obtained using an (generally nonimplementable) optimal fixed-rate restart strategy. We compare the proposed algorithm with other restart schemes by applying them to a model predictive control case study.

Transformer fast gradient method with relative positional embedding: a mutual translation model between English and Chinese

Transformer fast gradient method with relative positional embedding: a mutual translation model between English and Chinese

Co-optimization strategies for connected and automated fuel cell hybrid vehicles in dynamic curving scenarios

The performance of speed planning and energy management for connected and automated fuel cell hybrid vehicles (CAFCHVs) in the curve directly affects the curve passage, operating safety and energy economy. However, the uncertainty of complex traffic conditions (such as the dynamic state of the preceding and ego vehicle, road adhesion coefficient, and curve radius) and the lateral stability of CAFCHV lead to the difficulty of online speed planning and energy management. To address this problem, a co-optimization strategy is implemented in this study. First, according to the stability condition of CAFCHV in the curve, the critical safe speed is obtained by phase plane analysis. In addition, combing the timeliness, future information of driving conditions, and the current state of preceding and ego-vehicles, the gradient-based model prediction control (GRAMPC) leveraging the fast projection gradient method is adopted to calculate the safe and optimal speed sequence. Meanwhile, the energy management strategy (EMS) based on the power ratio adaptive equivalent consumption minimization strategy (PR-AECMS) is utilized for energy distribution. A multi-objective performance function is introduced to evaluate the total cost of hydrogen consumption and battery life extension. The simulation results reveal that the proposed strategy can obtain a safe and optimal speed sequence when CAFCHV operates on the curve road. And compared with the mode of tracking the speed of the preceding vehicle, the hydrogen consumption, SOC, battery degradation, and total cost are reduced by 1.4%, 1.9%, 9.9%, and 1.8% regulated by the planning mode, respectively.