Input Examples Research Articles

Soft prompt-tuning for unsupervised domain adaptation via self-supervision

Unsupervised domain adaptation methods aim to facilitate learning tasks in unlabeled target domains using labeled information from related source domains. Recently, prompt-tuning has emerged as a powerful instrument to incorporate templates that reformulate input examples into equivalent cloze-style phrases. However, there are still two great challenges for domain adaptation: (1) Existing prompt-tuning methods only rely on the general knowledge distributed in upstream pre-trained language models to alleviate the domain discrepancy. How to incorporate specific features in the source and target domains into prompt-tuning model is still divergent and under-explored; (2) In the prompt-tuning, either the crafted template methods are time-consuming and labor-intensive, or automatic prompt generation methods cannot achieve satisfied performance. To address these issues, in this paper, we propose an innovative Soft Prompt-tuning method for Unsupervised Domain Adaptation via Self-Supervision, which combines two novel ideas: Firstly, instead of only stimulating knowledge distributed in the pre-trained model, we further employ hierarchically clustered optimization strategies in a self-supervised manner to retrieve knowledge for the verbalizer construction in prompt-tuning. Secondly, we construct prompts with the special designed verbalizer that facilitate the transfer of learning representations across domains, which can consider both the automatic template generation and cross-domain classification performance. Extensive experimental results demonstrate that our method even outperforms SOTA baselines that utilize external open knowledge with much less computational time.

A Novel Adversarial Example Detection Method Based on Frequency Domain Reconstruction for Image Sensors.

Convolutional neural networks (CNNs) have been extensively used in numerous remote sensing image detection tasks owing to their exceptional performance. Nevertheless, CNNs are often vulnerable to adversarial examples, limiting the uses in different safety-critical scenarios. Recently, how to efficiently detect adversarial examples and improve the robustness of CNNs has drawn considerable focus. The existing adversarial example detection methods require modifying CNNs, which not only affects the model performance but also greatly enhances training cost. With the purpose of solving these problems, this study proposes a detection algorithm for adversarial examples that does not need modification of the CNN models and can simultaneously retain the classification accuracy of normal examples. Specifically, we design a method to detect adversarial examples using frequency domain reconstruction. After converting the input adversarial examples into the frequency domain by Fourier transform, the adversarial disturbance from adversarial attacks can be eliminated by modifying the frequency of the example. The inverse Fourier transform is then used to maximize the recovery of the original example. Firstly, we train a CNN to reconstruct input examples. Then, we insert Fourier transform, convolution operation, and inverse Fourier transform into the features of the input examples to automatically filter out adversarial frequencies. We refer to our proposed method as FDR (frequency domain reconstruction), which removes adversarial interference by converting input samples into frequency and reconstructing them back into the spatial domain to restore the image. In addition, we also introduce gradient masking into the proposed FDR method to enhance the detection accuracy of the model for complex adversarial examples. We conduct extensive experiments on five mainstream adversarial attacks on three benchmark datasets, and the experimental results show that FDR can outperform state-of-the-art solutions in detecting adversarial examples. Additionally, FDR does not require any modifications to the detector and can be integrated with other adversarial example detection methods to be installed in sensing devices to ensure detection safety.

An Adaptive Black-Box Defense Against Trojan Attacks (TrojDef).

Trojan backdoor is a poisoning attack against neural network (NN) classifiers in which adversaries try to exploit the (highly desirable) model reuse property to implant Trojans into model parameters for backdoor breaches through a poisoned training process. To misclassify an input to a target class, the attacker activates the backdoor by augmenting the input with a predefined trigger that is only known to her/him. Most of the proposed defenses against Trojan attacks assume a white-box setup, in which the defender either has access to the inner state of NN or is able to run backpropagation through it. In this work, we propose a more practical black-box defense, dubbed TrojDef. In a black-box setup, the defender can only run forward-pass of the NN. TrojDef is motivated by the Trojan poisoned training, in which the model is trained on both benign and Trojan inputs. TrojDef tries to identify and filter out Trojan inputs (i.e., inputs augmented with the Trojan trigger) by monitoring the changes in the prediction confidence when the input is repeatedly perturbed by random noise. We derive a function based on the prediction outputs which is called the prediction confidence bound to decide whether the input example is Trojan or not. The intuition is that Trojan inputs are more stable as the misclassification only depends on the trigger, while benign inputs will suffer when augmented with noise due to the perturbation of the classification features. Through mathematical analysis, we show that if the attacker is perfect in injecting the backdoor, the Trojan infected model will be trained to learn the appropriate prediction confidence bound, which is used to distinguish Trojan and benign inputs under arbitrary perturbations. However, because the attacker might not be perfect in injecting the backdoor, we introduce a nonlinear transform to the prediction confidence bound to improve the detection accuracy in practical settings. Extensive empirical evaluations show that TrojDef significantly outperforms the-state-of-the-art defenses and is highly stable under different settings, even when the classifier architecture, the training process, or the hyperparameters change.

Neural network analysis of safe life of the oil and gas industrial structures

Purpose is to study safe life of industrial (metal) structures under long-time operation in the corrosive-active media of oil and gas wells with the help of neural network analysis. Methods. The MATLAB system (MATrix LABoratory) was selected as the tool environment for interface modelling; the system is developed by Math Works Inc. and is a high-level programming language for technical computations. Of the three existing learning paradigms, we used the “with teacher” learning process, as we believed that a neural network had correct answers (network outputs) for each input example. The coefficients were adjusted so that the network gave answers being as close as possible to the known correct answers. Findings. An artificial neural network has helped obtain a generalized diagram of the expected areas of high viscoplastic characteristics of carbon steels used to manufacture metal structures in the oil and gas industry. While applying the trained neural networks, generalized dependences of the corrosion rates of structural steels on the parameters of media with different concentrations of chlorine ions, sulphate ions, hydrogen sulphide, carbon dioxide, carbon dioxide, and oxygen ions were obtained; they were the basis to predict corrosion behaviour of steels. Originality. For the first time, the possibility of applying neural network modelling to predict local corrosion damage of structural pipe steels has been shown in terms of the “steel 20 – oxygen and chloride-containing medium” system. For the first time, the technological possibility has been demonstrated to use neural network analysis for engineering predictive assessment of corrosion activity of binary systems of simulated solutions, which are most often found under industrial conditions of the oil and gas sector of the economy. Practical implications. The proposed technology of using the neural network analysis will make it possible to expand a range of predicted values beyond experimental data, i.e. to predict the value of Vcor in very dilute or concentrated salt solutions within the acidified and neutral pH ranges. It should be noted that the error of the prediction results shown by the neural network will increase along with distancing from the scope of experimental data.

Boosting Adversarial Transferability across Model Genus by Deformation-Constrained Warping

Adversarial examples generated by a surrogate model typically exhibit limited transferability to unknown target systems. To address this problem, many transferability enhancement approaches (e.g., input transformation and model augmentation) have been proposed. However, they show poor performances in attacking systems having different model genera from the surrogate model. In this paper, we propose a novel and generic attacking strategy, called Deformation-Constrained Warping Attack (DeCoWA), that can be effectively applied to cross model genus attack. Specifically, DeCoWA firstly augments input examples via an elastic deformation, namely Deformation-Constrained Warping (DeCoW), to obtain rich local details of the augmented input. To avoid severe distortion of global semantics led by random deformation, DeCoW further constrains the strength and direction of the warping transformation by a novel adaptive control strategy. Extensive experiments demonstrate that the transferable examples crafted by our DeCoWA on CNN surrogates can significantly hinder the performance of Transformers (and vice versa) on various tasks, including image classification, video action recognition, and audio recognition. Code is made available at https://github.com/LinQinLiang/DeCoWA.

DefenseFea: An Input Transformation Feature Searching Algorithm Based Latent Space for Adversarial Defense

Abstract Deep neural networks based image classification systems could suffer from adversarial attack algorithms, which generate input examples by adding deliberately crafted yet imperceptible noise to original input images. These crafted examples can fool systems and further threaten their security. In this paper, we propose to use latent space protect image classification. Specifically, we train a feature searching network to make up the difference between adversarial examples and clean examples with label guided loss function. We name it DefenseFea(input transformation based defense with label guided loss function), experimental result shows that DefenseFea can improve the rate of adversarial examples that achieved a success rate of about 99% on a specific set of 5000 images from ILSVRC 2012. This study plays a positive role in the further investigation of the relationship between adversarial examples and clean examples.

TRAVOLTA: GPU acceleration and algorithmic improvements for constructing quantum optimal control fields in photo-excited systems

We present an open-source software package, TRAVOLTA (Terrific Refinements to Accelerate, Validate, and Optimize Large Time-dependent Algorithms), for carrying out massively parallelized quantum optimal control calculations on GPUs. The TRAVOLTA software package is a significant overhaul of our previous NIC-CAGE algorithm and also includes algorithmic improvements to the gradient ascent procedure to enable faster convergence. We examine three different variants of GPU parallelization to assess their performance in constructing optimal control fields in a variety of quantum systems. In addition, we provide several examples with extensive benchmarks of our GPU-enhanced TRAVOLTA code to show that it generates the same results as previous CPU-based algorithms but with a speedup that is more than ten times faster. Our GPU enhancements and algorithmic improvements enable large quantum optimal control calculations that can be efficiently and routinely executed on modern multi-core computational hardware. Program summaryProgram Title: TRAVOLTACPC Library link to program files:https://doi.org/10.17632/grwppm37rn.1Licensing provisions: GNU General Public License 3Programming language: C++, openBLAS, and CUDASupplementary material: Brief review of LU decomposition, raw numerical values used to generate Fig. 6 in the main text, and input examples for the TRAVOLTA software package.Nature of problem: The TRAVOLTA software package utilizes GPU accelerated routines and new algorithmic improvements to compute optimized electric fields that can drive a system from a known initial vibrational eigenstate to a specified final quantum state with a large (≈1) transition probability.Solution method: Quantum control, GPU acceleration, analytic gradients, Crank-Nicolson propagation, and gradient ascent optimization.

Detecting Adversarial Examples on Deep Neural Networks With Mutual Information Neural Estimation

Despite achieving exceptional performance, deep neural networks (DNNs) suffer from the harassment caused by adversarial examples, which are produced by corrupting clean examples with tiny perturbations. Many powerful defense methods have been presented such as training data augmentation and input reconstruction which, however, usually rely on the prior knowledge of the targeted models or attacks. A clean example and its adversarial version are very similar but have different high-level representations in a victim model. If we can obtain a space in which the representations of similar examples are also similar, then adversarial examples can be picked out by comparing the representations of input examples in this space and the high-level space of the victim model. Inspired by this, we propose a novel approach for detecting adversarial images, which can protect any pre-trained DNN classifiers and resist an endless stream of new attacks. Specifically, we first adopt a dual autoencoder to project images to a latent space. The dual autoencoder uses the self-supervised learning to ensure that small modifications to samples do not significantly alter their latent representations. Next, the mutual information neural estimation is utilized to enhance the discrimination of the latent representations. We then leverage the prior distribution matching to regularize the latent representations. To easily compare the representations of examples in the two spaces, and not rely on the prior knowledge of the targeted model, a simple fully connected neural network is used to embed the learned representations into an eigenspace, which is consistent with the output eigenspace of the targeted model. Through the distribution similarity of an input example in the two eigenspaces, we can judge whether the input example is adversarial or not. Extensive experiments on MNIST, CIFAR-10, and ImageNet show that the proposed method has superior defense performance and transferability than state-of-the-arts.

INTRODUCTION TO HEALTHY LOW-SUGAR FOODS USING ARTIFICIAL NEURAL NETWORKS WITH THE PERCEPTRON METHOD

With tecnology, the figuring system has become simpler. Tecnology are utilized to demonstrate the organic nerves of the human mind, Tecnology are prepared and shown the proper behavior like human nerve cells that can perceive basic examples. Counterfeit brain networks can perceive input examples and will give yield that matches the objective to be accomplished. In this review, the Perceptron technique was utilized to distinguish input designs as sound, low-cholesterol food varieties. This technique works by changing the contribution to the objective and changing the loads until no more mistakes are tracked down in every age. This side effect of serving a low-sugar diet shows that meat, fresh fish, and vegetables are food sources that contain high levels of sugar. Keyword: neural networks, perceptron, sugar.

An Optimal Transport Analysis on Generalization in Deep Learning.

Deep neural networks (DNNs) have achieved state-of-the-art performance in various learning tasks, such as computer vision, natural language processing, and speech recognition. However, the fundamental theory of generalization still remains obscure in deep learning-why DNN models can generalize well, despite that they are heavily overparametrized in both depth and width? Recently, some work shows that traditional theory of analyzing the generalization error of learning models fails to explain the generalization of DNNs. The failure is mainly because of one simple fact that the worse case analysis of generalization error for learning models would be too loose for models with large parameter space, such as DNNs. In this work, we propose a new analysis of generalization in DNNs from an optimal transport perspective. Unlike traditional worse-case uniform convergence analysis in learning theory, our analysis of generalization error is dependent on both the learning algorithm and the data distribution and is the average-case analysis. Thus, our theory can be more practical and accurate to describe the generalization behavior of DNNs. More specifically, in this article, we try to answer a fundamental yet unsolved question in deep learning-why deeper models can generalize well than shallow models? The main contribution of this article can be summarized in four aspects. First, under a general learning framework, we derive upper bounds on the generalization error of learning algorithms by their algorithmic transport cost: the expected Wasserstein distance between the output hypothesis and the output hypothesis conditioned on an input example. We further provide several upper bounds on the algorithmic transport cost in terms of total variation distance, relative entropy, and Vapnik-Chervonenkis (VC) dimension. Moreover, we also study different conditions for loss functions under which the generalization error of a learning algorithm can be upper bounded by different probability metrics between distributions relating to the output hypothesis and/or the input data. Finally, under our established framework, we obtain our main results, showing that the generalization error in DNNs decreases exponentially to zero as the number of layers increases.

Maximum-Entropy Multi-Agent Dynamic Games: Forward and Inverse Solutions

In this article, we study the problem of multiple stochastic agents interacting in a dynamic game scenario with continuous state and action spaces. We define a new notion of stochastic Nash equilibrium for boundedly rational agents, which we call the entropic cost equilibrium (ECE). We show that ECE is a natural extension to multiple agents of maximum entropy optimality for a single agent. We solve both the “forward” and “inverse” problems for the multi-agent ECE game. For the forward problem, we provide a Riccati algorithm to compute closed-form ECE feedback policies for the agents, which are exact in the linear-quadratic-gaussian case. We give an iterative variant to find locally ECE feedback policies for the nonlinear case. For the inverse problem, we present an algorithm to infer the cost functions of the multiple interacting agents given noisy, boundedly rational input and state trajectory examples from agents acting in an ECE. The effectiveness of our algorithms is demonstrated in a simulated multi-agent collision avoidance scenario, and with data from the INTERACTION traffic dataset. In both cases, we show that, by taking into account the agents' game theoretic interactions using our algorithm, a more accurate model of agents' costs can be learned, compared with standard inverse optimal control methods.

Erasure of Unaligned Attributes from Neural Representations

AbstractWe present the Assignment-Maximization Spectral Attribute removaL (AMSAL) algorithm, which erases information from neural representations when the information to be erased is implicit rather than directly being aligned to each input example. Our algorithm works by alternating between two steps. In one, it finds an assignment of the input representations to the information to be erased, and in the other, it creates projections of both the input representations and the information to be erased into a joint latent space. We test our algorithm on an extensive array of datasets, including a Twitter dataset with multiple guarded attributes, the BiasBios dataset, and the BiasBench benchmark. The latter benchmark includes four datasets with various types of protected attributes. Our results demonstrate that bias can often be removed in our setup. We also discuss the limitations of our approach when there is a strong entanglement between the main task and the information to be erased.1

BHI: Embedded invisible watermark as adversarial example based on Basin-Hopping improvement

Traditional research on generating adversarial examples has mainly focused on artificially adding imperceptible perturbations to input examples. Typically, the perturbation information in adversarial examples has no practical meaning. This study proposes a novel scheme for generating adversarial examples by embedding invisible watermarks based on basin-hopping improvement (BHI). To produce adversarial examples, the proposed BHI scheme is implemented by embedding invisible watermarks into original images based on a data-hiding technique. Specifically, the BHI scheme determines the corresponding coordinates of host images and watermark sizes. By considering the specific coordinates and size, the BHI scheme invisibly embeds the watermarks into the host images to generate adversarial examples. Experimental results show that the attack success rate of the BHI scheme reaches 93.5%. The proposed BHI scheme not only completes the function of an adversarial attack but also protects the copyright of adversarial examples. Moreover, the adversarial examples exhibit outstanding visual and robust performance, providing additional visual protection to avoid the risk of attack.

CORPORATE STRATEGY FOR MANAGING EVOLUTIONARY PROCESSES OF THE ENTERPRISE DEVELOPMENT

The method for analyzing phase portraits of bifurcation diagrams of enterprise life cycle scenario models using IT pattern recognition has been proposed, which allowed to numerically determine the geometric parameters of the spatial forms that form the surfaces of phase portraits, and their size, corresponding to the loci of stability and interaction of four populations in a network of trophic relations. To analyze the results of the recognition of phase portraits of ELC bifurcation diagrams, singular points of perturbation, attenuation, cycles, equilibrium, their stability, the input examples of the state of bifurcation have been classified into five separate clusters in accordance with the ELC development scenarios. An IT support algorithm for the corporate strategy for managing the evolutionary processes of enterprise development has been developed. It covers the following stages: analysis and evaluation of the internal and external environment of the enterprise population and trophic relations; identification of possible motives for changing trophic relations and causes of synergy; studies of the nonlinear dynamics of ELC model scenarios; analysis of many strategic alternatives; selection of a strategy and preparation of a strategic plan; monitoring the results of achieving a synergistic effect, which will allow choosing strategies depending on the ELC stages and ensuring the competitiveness of the enterprise.

Adversarial Machine Learning for Network Intrusion Detection Systems: A Comprehensive Survey

Network-based Intrusion Detection System (NIDS) forms the frontline defence against network attacks that compromise the security of the data, systems, and networks. In recent years, Deep Neural Networks (DNNs) have been increasingly used in NIDS to detect malicious traffic due to their high detection accuracy. However, DNNs are vulnerable to adversarial attacks that modify an input example with imperceivable perturbation, which causes a misclassification by the DNN. In security-sensitive domains, such as NIDS, adversarial attacks pose a severe threat to network security. However, existing studies in adversarial learning against NIDS directly implement adversarial attacks designed for Computer Vision (CV) tasks, ignoring the fundamental differences in the detection pipeline and feature spaces between CV and NIDS. It remains a major research challenge to launch and detect adversarial attacks against NIDS. This article surveys the recent literature on NIDS, adversarial attacks, and network defences since 2015 to examine the differences in adversarial learning against deep neural networks in CV and NIDS. It provides the reader with a thorough understanding of DL-based NIDS, adversarial attacks and defences, and research trends in this field. We first present a taxonomy of DL-based NIDS and discuss the impact of taxonomy on adversarial learning. Next, we review existing white-box and black-box adversarial attacks on DNNs and their applicability in the NIDS domain. Finally, we review existing defence mechanisms against adversarial examples and their characteristics.

Enhancing the Adversarial Transferability with Channel Decomposition

The current adversarial attacks against deep learning models have achieved incredible success in the white-box scenario. However, they often exhibit weak transferability in the black-box scenario, especially when attacking those with defense mechanisms. In this work, we propose a new transfer-based black-box attack called the channel decomposition attack method (CDAM). It can attack multiple black-box models by enhancing the transferability of the adversarial examples. On the one hand, it tunes the gradient and stabilizes the update direction by decomposing the channels of the input example and calculating the aggregate gradient. On the other hand, it helps to escape from local optima by initializing the data point with random noise. Besides, it could combine with other transfer-based attacks flexibly. Extensive experiments on the standard ImageNet dataset show that our method could significantly improve the transferability of adversarial attacks. Compared with the state-of-the-art method, our approach improves the average success rate from 88.2% to 96.6% when attacking three adversarially trained black-box models, demonstrating the remaining shortcomings of existing deep learning models.

Generating Deep Learning Model-Specific Explanations at the End User’s Side

End users who cannot afford to collect and label big data to train accurate deep learning (DL) models resort to Machine Learning as a Service (MLaaS) providers, who provide paid access to accurate DL models. However, the lack of transparency in how the providers’ models make predictions causes a problem of trust. A way to increase trust (and also to align with ethical regulations) is for predictions to be accompanied by explanations locally and independently generated by the end users (rather than by explanations offered by the model providers). Explanation methods using internal components of DL models (a.k.a. model-specific explanations) are more accurate and effective than those relying solely on the inputs and outputs (a.k.a. model-agnostic explanations). However, end users lack white-box access to the internal components of the providers’ models. To tackle this issue, we propose a novel approach allowing an end user to locally generate model-specific explanations for a DL classification model accessed via a provider’s API. First, we approximate the provider’s model with a local surrogate model. We then use the surrogate model’s components to locally generate model-specific explanations that approximate the explanations obtainable with white-box access to the provider’s DL model. Specifically, we leverage the surrogate model’s gradients to generate adversarial examples that counterfactually explain why an input example is classified into a specific class. Our approach only requires the end user to have unlabeled data of size [Formula: see text] of the provider’s training data and with a similar distribution; given the small size and unlabeled nature of these data, they can be assumed to be already available to the end user or even to be supplied by the provider to build trust in his model. We demonstrate the accuracy and effectiveness of our approach through extensive experiments on two ML tasks: image classification and tabular data classification. The locally generated explanations are consistent with those obtainable with white-box access to the provider’s model, thus giving end users an independent and reliable way to determine if the provider’s model is trustworthy.

T-BFA: Targeted Bit-Flip Adversarial Weight Attack.

Traditional Deep Neural Network (DNN) security is mostly related to the well-known adversarial input example attack.Recently, another dimension of adversarial attack, namely, attack on DNN weight parameters, has been shown to be very powerful. Asa representative one, the Bit-Flip based adversarial weight Attack (BFA) injects an extremely small amount of faults into weight parameters to hijack the executing DNN function. Prior works of BFA focus on un-targeted attacks that can hack all inputs into a random output class by flipping a very small number of weight bits stored in computer memory. This paper proposes the first work oftargetedBFA based (T-BFA) adversarial weight attack on DNNs, which can intentionally mislead selected inputs to a target output class. The objective is achieved by identifying the weight bits that are highly associated with classification of a targeted output through a class-dependent weight bit searching algorithm. Our proposed T-BFA performance is successfully demonstrated on multiple DNN architectures for image classification tasks. For example, by merely flipping 27 out of 88 million weight bits of ResNet-18, our T-BFA can misclassify all the images from Hen class into Goose class (i.e., 100% attack success rate) in ImageNet dataset, while maintaining 59.35% validation accuracy.

Continual semi-supervised learning through contrastive interpolation consistency

Continual Learning (CL) investigates how to train Deep Networks on a stream of tasks without incurring forgetting. CL settings proposed in literature assume that every incoming example is paired with ground-truth annotations. However, this clashes with many real-world applications: gathering labeled data, which is in itself tedious and expensive, becomes infeasible when data flow as a stream. This work explores Continual Semi-Supervised Learning (CSSL): here, only a small fraction of labeled input examples are shown to the learner. We assess how current CL methods (e.g.: EWC, LwF, iCaRL, ER, GDumb, DER) perform in this novel and challenging scenario, where overfitting entangles forgetting. Subsequently, we design a novel CSSL method that exploits metric learning and consistency regularization to leverage unlabeled examples while learning. We show that our proposal exhibits higher resilience to diminishing supervision and, even more surprisingly, relying only on 25% supervision suffices to outperform SOTA methods trained under full supervision.

Continuous and Distribution-Free Probabilistic Wind Power Forecasting: A Conditional Normalizing Flow Approach

We present a data-driven approach for probabilistic wind power forecasting based on conditional normalizing flow (CNF). In contrast with the existing, this approach is distribution-free (as for non-parametric and quantile-based approaches) and can directly yield continuous probability densities, hence avoiding quantile crossing. It relies on a base distribution and a set of bijective mappings. Both the shape parameters of the base distribution and the bijective mappings are approximated with neural networks. Spline-based conditional normalizing flow is considered owing to its non-affine characteristics. Over the training phase, the model sequentially maps input examples onto samples of base distribution, given the conditional contexts, where parameters are estimated through maximum likelihood. To issue probabilistic forecasts, one eventually maps samples of the base distribution into samples of a desired distribution. Case studies based on open datasets validate the effectiveness of the proposed model, and allows us to discuss its advantages and caveats with respect to the state of the art.