Malicious Inputs Research Articles

Adversarial Deep Learning: A Survey on Adversarial Attacks and Defense Mechanisms on Image Classification

The popularity of adapting deep neural networks (DNNs) in solving hard problems has increased substantially. Specifically, in the field of computer vision, DNNs are becoming a core element in developing many image and video classification and recognition applications. However, DNNs are vulnerable to adversarial attacks, in which, given a well-trained image classification model, a malicious input can be crafted by adding mere perturbations to misclassify the image. This phenomena raise many security concerns in utilizing DNNs in critical life applications which attracts the attention of academic and industry researchers. As a result, multiple studies have proposed discussing novel attacks that can compromise the integrity of state-of-the-art image classification neural networks. The raise of these attacks urges the research community to explore countermeasure methods to mitigate these attacks and increase the reliability of adapting DDNs in different major applications. Hence, various defense strategies have been proposed to protect DNNs against adversarial attacks. In this paper, we thoroughly review the most recent and state-of-the-art adversarial attack methods by providing an in-depth analysis and explanation of the working process of these attacks. In our review, we focus on explaining the mathematical concepts and terminologies of the adversarial attacks, which provide a comprehensive and solid survey to the research community. Additionally, we provide a comprehensive review of the most recent defense mechanisms and discuss their effectiveness in defending DNNs against adversarial attacks. Finally, we highlight the current challenges and open issues in this field as well as future research directions.

FooBaR: Fault Fooling Backdoor Attack on Neural Network Training

Neural network implementations are known to be vulnerable to physical attack vectors such as fault injection attacks. As of now, these attacks were only utilized during the inference phase. In this work, we explore a novel attack paradigm by injecting faults during the training phase in a way that the resulting network can be attacked during deployment without the necessity of further faulting. We discuss attacks against ReLU activation functions that make it possible to generate a family of malicious inputs, which are called fooling inputs, to be used at inference time to induce controlled misclassifications. Such malicious inputs are obtained by mathematically solving a system of linear equations that would cause a particular behaviour on the attacked activation functions, similar to the one induced in training through faulting. We call such attacks fooling backdoors as the faults at training phase inject backdoors into the network that allow an attacker to produce fooling inputs. We evaluate our approach against multi-layer perceptron networks and convolutional networks on a popular image classification task obtaining high attack success rates (60% - 100%) and high classification confidence when as little as 25 neurons are attacked while preserving high accuracy on the original classification task.

White-box attacks on hate-speech BERT classifiers in German with explicit and implicit character-level defense

Attention-based transformer models have achieved state-of-the-art results in natural language processing (NLP). However, recent work shows that the underlying attention mechanism can be exploited by adversaries to craft malicious inputs designed to induce spurious outputs, thereby harming model performance and trustworthiness. Unlike in the vision domain, the literature examining neural networks under adversarial conditions in the NLP domain is limited and most of it focuses mainly on the English language. In this article, we first analyze the adversarial robustness of Bidirectional Encoder Representations from Transformers (BERT) models for German data sets. Second, we introduce two novel NLP attacks: a character-level and a word-level attacks, both of which utilize attention scores to calculate where to inject character-level and word-level noise, respectively. Finally, we present two defense strategies against the attacks above. The first implicit character-level defense is a variant of adversarial training, which trains a new classifier capable of abstaining/rejecting certain (ideally adversarial) inputs. The other explicit character-level defense learns a latent representation of the complete training data vocabulary and then maps all tokens of an input example to the same latent space, enabling the replacement of all out-of-vocabulary tokens with the most similar in-vocabulary tokens based on the cosine similarity metric.

RESFIT: A Reputation and Security Monitoring Platform for IoT Applications

The fast-paced adoption of smart devices has changed the Internet of Things (IoT) landscape, leading to the growth of smart environments and inclusion in many aspects of our society. In IoT applications, data collected from sensors and mobile devices are aggregated, processed, and analyzed to extract useful information and develop intelligent services. If the collected data is not trustworthy due to the damage or malicious input of some sensors, the quality of the service will be impacted. For reliable data collection and mining, it is mandatory to define robust security and trust models, suitable for the IoT application context. In this paper, we propose RESFIT, a platform that implements a reputation-based trust mechanism and an advanced application level firewall to cope with the above mentioned issues. Having a gateway-centric architecture, the proposed platform ensures minimal resource consumption at the node layer, and an integrated overview and control of the system state, through the cloud component and smartphone management application.

Determining Sequence of Image Processing Technique (IPT) to Detect Adversarial Attacks

Developing secure machine learning models from adversarial examples is challenging as various methods are continually being developed to generate adversarial attacks. In this work, we propose an evolutionary approach to automatically determine Image Processing Techniques Sequence (IPTS) for detecting malicious inputs. Accordingly, we first used a diverse set of attack methods including adaptive attack methods (on our defense) to generate adversarial samples from the clean dataset. A detection framework based on a genetic algorithm (GA) is developed to find the optimal IPTS, where the optimality is estimated by different fitness measures such as Euclidean distance, entropy loss, average histogram, local binary pattern and loss functions. The "image difference" between the original and processed images is used to extract the features, which are then fed to a classification scheme in order to determine whether the input sample is adversarial or clean. This paper described our methodology and performed experiments using multiple data-sets tested with several adversarial attacks. For each attack-type and dataset, it generates unique IPTS. A set of IPTS selected dynamically in testing time which works as a filter for the adversarial attack. Our empirical experiments exhibited promising results indicating the approach can efficiently be used as processing for any AI model.

Error-Correcting Output Codes with Ensemble Diversity for Robust Learning in Neural Networks

Though deep learning has been applied successfully in many scenarios, malicious inputs with human-imperceptible perturbations can make it vulnerable in real applications. This paper proposes an error-correcting neural network (ECNN) that combines a set of binary classifiers to combat adversarial examples in the multi-class classification problem. To build an ECNN, we propose to design a code matrix so that the minimum Hamming distance between any two rows (i.e., two codewords) and the minimum shared information distance between any two columns (i.e., two partitions of class labels) are simultaneously maximized. Maximizing row distances can increase the system fault tolerance while maximizing column distances helps increase the diversity between binary classifiers. We propose an end-to-end training method for our ECNN, which allows further improvement of the diversity between binary classifiers. The end-to-end training renders our proposed ECNN different from the traditional error-correcting output code (ECOC) based methods that train binary classifiers independently. ECNN is complementary to other existing defense approaches such as adversarial training and can be applied in conjunction with them. We empirically demonstrate that our proposed ECNN is effective against the state-of-the-art white-box and black-box attacks on several datasets while maintaining good classification accuracy on normal examples.

A Black-Box Attack Method against Machine-Learning-Based Anomaly Network Flow Detection Models

In recent years, machine learning has made tremendous progress in the fields of computer vision, natural language processing, and cybersecurity; however, we cannot ignore that machine learning models are vulnerable to adversarial examples, with some minor malicious input modifications, while appearing unmodified to human observers, the outputs of machine learning-based model can be misled easily. Likewise, attackers can bypass machine-learning-based security defenses model to attack systems in real time by generating adversarial examples. In this paper, we propose a black-box attack method against machine-learning-based anomaly network flow detection algorithms. Our attack strategy consists in training another model to substitute for the target machine learning model. Based on the overall understanding of the substitute model and the migration of the adversarial examples, we use the substitute model to craft adversarial examples. The experiment has shown that our method can attack the target model effectively. We attack several kinds of network flow detection models, which are based on different kinds of machine learning methods, and we find that the adversarial examples crafted by our method can bypass the detection of the target model with high probability.

White-Box Attacks on Hate-speech BERT Classifiers in German with Explicit and Implicit Character Level Defense

Attention based Transformer models have achieved state-of-the-art results in natural language processing (NLP). However, recent work shows that the underlying attention mechanism can be exploited by adversaries to craft malicious inputs designed to induce spurious outputs, thereby harming model performance and trustworthiness. Unlike in the vision domain, the literature examining neural networks under adversarial conditions in the NLP domain is limited and most of it focuses mainly on the English language. In this paper, we first analyze the adversarial robustness of Bidirectional Encoder Representations from Transformers (BERT) models for German datasets. Second, we introduce two novel NLP attacks. Namely, a character-level and a word-level attacks, both of which utilize attention scores to calculate where to inject character-level and word-level noise, respectively. Finally, we present two defense strategies against the attacks above. The first implicit character-level defense is a variant of adversarial training, which trains a new classifier capable of abstaining/rejecting certain (ideally adversarial) inputs. The other explicit character-level defense learns a latent representation of the complete training data vocabulary and then maps all tokens of an input example to the same latent space, enabling the replacement of all out of vocabulary tokens with the most similar in-vocabulary tokens based on the cosine similarity metric.

White-Box Attacks on Hate-speech BERT Classifiers in German with Explicit and Implicit Character Level Defense

Attention based Transformer models have achieved state-of-the-art results in natural language processing (NLP). However, recent work shows that the underlying attention mechanism can be exploited by adversaries to craft malicious inputs designed to induce spurious outputs, thereby harming model performance and trustworthi- ness. Unlike in the vision domain, the literature examining neural networks under adversarial conditions in the NLP domain is limited and most of it focuses mainly on the English language. In this paper, we first analyze the adver- sarial robustness of Bidirectional Encoder Representations from Transformers (BERT) models for German datasets. Second, we introduce two novel NLP attacks. Namely, a character-level and a word-level attacks, both of which uti- lize attention scores to calculate where to inject character-level and word-level noise, respectively. Finally, we present two defense strategies against the attacks above. The first implicit character-level defense is a variant of adversar- ial training, which trains a new classifier capable of abstaining/rejecting certain (ideally adversarial) inputs. The other explicit character-level defense learns a latent representation of the complete training data vocabulary and then maps all tokens of an input example to the same latent space, enabling the replacement of all out of vocabulary tokens with the most similar in-vocabulary tokens based on the cosine similarity metric.

Cyclic Defense GAN Against Speech Adversarial Attacks

This paper proposes a new defense approach for counteracting state-of-the-art white and black-box adversarial attack algorithms. Our approach fits into the implicit reactive defense algorithm category since it does not directly manipulate the potentially malicious input signals. Instead, it reconstructs a similar signal with a synthesized spectrogram using a cyclic generative adversarial network. This cyclic framework helps to yield a stable generative model. Finally, we feed the reconstructed signal into the speech-to-text model for transcription. The conducted experiments on targeted and non-targeted adversarial attacks developed for attacking DeepSpeech, Kaldi, and Lingvo models demonstrate the proposed defense's effectiveness in adverse scenarios.

Defense against neural trojan attacks: A survey

Deep learning techniques have become significantly prevalent in many real-world problems including a variety of detection, recognition, and classification tasks. To obtain high-performance neural networks, an enormous amount of training datasets, memory, and time-consuming computations are required which has increased the demands for outsource training among users. As a result, the machine-learning-as-a-service(MLaaS) providers or a third party can gain an opportunity to put the model’s security at risk by training the model with malicious inputs. The malicious functionality inserted into the neural network by the adversary will be activated in the presence of specific inputs. These kinds of attacks to neural networks, called trojan or backdoor attacks, are very stealthy and hard to detect because they do not affect the network performance on clean datasets. In this paper, we refer to two important threat models and we focus on the detection and mitigation techniques against these types of attacks on neural networks which has been proposed recently. We summarize, discuss, and compare the defense methods and their corresponding results.

A game-theoretic method for resilient control design in industrial multi-agent CPSs with Markovian and coupled dynamics

Multi-agent cyber-physical systems (CPSs) are ubiquitous in various modern critical infrastructures, especially in industrial automation control involved in the petrochemical, metallurgy, textile, and pharmaceutical, which are vulnerable for their hierarchically and horizontally interconnected architecture. In this paper, we investigate a resilient control design problem for industrial multi-agent CPSs of centralised information structure, with Markovian and physically coupled dynamics from a game-theoretic perspective. First, we investigate a worst-case design problem and solve for worst-case disturbance strategy and optimal disturbance attenuation level parameter through an equivalent two-person zero-sum differential game in both finite- and infinite-horizon. Second, we study a corresponding N-person worst-case nonzero-sum differential game and derive state-feedback Nash equilibrium solution of robustness property as resilient control strategy for each agent. The experimental case study is given to demonstrate that using resilient control strategy, each agent could maintain its physical plant operationally normal under cyber attack, malicious disturbance input and physical state interdependency.

Adversarial Attack on Machine Learning Models

Machine Learning (ML) models are applied in a variety of tasks such as network intrusion detection or malware classification. Yet, these models are vulnerable to a class of malicious inputs known as adversarial examples. These are slightly perturbed inputs that are classified incorrectly by the ML model. The mitigation of these adversarial inputs remains an open problem. As a step towards understanding adversarial examples, we show that they are not drawn from the same distribution than the original data, and can thus be detected using statistical tests. Using this knowledge, we introduce a complimentary approach to identify specific inputs that are adversarial. Specifically, we augment our ML model with an additional output, in which the model is trained to classify all adversarial inputs.

Automatic Generation of Tests to Exploit XML Injection Vulnerabilities in Web Applications

Modern enterprise systems can be composed of many web services (e.g., SOAP and RESTful). Users of such systems might not have direct access to those services, and rather interact with them through a single-entry point which provides a GUI (e.g., a web page or a mobile app). Although the interactions with such entry point might be secure, a hacker could trick such systems to send malicious inputs to those internal web services. A typical example is XML injection targeting SOAP communications. Previous work has shown that it is possible to automatically generate such kind of attacks using search-based techniques. In this paper, we improve upon previous results by providing more efficient techniques to generate such attacks. In particular, we investigate four different algorithms and two different fitness functions. A large empirical study, involving also two industrial systems, shows that our technique is effective at automatically generating XML injection attacks.

Dynamically Discovering Likely Memory Layout to Perform Accurate Fuzzing

Malicious Input through Buffer Overflow (MiBO) vulnerabilities play important roles in cyber security. To identify MiBO vulnerabilities, white-box testing approaches analyze instructions in all possible execution paths. Black-box testing approaches try to trigger MiBO vulnerabilities using different inputs. However, only limited coverage can be achieved: the identified MiBO vulnerabilities, when being “hit” by a test input, must cause exceptions (e.g., crashes). Type information could help to catch the non-crash MiBO vulnerabilities, but such information is not contained in binary code. In this paper, we present a white-box fuzzing method to detect non-crash MiBO vulnerabilities. Without source code, we dynamically discover likely memory layouts to help the fuzzing process. This is very challenging since memory addresses and layouts keep changing with the running of software. In different executions with different inputs, the layouts may also change. To address these challenges, we selectively analyze memory operations to identify memory layouts. If a buffer border identified from the memory layout is exceeded, an error will be reported. The fuzzing results will be compared with the layout for future input generation, which greatly increases the opportunity to expose MiBO vulnerabilities. We implemented a prototype called ArtFuzz and performed several evaluations. ArtFuzz discovered 23 real MiBO vulnerabilities (including 8 zero-day MiBO vulnerabilities) in nine applications.

Machine Learning in Adversarial Settings

Recent advances in machine learning have led to innovative applications and services that use computational structures to reason about complex phenomenon. Over the past several years, the security and machine-learning communities have developed novel techniques for constructing adversarial samples--malicious inputs crafted to mislead (and therefore corrupt the integrity of) systems built on computationally learned models. The authors consider the underlying causes of adversarial samples and the future countermeasures that might mitigate them.

SQLPIL: SQL injection prevention by input labeling

AbstractSQL injection attacks (SQLIAs) aim at exploiting vulnerabilities in web applications in order to execute malicious SQL commands. It is established that prepared statements are resilient to SQLIAs, and thus, developers are advised to use them when constructing SQL queries as opposed to applying string concatenation operations. Unfortunately, this recommended programming practice is not as pervasive as it should be. This paper addresses this shortcoming by presenting SQL injection Prevention by Input Labeling (SQLPIL), an effective, light, and fully automated tool that leverages prepared statements to prevent SQLIAs at runtime. Given a Java program in which SQL queries are built as strings, SQLPIL dynamically transforms the strings into secure prepared statements right before their execution, thus guaranteeing that malicious input will always be treated as data and never as SQL commands. We empirically evaluated our Java implementation of SQLPIL using a benchmark that includes five JSP commercial applications, a number of legitimate queries, and a number of attacks of representative types. The results were promising as all attacks were prevented, and all legitimate runs executed successfully; in other words, the technique exhibited no false alarms when applied on typical applications. Also, the runtime cost was acceptable, assuming typical settings. Copyright © 2015 John Wiley & Sons, Ltd.

Detecting SQL injection attacks using query result size

Web applications are becoming an essential part of our everyday lives, with many of our activities dependent on the functionality and security of these applications. Web applications are ubiquitous, perform mission critical tasks, and handle sensitive user data. As the scale of these applications grows, injection vulnerabilities, such as SQL injections, become major security challenges. Most of these vulnerabilities stem from a lack of input validation; that is, web applications use malicious input as part of a sensitive operation without properly checking or sanitizing the input values. SQL injection attacks target databases that are accessible through a web front-end; moreover, they take advantage of flaws in the input validation logic of web components. In this paper, we exhibit a novel scheme that automatically transforms web applications, rendering them safe against SQL injection attacks. Our technique dynamically analyzes the developer-intended query result size for any input, and detects attacks by comparing this against the result of the actual query. We implement this technique in a tool for protecting Java-based web applications. An experimental evaluation demonstrates that our technique is effective against SQL injection vulnerabilities.

Security Testing of Web Applications: Issues and Challenges

Due to the increasing complexity of web systems, security testing has become indispensable and critical activity of web application development life cycle. Security testing aims to maintain the confidentiality of the data, to check against any information leakage and to maintain the functionality as intended. It checks whether the security requirements are fulfilled by the web applications when they are subjected to malicious input data. Due to the rising explosion in the security vulnerabilities, there occurs a need to understand its unique challenges and issues which will eventually serve as a useful input for the security testing tool developers and test managers for their relative projects.

Runtime Monitoring Framework for SQL Injection Attacks

The increasing use of web applications to provide reliable online services, such as banking, shopping, etc., and to store sensitive user data has made them vulnerable to attacks that target them. In particular, SQL injection, which allows attackers to gain unauthorized access to the database by injecting specially crafted input strings, is one of the most serious threats to web applications. Although researchers and practitioners have proposed various methods to address the SQL injection problem, organizations continue to be its victim, as attackers are successfully able to circumvent the employed techniques. In this paper, we present and evaluate Runtime Monitoring Framework to detect and prevent SQL Injection Attacks on web applications. At its core, the framework leverages the knowledge gained from pre-deployment testing of web applications to identify valid/legal execution paths. Monitors are then developed and instrumented to observe the application's behavior and check it for compliance with the valid/legal execution paths obtained; any deviation in the application's behavior is identified as a possible SQL Injection Attack. We conducted an extensive evaluation of the framework by targeting subject applications with a large number of both legitimate and malicious inputs, and assessed its ability to detect and prevent SQL Injection Attacks. The framework successfully allowed all the legitimate inputs to access the database without generating any false positives, and was able to effectively detect and prevent attacks without generating any false negatives. Moreover, the framework imposed a low runtime overhead on the subject applications compared to other techniques.