Small Input Perturbations Research Articles

Rethinking Robustness of Model Attributions

For machine learning models to be reliable and trustworthy, their decisions must be interpretable. As these models find increasing use in safety-critical applications, it is important that not just the model predictions but also their explanations (as feature attributions) be robust to small human-imperceptible input perturbations. Recent works have shown that many attribution methods are fragile and have proposed improvements in either these methods or the model training. We observe two main causes for fragile attributions: first, the existing metrics of robustness (e.g., top-k intersection) overpenalize even reasonable local shifts in attribution, thereby making random perturbations to appear as a strong attack, and second, the attribution can be concentrated in a small region even when there are multiple important parts in an image. To rectify this, we propose simple ways to strengthen existing metrics and attribution methods that incorporate locality of pixels in robustness metrics and diversity of pixel locations in attributions. Towards the role of model training in attributional robustness, we empirically observe that adversarially trained models have more robust attributions on smaller datasets, however, this advantage disappears in larger datasets. Code is made available at https://github.com/ksandeshk/LENS.

Evaluation of Malware Classification Models for Heterogeneous Data.

Machine learning (ML) has found widespread application in various domains. Additionally, ML-based techniques have been employed to address security issues in technology, with numerous studies showcasing their potential and effectiveness in tackling security problems. Over the years, ML methods for identifying malicious software have been developed across various security domains. However, recent research has highlighted the susceptibility of ML models to small input perturbations, known as adversarial examples, which can significantly alter model predictions. While prior studies on adversarial examples primarily focused on ML models for image processing, they have progressively extended to other applications, including security. Interestingly, adversarial attacks have proven to be particularly effective in the realm of malware classification. This study aims to explore the transparency of malware classification and develop an explanation method for malware classifiers. The challenge at hand is more complex than those associated with explainable AI for homogeneous data due to the intricate data structure of malware compared to traditional image datasets. The research revealed that existing explanations fall short in interpreting heterogeneous data. Our employed methods demonstrated that current malware detectors, despite high classification accuracy, may provide a misleading sense of security and measuring classification accuracy is insufficient for validating detectors.

Robust machine learning modeling for predictive control using Lipschitz-Constrained Neural Networks

Neural networks (NNs) have emerged as a state-of-the-art method for modeling nonlinear systems in model predictive control (MPC). However, the robustness of NNs, in terms of sensitivity to small input perturbations, remains a critical challenge for practical applications. To address this, we develop Lipschitz-Constrained Neural Networks (LCNNs) for modeling nonlinear systems and derive rigorous theoretical results to demonstrate their effectiveness in approximating Lipschitz functions, reducing input sensitivity, and preventing over-fitting. Specifically, we first prove a universal approximation theorem to show that LCNNs using SpectralDense layers can approximate any 1-Lipschitz target function. Then, we prove a probabilistic generalization error bound for LCNNs using SpectralDense layers by using their empirical Rademacher complexity. Finally, the LCNNs are incorporated into the MPC scheme, and a chemical process example is utilized to show that LCNN-based MPC outperforms MPC using conventional feedforward NNs in the presence of training data noise.

Towards an adversarially robust convolutional neural network for automated diagnosis of malaria infection from microscopy images

Malaria is a life-threatening mosquito-borne disease of global importance, and it is most prevalent in the low-income countries of the developing world. The diagnosis of malaria infection is conventionally performed by laboratorians through visual examination of blood smear images under a microscope. However, the results of such diagnostic testing depend on the laboratory technicians' experience with staining and image interpretation, which can be severely lacking in low-resource settings. Convolutional neural networks (CNNs) have great potential for automated malaria diagnosis from images without the need of human expert interaction. However, studies have shown that current CNNs are not robust to adversarial noises which are small input perturbations that cause CNNs to incorrectly classify medical images. In this study, we developed a CNN for malaria parasite identification and evaluated it on a dataset of 27,558 cell images through patient-level cross validation. To improve robustness against adversarial noises for this application, the structure of the CNN was specially designed to limit the receptive field of the output, and adversarial training was applied with gradually increasing level of noises. Compared to three baseline networks (ResNet-18, MobileNet, and MnasNet) and two variants of the CNN, our custom-designed CNN achieved a substantially lower computation cost and better adversarial robustness against PGD-generated adversarial noises on the malaria image dataset. Our study holds the promise of developing a robust and affordable solution for automated diagnosis of malarial infection.

Robust Classification Using Contractive Hamiltonian Neural ODEs

Deep neural networks can be fragile and sensitive to small input perturbations that might cause a significant change in the output. In this letter, we employ contraction theory to improve the robustness of neural ODEs (NODEs). A dynamical system is contractive if all solutions with different initial conditions converge to each other exponentially fast. As a consequence, perturbations in initial conditions become less and less relevant over time. Since in NODEs the input data corresponds to the initial condition of dynamical systems, we show contractivity can mitigate the effect of input perturbations. More precisely, inspired by NODEs with Hamiltonian dynamics, we propose a class of contractive Hamiltonian NODEs (CH-NODEs). By properly tuning a scalar parameter, CH-NODEs ensure contractivity by design and can be trained using standard backpropagation. Moreover, CH-NODEs enjoy built-in guarantees of non-exploding gradients, which ensure a well-posed training process. Finally, we demonstrate the robustness of CH-NODEs on the MNIST image classification problem with noisy test data.

Combating Adversaries with Anti-adversaries

Deep neural networks are vulnerable to small input perturbations known as adversarial attacks. Inspired by the fact that these adversaries are constructed by iteratively minimizing the confidence of a network for the true class label, we propose the anti-adversary layer, aimed at countering this effect. In particular, our layer generates an input perturbation in the opposite direction of the adversarial one and feeds the classifier a perturbed version of the input. Our approach is training-free and theoretically supported. We verify the effectiveness of our approach by combining our layer with both nominally and robustly trained models and conduct large-scale experiments from black-box to adaptive attacks on CIFAR10, CIFAR100, and ImageNet. Our layer significantly enhances model robustness while coming at no cost on clean accuracy.

Interactive Analysis of CNN Robustness

AbstractWhile convolutional neural networks (CNNs) have found wide adoption as state‐of‐the‐art models for image‐related tasks, their predictions are often highly sensitive to small input perturbations, which the human vision is robust against. This paper presents Perturber, a web‐based application that allows users to instantaneously explore how CNN activations and predictions evolve when a 3D input scene is interactively perturbed. Perturber offers a large variety of scene modifications, such as camera controls, lighting and shading effects, background modifications, object morphing, as well as adversarial attacks, to facilitate the discovery of potential vulnerabilities. Fine‐tuned model versions can be directly compared for qualitative evaluation of their robustness. Case studies with machine learning experts have shown that Perturber helps users to quickly generate hypotheses about model vulnerabilities and to qualitatively compare model behavior. Using quantitative analyses, we could replicate users’ insights with other CNN architectures and input images, yielding new insights about the vulnerability of adversarially trained models.

Two Sides of the Same Coin: Boons and Banes of Machine Learning in Hardware Security

The last decade has witnessed remarkable research advances at the intersection of machine learning (ML) and hardware security. The confluence of the two technologies has created many interesting and unique opportunities, but also left some issues in their wake. ML schemes have been extensively used to enhance the security and trust of embedded systems like hardware Trojans and malware detection. On the other hand, ML-based approaches have also been adopted by adversaries to assist side-channel attacks, reverse engineer integrated circuits and break hardware security primitives like Physically Unclonable Functions (PUFs). Deep learning is a subfield of ML. It can continuously learn from a large amount of labeled data with a layered structure. Despite the impressive outcomes demonstrated by deep learning in many application scenarios, the dark side of it has not been fully exposed yet. The inability to fully understand and explain what has been done within the super-intelligence can turn an inherently benevolent system into malevolent. Recent research has revealed that the outputs of Deep Neural Networks (DNNs) can be easily corrupted by imperceptibly small input perturbations. As computations are brought nearer to the source of data creation, the attack surface of DNN has also been extended from the input data to the edge devices. Accordingly, due to the opportunities of ML-assisted security and the vulnerabilities of ML implementation, in this paper, we will survey the applications, vulnerabilities and fortification of ML from the perspective of hardware security. We will discuss the possible future research directions, and thereby, sharing a roadmap for the hardware security community in general.

Computer Simulations for Stability Analysis of a Numerical Procedure for Crossbar Packet Switch

Abstract The paper is concerned with the problem of calculating the bound of the throughput (THR) of a crossbar packet switch. For this purpose a suitable numerical procedure is used. The input data for the procedure are the results of computer simulations executed on a grid-computational structure (www.hpc.acad.bg). The modeling of the THR uses MiMa-algorithm for a crossbar packet switch, specified by the apparatus of Generalized Nets. For studying the stability of the numerical procedure we use a modified family of patterns for i.i.d. Bernoulli uniform load traffic simulation. The obtained results show that the numerical procedure is stable which means that small input perturbations result in small changes in the output.

A reanalysis of “Two types of asynchronous activity in networks of excitatory and inhibitory spiking neurons”

Neuronal activity in the central nervous system varies strongly in time and across neuronal populations. It is a longstanding proposal that such fluctuations generically arise from chaotic network dynamics. Various theoretical studies predict that the rich dynamics of rate models operating in the chaotic regime can subserve circuit computation and learning. Neurons in the brain, however, communicate via spikes and it is a theoretical challenge to obtain similar rate fluctuations in networks of spiking neuron models. A recent study investigated spiking balanced networks of leaky integrate and fire (LIF) neurons and compared their dynamics to a matched rate network with identical topology, where single unit input-output functions were chosen from isolated LIF neurons receiving Gaussian white noise input. A mathematical analogy between the chaotic instability in networks of rate units and the spiking network dynamics was proposed. Here we revisit the behavior of the spiking LIF networks and these matched rate networks. We find expected hallmarks of a chaotic instability in the rate network: For supercritical coupling strength near the transition point, the autocorrelation time diverges. For subcritical coupling strengths, we observe critical slowing down in response to small external perturbations. In the spiking network, we found in contrast that the timescale of the autocorrelations is insensitive to the coupling strength and that rate deviations resulting from small input perturbations rapidly decay. The decay speed even accelerates for increasing coupling strength. In conclusion, our reanalysis demonstrates fundamental differences between the behavior of pulse-coupled spiking LIF networks and rate networks with matched topology and input-output function. In particular there is no indication of a corresponding chaotic instability in the spiking network.

A reanalysis of "Two types of asynchronous activity in networks of excitatory and inhibitory spiking neurons".

Neuronal activity in the central nervous system varies strongly in time and across neuronal populations. It is a longstanding proposal that such fluctuations generically arise from chaotic network dynamics. Various theoretical studies predict that the rich dynamics of rate models operating in the chaotic regime can subserve circuit computation and learning. Neurons in the brain, however, communicate via spikes and it is a theoretical challenge to obtain similar rate fluctuations in networks of spiking neuron models. A recent study investigated spiking balanced networks of leaky integrate and fire (LIF) neurons and compared their dynamics to a matched rate network with identical topology, where single unit input-output functions were chosen from isolated LIF neurons receiving Gaussian white noise input. A mathematical analogy between the chaotic instability in networks of rate units and the spiking network dynamics was proposed. Here we revisit the behavior of the spiking LIF networks and these matched rate networks. We find expected hallmarks of a chaotic instability in the rate network: For supercritical coupling strength near the transition point, the autocorrelation time diverges. For subcritical coupling strengths, we observe critical slowing down in response to small external perturbations. In the spiking network, we found in contrast that the timescale of the autocorrelations is insensitive to the coupling strength and that rate deviations resulting from small input perturbations rapidly decay. The decay speed even accelerates for increasing coupling strength. In conclusion, our reanalysis demonstrates fundamental differences between the behavior of pulse-coupled spiking LIF networks and rate networks with matched topology and input-output function. In particular there is no indication of a corresponding chaotic instability in the spiking network.

Robustness Analysis of Floating-Point Programs by Self-Composition

Robustness is a key property for critical systems that run in uncertain environments, to ensure that small input perturbations can cause only small output changes. Current critical systems often involve lots of floating-point computations which are inexact. Robustness analysis of floating-point programs needs to consider both the uncertain inputs and the inexact computation. In this paper, we propose to leverage the idea of self-composition to transform the robustness analysis problem into a reachability problem, which enables the use of standard reachability analysis techniques such as software model checking and symbolic execution for robustness analysis. To handle floating-point arithmetic, we employ an abstraction that encompasses the effect of rounding and that can encompass all rounding modes. It converts floating-point expressions into linear expressions with interval coefficients in exact real arithmetic. On this basis, we employ interval linear programming to compute the maximum output change or maximum allowed input perturbation for the abstracted programs. Preliminary experimental results of our prototype implementation are encouraging.

Feature selection for k-means clustering stability: theoretical analysis and an algorithm

Stability of a learning algorithm with respect to small input perturbations is an important property, as it implies that the derived models are robust with respect to the presence of noisy features and/or data sample fluctuations. The qualitative nature of the stability property enhardens the development of practical, stability optimizing, data mining algorithms as several issues naturally arise, such as: how "much" stability is enough, or how can stability be effectively associated with intrinsic data properties. In the context of this work we take into account these issues and explore the effect of stability maximization in the continuous (PCA-based) k-means clustering problem. Our analysis is based on both mathematical optimization and statistical arguments that complement each other and allow for the solid interpretation of the algorithm's stability properties. Interestingly, we derive that stability maximization naturally introduces a tradeoff between cluster separation and variance, leading to the selection of features that have a high cluster separation index that is not artificially inflated by the features variance. The proposed algorithmic setup is based on a Sparse PCA approach, that selects the features that maximize stability in a greedy fashion. In our study, we also analyze several properties of Sparse PCA relevant to stability that promote Sparse PCA as a viable feature selection mechanism for clustering. The practical relevance of the proposed method is demonstrated in the context of cancer research, where we consider the problem of detecting potential tumor biomarkers using microarray gene expression data. The application of our method to a leukemia dataset shows that the tradeoff between cluster separation and variance leads to the selection of features corresponding to important biomarker genes. Some of them have relative low variance and are not detected without the direct optimization of stability in Sparse PCA based k-means. Apart from the qualitative evaluation, we have also verified our approach as a feature selection method for $$k$$ k -means clustering using four cancer research datasets. The quantitative empirical results illustrate the practical utility of our framework as a feature selection mechanism for clustering.

Numerical Errors in Minimization Based Binding Energy Calculations

This work examines the effect of small input perturbations on binding energies computed from differences between energy minimized structures, such as the Prime MM-GBSA and MOE MM-GB/VI methods. The applied perturbations include translations of the cognate ligand in the binding site by a maximum of 0.1 Å along each coordinate or the permutation of the order of atoms of the cognate ligand without any changes to the atom coordinates. These seemingly inconsequential input changes can lead to as much as 17 kcal/mol differences in the computed binding energy. The calculated binding energies cluster around discrete values, which correspond to specific ligand poses. It appears that the largest variations are observed for target-ligand systems in which there is a possibility for multiple poses with strong hydrogen bonds. The barriers between different poses can appear fractal-like, making it difficult to predict which solution will be produced from a given input. Including protein flexibility in MM-GBSA calculations further increases numerical instability, and the protein strain terms seem to be the major factor contributing to this sensitivity. In such calculations it appears unwise to extend the flexible region beyond 6 Å.

Controlled linear perturbation

We present an algorithmic solution to the robustness problem in computational geometry, called controlled linear perturbation, and demonstrate it on Minkowski sums of polyhedra. The robustness problem is how to implement real RAM algorithms accurately and efficiently using computer arithmetic. Approximate computation in floating point arithmetic is efficient but can assign incorrect signs to geometric predicates, which can cause combinatorial errors in the algorithm output. We make approximate computation accurate by performing small input perturbations, which we compute using differential calculus. This strategy supports fast, accurate Minkowski sum computation. The only prior robust implementation uses a less efficient algorithm, requires exact algebraic computation, and is far slower based on our extensive testing.

Reducing Docking Score Variations Arising from Input Differences

The variability of docking results as a function of variations in ligand input conformations was studied for the GOLD, Glide, FlexX, and Surflex programs. It is concluded that there are two major effects leading to such variability: the adequacy of conformational search during docking and random "chaotic" effects arising from sensitivity to small input perturbations. It is shown that although the former is generally the stronger effect, the latter is also highly significant for almost all docking engines. The strong target-to-target variation of the magnitude of these effects is emphasized. The performance of different packages is compared using these measures. Guidelines are provided for different programs to reduce variability and improve reproducibility, which involve using a small number of input conformations as starting points for docking, followed by the selection of the top scoring docked pose from the results as the best docked solution.

Investigation of a novel nonlinear detector of weak acoustic signals in noise

Recent work has identified and explored a method of detecting weak signals in Gaussian noise, based on the response of a nonlinear system to small input perturbations. The technique is applicable to the detection of weak underwater acoustic signals in unwanted ambient sea noise. Much of the research to date has been carried out using a forced Duffing oscillator as the nonlinear system, with parameters selected so as to place the system at the point of transition from a chaotic state to a stable periodic state. The additive input perturbation, most commonly comprising a simulated single frequency embedded in Gaussian noise with a very low input SNR, has been found to initiate the transition as indicated by the appearance of a stable trajectory in phase space. The attendant increase in amplitude at the output of the detector, at or near to the signal frequency, constitutes a detection of that weak signal. This paper presents a quantitative investigation of the technique as applied to underwater acoustic signals (real and simulated), using receiver operating characteristics (ROC) as a measure of its performance. Both the Duffing system, and a more novel version of the technique using a forced Lorenz system, are investigated.

Small-Signal Analysis of Oscillators Using Generalized Multitime Partial Differential Equations

Standard small-signal analysis methods for circuits break down for oscillators because small-input perturbations result in arbitrarily large-output changes, thus invalidating fundamental assumptions for small-signal analysis. In this paper, we propose a novel oscillator ac approach remedying this situation, thus restoring validity and rigour to small-signal analysis of oscillators. Our approach centers around a novel general equation formulation for circuits that we term the Generalized Multitime Partial Differential Equations (GeMPDE). While this formulation is broadly applicable to any kind of circuit or dynamical system, we show that it has unique advantages for oscillators in that small-input perturbations now lead to small output ones, thus making small-signal analysis valid. A key feature of our approach is to solve for bivariate-frequency variables with the help of novel augmenting-phase-condition equations. Unlike prior oscillator-analysis methods, which require special handling of the phase mode, our GeMPDE-based small-signal analysis provides both amplitude and frequency characteristics in a unified manner and is applicable to any kind of oscillator described by differential equations. We obtain speedups of 1-2 orders of magnitude over the transient-simulation approach commonly used today by designers for oscillator-perturbation analysis

Jacobians of an Operational Prognostic Cloud Scheme

Abstract A detailed examination of the Jacobian matrix of sensitivities of the prognostic cloud scheme operational at the European Centre for Medium-Range Weather Forecasts (ECMWF) is presented. These Jacobians exhibit sensitivities of output variables (e.g., cloud condensate) to small input perturbations on temperature and moisture. The coupling of the cloud scheme with the ECMWF convective mass-flux scheme is considered. The sensitivity of the cloud scheme is split into all its contributing parts in order to extract the dominant terms. A selection of contrasted convective cases normally present in a regular operational forecast is considered. Some comparisons are made with a simpler diagnostic cloud scheme. It is shown that the main contributing terms to the sensitivity in cloud condensate l are the following for cases of deep convection: detrainment from moist convection, evaporation processes, and conversion of cloud water into rain. The structure of the Jacobians in terms of temperature and moisture ...

A MEASURE OF RELATIVE ROBUSTNESS FOR FEEDFORWARD NEURAL NETWORKS SUBJECT TO SMALL INPUT PERTURBATIONS

The relative robustness of artificial neural networks subject to small input perturbations (e.g. measurement noises) is an important issue in real world applications. This paper uses the concept of input-output sensitivity analysis to derive a relative network robustness measure for different feedforward neural network configurations. For illustration purposes, this measure is used to compare different neural network configurations designed for detecting incipient faults in induction motors. Analytical and simulation results are presented to show that the relative network robustness measure derived in this paper is an effective indicator of the relative performance of different feedforward neural network configurations in noisy environments and that this measure should be considered in the design of neural networks for real time applications. The concept of input-output sensitivity analysis and relative network robustness measure presented can be extended to analyze other neural networks designed for on-line applications.