Output Of Deep Neural Network Research Articles

The R2D2 Deep Neural Network Series Paradigm for Fast Precision Imaging in Radio Astronomy

Radio-interferometric imaging entails solving high-resolution high-dynamic-range inverse problems from large data volumes. Recent image reconstruction techniques grounded in optimization theory have demonstrated remarkable capability for imaging precision, well beyond CLEAN’s capability. These range from advanced proximal algorithms propelled by handcrafted regularization operators, such as the SARA family, to hybrid plug-and-play (PnP) algorithms propelled by learned regularization denoisers, such as AIRI. Optimization and PnP structures are however highly iterative, which hinders their ability to handle the extreme data sizes expected from future instruments. To address this scalability challenge, we introduce a novel deep-learning approach, dubbed “Residual-to-Residual DNN series for high-Dynamic-range imaging” or in short R2D2. R2D2's reconstruction is formed as a series of residual images, iteratively estimated as outputs of deep neural networks (DNNs) taking the previous iteration’s image estimate and associated data residual as inputs. It thus takes a hybrid structure between a PnP algorithm and a learned version of the matching pursuit algorithm that underpins CLEAN. We present a comprehensive study of our approach, featuring its multiple incarnations distinguished by their DNN architectures. We provide a detailed description of its training process, targeting a telescope-specific approach. R2D2's capability to deliver high precision is demonstrated in simulation, across a variety of image and observation settings using the Very Large Array. Its reconstruction speed is also demonstrated: with only a few iterations required to clean data residuals at dynamic ranges up to 105, R2D2 opens the door to fast precision imaging. R2D2 codes are available in the BASPLib (https://basp-group.github.io/BASPLib/) library on GitHub.

Quantitative analysis and cancellation of rotating modulation noise in gravity gradient measurement

Rotating accelerometer gravity gradiometer (RAGG) is an important instrument for auxiliary navigation and mineral exploration. The turntable in RAGG modulates the gravity gradient signal to a high frequency to avoid the influence of low-frequency noise. In order to quantitatively analyze and cancel the rotating modulation noise in gravity gradient measurement, the theoretical RAGG output error model with rotating modulation noise is derived and experimentally verified on a high-precision air-bearing turntable. The experimental results indicate that the dominant rotating modulation noise is turntable vibration noise, which results from the supply pressure fluctuations revealed by further analysis. Based on the rotating modulation noise characteristics, a deep neural network (DNN) is trained to cancel the rotating modulation noise from the RAGG output. Positively, the root-mean-square error (RMSE) of the trained DNN output is 1.7 E in sample data. Furthermore, a gravity gradient calibration platform is established to validate the effectiveness of this method by generating a gravity gradient signal with the same frequency as the rotating modulation noise and amplitude of 100 E. After cancelling the noise by the trained DNN, the RMSE of RAGG output reduced from 94 E to 7.7 E, while the RMSE of the result processed by wavelet analysis remains at 87 E. These results demonstrate that the trained DNN successfully cancels 92% of the rotating modulation noise in actual RAGG measurement without affecting the gravity gradient signal.

CriticalFuzz: A critical neuron coverage-guided fuzz testing framework for deep neural networks

Context:Deep neural networks (DNN) have been widely deployed in safety-critical domains, such as autonomous cars and healthcare, where error behaviors can lead to serious accidents, testing DNN is extremely important. Neuron coverage-guided fuzz testing (NCFT) has become an effective whitebox testing approach for testing DNN, which iteratively generates new test cases with the guidance of neuron coverage to explore different logics of DNN, and has found numerous defects. However, existing NCFT approaches ignore that the role of neurons is distinct for the final output of DNN. Given an input, only a fraction of neurons determines the final output of the DNN. These neurons hold the essential logic of the DNN. Objective:To ensure the quality of DNN and improve testing efficiency, NCFT should first cover neurons containing major logic of DNN. Method:In this paper, we propose the critical neurons that hold essential logic of DNN. In order to prioritize the detection of potential defects of critical neurons, we propose a fuzz testing framework, named CriticalFuzz, which mainly contains the energy-based test case generation and the critical neuron coverage criteria. The energy-based test case generation has the capability to produce test cases that are more likely to cover critical neurons and involves energy-based seed selection, power schedule, and seed mutation. The critical neuron coverage as a mechanism for providing feedback to guide the CriticalFuzz in prioritizing the coverage of critical neurons. To evaluate the significance of critical neurons and the performance of CriticalFuzz, we conducted experiments on popular DNNs and datasets. Results:The experiment results show that (1) the critical neurons have a 100% impact on the output of models, while the non-critical neurons have a lesser effect; (2) CriticalFuzz is effective in achieving 100% coverage of critical neurons and covering 10 classes of critical neurons, outperforming both DeepHunter and TensorFuzz. (3) CriticalFuzz exhibits exceptional error detection capabilities, successfully identifying thousands of errors across 10 diverse error classes within DNN. Conclusion:The critical neurons defined in this paper hold more significant logic of DNN than non-critical neurons. CriticalFuzz can preferentially cover critical neurons, thereby improving the efficiency of the NCFT process. Additionally, CriticalFuzz is capable of identifying a greater number of errors, thus enhancing the reliability and effectiveness of the NCFT.

MFABA: A More Faithful and Accelerated Boundary-Based Attribution Method for Deep Neural Networks

To better understand the output of deep neural networks (DNN), attribution based methods have been an important approach for model interpretability, which assign a score for each input dimension to indicate its importance towards the model outcome. Notably, the attribution methods use the ax- ioms of sensitivity and implementation invariance to ensure the validity and reliability of attribution results. Yet, the ex- isting attribution methods present challenges for effective in- terpretation and efficient computation. In this work, we in- troduce MFABA, an attribution algorithm that adheres to ax- ioms, as a novel method for interpreting DNN. Addition- ally, we provide the theoretical proof and in-depth analy- sis for MFABA algorithm, and conduct a large scale exper- iment. The results demonstrate its superiority by achieving over 101.5142 times faster speed than the state-of-the-art at- tribution algorithms. The effectiveness of MFABA is thor- oughly evaluated through the statistical analysis in compar- ison to other methods, and the full implementation package is open-source at: https://github.com/LMBTough/MFABA.

Gradient Correction for White-Box Adversarial Attacks.

Deep neural networks (DNNs) play key roles in various artificial intelligence applications such as image classification and object recognition. However, a growing number of studies have shown that there exist adversarial examples in DNNs, which are almost imperceptibly different from the original samples but can greatly change the output of DNNs. Recently, many white-box attack algorithms have been proposed, and most of the algorithms concentrate on how to make the best use of gradients per iteration to improve adversarial performance. In this article, we focus on the properties of the widely used activation function, rectified linear unit (ReLU), and find that there exist two phenomena (i.e., wrong blocking and over transmission) misguiding the calculation of gradients for ReLU during backpropagation. Both issues enlarge the difference between the predicted changes of the loss function from gradients and corresponding actual changes and misguide the optimized direction, which results in larger perturbations. Therefore, we propose a universal gradient correction adversarial example generation method, called ADV-ReLU, to enhance the performance of gradient-based white-box attack algorithms such as fast gradient signed method (FGSM), iterative FGSM (I-FGSM), momentum I-FGSM (MI-FGSM), and variance tuning MI-FGSM (VMI-FGSM). Through backpropagation, our approach calculates the gradient of the loss function with respect to the network input, maps the values to scores, and selects a part of them to update the misguided gradients. Comprehensive experimental results on ImageNet and CIFAR10 demonstrate that our ADV-ReLU can be easily integrated into many state-of-the-art gradient-based white-box attack algorithms, as well as transferred to black-box attacks, to further decrease perturbations measured in the l2 -norm.

Binning high-dimensional classifier output for HEP analyses through a clustering algorithm

Binning high-dimensional classifier output for HEP analyses through a clustering algorithm

Detection of backdoor attacks using targeted universal adversarial perturbations for deep neural networks

Backdoor attacks on deep neural networks (DNNs) using targeted universal adversarial perturbations (TUAPs) do not require training datasets and model tampering, and triggers based on TUAPs can make DNNs output any class the adversary wants. Retraining DNNs using adversarial training for security is time-consuming and does not apply to DNNs in runtime. We want to detect backdoors using a black-box testing approach. We observe that after superimposing random noise on the input of a backdoor attack, the output still tends to remain the same, so we propose Sequential Analysis method based on the Metamorphosis Testing (SAMT). We designed two metamorphic relations for test case generation. Using sequential sampling, we calculate the label stability rate (LSR) and infer whether the image to be verified contains a trigger based on the sequential probability ratio change. The experimental results show that our method has a higher backdoor detection success rate (dsr) than the state-of-the-art detection algorithms. Moreover, our method does not need to use the model structure of DNNs, which has more adaptability and generalization ability. Based on our proposed method, we can simply add a backdoor detection layer to detect backdoors as early as possible, which can eventually alleviate the harm of such backdoors.

A deep reinforcement learning assisted task offloading and resource allocation approach towards self-driving object detection

With the development of communication technology and mobile edge computing (MEC), self-driving has received more and more research interests. However, most object detection tasks for self-driving vehicles are still performed at vehicle terminals, which often requires a trade-off between detection accuracy and speed. To achieve efficient object detection without sacrificing accuracy, we propose an end–edge collaboration object detection approach based on Deep Reinforcement Learning (DRL) with a task prioritization mechanism. We use a time utility function to measure the efficiency of object detection task and aim to provide an online approach to maximize the average sum of the time utilities in all slots. Since this is an NP-hard mixed-integer nonlinear programming (MINLP) problem, we propose an online approach for task offloading and resource allocation based on Deep Reinforcement learning and Piecewise Linearization (DRPL). A deep neural network (DNN) is implemented as a flexible solution for learning offloading strategies based on road traffic conditions and wireless network environment, which can significantly reduce computational complexity. In addition, to accelerate DRPL network convergence, DNN outputs are grouped by in-vehicle cameras to form offloading strategies via permutation. Numerical results show that the DRPL scheme is at least 10% more effective and superior in terms of time utility compared to several representative offloading schemes for various vehicle local computing resource scenarios.

A DNN-based reliability evaluation method for multi-state series-parallel systems considering semi-Markov process

In order to evaluate the reliability of the multi-state series-parallel system considering semi-Markov process (MSSPS-SMP), the integral equations have been utilized to calculate the state probability distributions. Nevertheless, to solve the formulated integral equations, it is impossible to avoid time-consuming convolution operations for any numerical method, which can result in significant reliability evaluation complexity. To address the above problems, a deep neural network (DNN)-based method is proposed for the reliability evaluation of the MSSPS-SMP. For a multi-state component, the reliability parameters representing the arbitrary distributions of the SMP are firstly extracted as the input feature to DNN, while the corresponding state probability distributions serve as the output of DNN naturally. On this basis, the DNN is deployed to establish a direct mapping relationship between the reliability parameters and state probability distributions. Instead of repeating complicated calculations of SMP-related convolution operation, the well-trained DNN model can effectively determine the performance distributions of multi-state components given the varying reliability parameters. On this basis, the Lz-transform technique is utilized to develop the unified representations of dynamic reliability models of various multi-state components considering SMP. Combined with the Lz-transform, the time-varying performance distribution of the MSSPS-SMP with complicated structures of several components can be determined.

Detection of adversarial attacks based on differences in image entropy

AbstractAlthough deep neural networks (DNNs) have achieved high performance across various applications, they are often deceived by adversarial examples generated by adding small perturbations. To combat adversarial attacks, many detection methods have been proposed, including feature squeezing and trapdoor. However, these methods rely on the output of DNNs or involve training a separate network to detect adversarial examples, which leads to high computational costs and low efficiency. In this study, we propose a simple and effective approach called the entropy-based detector (EBD) to protect DNNs from various adversarial attacks. EBD detects adversarial examples by comparing the difference in entropy between the input sample before and after bit depth reduction. We show that EBD can detect over 98% of the adversarial examples generated by attacks using fast-gradient sign method, basic iterative method, momentum iterative method, DeepFool and CW attacks when the false positive rate is 2.5% for CIFAR-10 and ImageNet datasets.

Physics-guided neural network for channeled spectropolarimeter spectral reconstruction.

A reconstruction method incorporates the complete physical model into a traditional deep neural network (DNN) is proposed for channeled spectropolarimeter (CSP). Unlike traditional DNN-based methods that need to employ training datasets, the method starts from randomly initialized parameters which are constrained by the CSP physical model. It iterates through the gradient descent algorithm to obtain the estimation of the DNN parameters and then to obtain the mapping relationship. As a result, it eliminates the need for thousands of sets of ground truth data, while also leveraging the physical model to achieve high-precision reconstruction. As seen, the physical model participates in the optimization process of DNN parameters, thus achieving physical guidance for the DNN output results. Based on the characteristic of the network, we designate this method as the physics-guided neural network (PGNN). Both simulations and experiments demonstrate the superior performance of the proposed method. Our approach will further promote the practical application of CSP in a wider range of fields.

WH 2 D 2 N 2 : Distributed AI-enabled OK-ASN Service for Web of Things

Model data-driven ontology and knowledge presentation for evolving semantic Asian social networks (OK-ASN) is a critical strategy for web of things (WoT) services. Meanwhile, Deep Neural Network (DNN)-based OK-ASN service in WoT is growing rapidly. However, most DNN-based services cannot utilize the potential of WoT fully, as heterogeneity exists in WoT. Therefore, this article proposes a novel framework called Web-based Heterogeneous Hierarchical Distributed Deep Neural Network ( WH 2 D 2 N 2 ) to deploy the DNNs for OK-ASN services on WoT, overcoming the heterogeneity. The architecture of the system and the designed Edge-Cloud-Joint execute scheme utilize heterogeneous devices to make DNN inference ubiquitous and output two types of results to meet various requirements. To bring robustness to OK-ASN services, a global scheduling is designed to arrange the workflow dynamically. The results of our experiments prove the efficiency of the execute scheme and the global scheduling in the system.

EAM: Ensemble of approximate multipliers for robust DNNs

Deep neural networks (DNNs) are one of the most prominent technologies of our time, as they achieve state-of-the-art performance in a wide range of real-world applications such as healthcare, online banking, etc. However, recent advances in deep learning have opened up a largely unexplored surface for adversarial attacks, jeopardizing the integrity of DNN systems. DNNs are vulnerable to adversarial samples that are generated by perturbing correctly classified inputs to cause DNN models to mispredict. This can potentially lead to catastrophic consequences, especially in critical and security-sensitive applications such as autonomous driving and video surveillance systems.We propose ensemble of approximate multipliers (EAM), an architectural technique that uses a combination of approximate multipliers to protect DNNs against adversarial attacks. Approximate computing is known for its effectiveness in improving the energy-efficiency of computing platforms at the cost of slight accuracy loss. Using approximation in the form of inexact multipliers is also effective for increasing robustness by injecting input-dependent noises into the outputs of DNNs. However, resiliency of DNNs against malicious attacks depends on the type of an approximate multiplier. Based on level of approximation, robustness of DNNs changes by a large margin. We exploit this variability and propose using an ensemble of different types of approximate multipliers. EAM increases robustness across a wide range of attack scenarios. In particular, we show that successful attacks against an exact DNN have poor transferability to EAM. The transferability is also poor for a black-box model where perturbed inputs are generated by a substitute model. In addition, EAM is resilient against a white-box attack scenario where an attacker has full knowledge of approximate hardware. Our ensemble technique does not require duplication of computing cores nor memory units. EAM only uses additional approximate compressors within the original multipliers. We conduct extensive experiments on a set of strong adversarial attacks. We empirically show that EAM increases robustness over the exact model with negligible impact on accuracy of benign inputs.

Integrating deep neural network with logic rules for credit scoring

Credit scoring is an important topic in financial activities and bankruptcy prediction that has been extensively explored using deep neural network (DNN) methods. DNN-based credit scoring models rely heavily on a large amount of labeled data. The accuracy of DNN-based credit assessment models relies heavily on large amounts of labeled data. However, purely data-driven learning makes it difficult to encode human intent to guide the model to capture the desired patterns and leads to low transparency of the model. Therefore, the Probabilistic Soft Logic Posterior Regularization (PSLPR) framework is proposed for integrating prior knowledge of logic rule with neural network. First, the PSLPR framework calculates the rule satisfaction distance for each instance using a probabilistic soft logic formula. Second, the logic rules are integrated into the posterior distribution of the DNN output to form a logic output. Finally, a novel discrepancy loss which measures the difference between the real label and the logic output is used to incorporate logic rules into the parameters of the neural network. Extensive experiments were conducted on two datasets, the Australian credit dataset and the credit card customer default dataset. To evaluate the obtained systems, several performance metrics were used, including PCC, Recall, F1 and AUC. The results show that compared to the standard DNN model, the four evaluation metrics are increased by 7.14%, 14.29%, 8.15%, and 5.43% respectively on the Australian credit dataset.

Profiling Deep Learning Side-Channel Attacks Using Multi-Label against AES Circuits with RSM Countermeasure

Deep learning side-channel attacks (DL-SCAs) have been actively studied in recent years. In the DL-SCAs, deep neural networks (DNNs) are trained to predict the internal states of the cryptographic operation from the side-channel information such as power traces. It is important to select suitable DNN output labels expressing an internal states for successful DL-SCAs. We focus on the multi-label method proposed by Zhang et al. for the hardware-implemented advanced encryption standard (AES). They used the power traces supplied from the AES-HD public dataset, and reported to reveal a single key byte on conditions in which the target key was the same as the key used for DNN training (profiling key). In this paper, we discuss an improvement for revealing all the 16 key bytes in practical conditions in which the target key is different from the profiling key. We prepare hardware-implemented AES without SCA countermeasures on ASIC for the experimental environment. First, our experimental results show that the DNN using multi-label does not learn side-channel leakage sufficiently from the power traces acquired with only one key. Second, we report that DNN using multi-label learns the most of side-channel leakage by using three kinds of profiling keys, and all the 16 target key bytes are successfully revealed even if the target key is different from the profiling keys. Finally, we applied the proposed method, DL-SCA using multi-label and three profiling keys against hardware-implemented AES with rotating S-boxes masking (RSM) countermeasures. The experimental result shows that all the 16 key bytes are successfully revealed by using only 2,000 attack traces. We also studied the reasons for the high performance of the proposed method against RSM countermeasures and found that the information from the weak bits is effectively exploited.

Cross-Layer Codebook Allocation for Uplink SCMA and PDNOMA-SCMA Video Transmission Systems and a Deep Learning-Based Approach

Sparse code multiple access (SCMA) and power-domain nonorthogonal multiple access (PDNOMA) have been considered for use with fifth-generation (5G) and beyond mobile communications, in order to increase the bandwidth efficiency, reduce the latency in terms of waiting for resources, and increase the numbers of connections. Existing codebook allocation schemes for SCMA and PDNOMA-SCMA systems consider only the physical layer. We propose an iterative cross-physical-and-application-layer codebook allocation for uplink video communications. We allocate the codebook according to the channel state information of the physical layer, and codebooks are exchanged among users to minimize the sum of the video mean square error in order to increase the average peak signal-to-noise ratio (PSNR). Furthermore, we propose a deep neural network (DNN) approach to reduce the computational complexity. We propose a novel postprocessing algorithm that is applied to the DNN output to ensure the constraints of codebook allocation are satisfied. Simulation results show that the proposed iterative PDNOMA-SCMA cross-layer codebook allocation scheme outperforms an existing scheme that uses only the physical layer by 1.27 dB in terms of PSNR (video quality) for 12 users, four subcarriers, and SNR = 21 dB. The proposed DNN-based approach has an execution time that is 67% lower than the proposed iterative PDNOMA-SCMA cross-layer codebook allocation scheme, at the cost of a performance degradation of 0.84 dB.

Visualization and accuracy improvement of soil classification using laser-induced breakdown spectroscopy with deep learning

Deep learning method is applied to spectral detection due to the advantage of not needing feature engineering. In this work, the deep neural network (DNN) model is designed to perform data mining on the laser-induced breakdown spectroscopy (LIBS) spectra of the ore. The potential of heat diffusion for an affinity-based transition embedding model is first used to perform nonlinear mapping of fully connected layer data in the DNN model. Compared with traditional methods, the DNN model has the highest recognition accuracy rate (75.92%). A training set update method based on DNN output is proposed, and the final model has a recognition accuracy of 85.54%. The method of training set update proposed in this work can not only obtain the sample labels quickly but also improve the accuracy of deep learning models. The results demonstrate that LIBS combined with the DNN model is a valuable tool for ore classification at a high accuracy rate.

What should AI see? Using the public’s opinion to determine the perception of an AI

Deep neural networks (DNN) have made impressive progress in the interpretation of image data so that it is conceivable and to some degree realistic to use them in safety critical applications like automated driving. From an ethical standpoint, the AI algorithm should take into account the vulnerability of objects or subjects on the street that ranges from “not at all”, e.g. the road itself, to “high vulnerability” of pedestrians. One way to take this into account is to define the cost of confusion of one semantic category with another and use cost-based decision rules for the interpretation of probabilities, which are the output of DNNs. However, it is an open problem how to define the cost structure, who should be in charge to do that, and thereby define what AI-algorithms will actually “see”. As one possible answer, we follow a participatory approach and set up an online survey to ask the public to define the cost structure. We present the survey design and the data acquired along with an evaluation that also distinguishes between perspective (car passenger vs. external traffic participant) and gender. Using simulation based F-tests, we find highly significant differences between the groups. These differences have consequences on the reliable detection of pedestrians in a safety critical distance to the self-driving car. We discuss the ethical problems that are related to this approach and also discuss the problems emerging from human–machine interaction through the survey from a psychological point of view. Finally, we include comments from industry leaders in the field of AI safety on the applicability of survey based elements in the design of AI functionalities in automated driving.

Universal BlackMarks: Key-Image-Free Blackbox Multi-Bit Watermarking of Deep Neural Networks

Existing methods for Deep Neural Networks (DNN) watermarking either require accessing the internal parameters of the DNN models (white-box watermarking), or rely on backdooring to enforce a desired behavior of the model when the DNN is fed with a specific set of key input images (black-box watermarking). In this letter, we propose a black-box multi-bit DNN watermarking algorithm, suitable for multiclass classification networks, whereby the presence of the watermark can be retrieved from the output of the network in correspondence to <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">any</i> input. To read the watermark, we first apply a power function to the softmax output of the DNN model to map it from an impulse-like to a smooth distibution. Then, we extract the watermark bits by projecting the output of the DNN onto a pseudorandom key vector. Watermark embedding is achieved by adding a proper regularizer term to the training loss. The effectiveness of the proposed method is demonstrated by applying it to various network architectures working on different datasets. The experimental results demonstrate the possibility to embed a robust watermark into the output of the host DNN with a negligible impact on the accuracy of the original task.

Adversarial data hiding with only one pixel

Making adversarial samples to fool deep neural network (DNN) is an emerging research direction of privacy protection, since the output of the attacker's DNN can be easily changed by the well-designed tiny perturbation added to the input vector. However, the added perturbation is meaningless. Why not embed some useful information to generate adversarial samples while integrating the functions of copyright and integrity protection of data hiding? This paper solves the problem by modifying only one pixel of the image, that is, data hiding and adversarial sample generation are achieved simultaneously by the only one modified pixel. In CIFAR-10 dataset, 11 additional bits can be embedded into the host images sized 32 × 32, and the successful rate of adversarial attack is close to the state-of-the-art works. This paper proposes a new idea to combine data hiding and adversarial sample generation, and gives a new method for privacy-preserved processing of image big data.