Output Of Deep Neural Network Research Articles

TPE-based Short-Term Load Forecasting Neural Network Structure Design and Parameter Optimization Method

Short-term load forecasting is generally applied in power system real-time dispatching and day-ahead generation planning, which is of great significance for power system economic dispatching and safe operation of the system. Much research on short-term load forecasting using neural networks has been carried out at home and abroad. However, how to obtain the optimal network and structure evolution accurately and quickly has evolved into a key challenge for short-term load forecasting, because the prediction effect of the neural network method is more easily affected by the existing network structure and parameters, and the personality difference of the prediction object itself makes it difficult for the parameters to be reused. Aiming at this problem, this paper proposes a TPE-based neural network structure design and parameter optimization method for short-term load forecasting. This method realizes fast optimization of parameters of different types, structures, parameters, and input and output of deep neural networks, and is widely applied in Guangxi power grid. It is verified by an actual example that the method can effectively improve prediction accuracy.

WITHDRAWN: Semi-Supervised Learning with Ensemble Deep Learning Networks for Descriptor Generation

Deep neural networks and machine learning have made many real-world tasks easier. They can analyze large data sets that humans can't. Deep neural networks can be utilized for image, video, and sound matching, making them an intriguing research area. This research focuses on semi-supervised training and proposes a "enhanced semi-supervised" learning strategy. Robots can learn from labeled and unlabeled data with semi-supervised learning. Supervised learning employs labeled data. Semi-supervised learning is used to govern the Deep neural network's output. Assigning values to input groups and accessing the output area have no interaction. This method seeks to deliver a more efficient learning approach with an equitable distribution of output throughout the output field of space, and the authors developed an ensemble strategy based on three deep learning approaches (Convolution Deep neural network, Alexnet, and MobileNetv2). The result is a more effective learning approach. The handwritten digit experiment was 90.74 percent accurate, whereas Alzheimer's detection was 99.76 percent accurate. When the proposed strategy was used on two experimental data sets, the accuracy was better than when different applications used Siamese Deep neural networks.

A Trustable Data-Driven Framework for Composite System Reliability Evaluation

Composite system reliability (CSR) evaluations suffer from heavy computational burdens because an enormous number of sampled system states must be analyzed. Recently, data-driven approaches are of growing interest because they can accelerate such evaluations by preidentifying success/failure samples. However, the current approaches are limited by unsatisfactory accuracy. Essentially, the main reason is that the data-driven results are not credible. In this article, a trustable data-driven framework based on a deep neural network (DNN) is proposed to enable accurate and credible results for CSR evaluations by the following two aspects: First, a fine-grained identification method with an effective learning strategy is proposed to effectively extract the data features of CSR evaluations and improve the identification accuracy of failure samples. Second, a secure strategy including adaptability criteria and a correction method for the DNN is proposed to guarantee the accuracy of data-driven results. The strategy can determine whether the DNN is utilized to evaluate CSR and reasonably correct the DNN outputs. The simulation results demonstrate the effectiveness of the proposed methods.

Deep Learning for Encrypted Traffic Classification and Unknown Data Detection.

Despite the widespread use of encryption techniques to provide confidentiality over Internet communications, mobile device users are still susceptible to privacy and security risks. In this paper, a novel Deep Neural Network (DNN) based on a user activity detection framework is proposed to identify fine-grained user activities performed on mobile applications (known as in-app activities) from a sniffed encrypted Internet traffic stream. One of the challenges is that there are countless applications, and it is practically impossible to collect and train a DNN model using all possible data from them. Therefore, in this work, we exploit the probability distribution of a DNN output layer to filter the data from applications that are not considered during the model training (i.e., unknown data). The proposed framework uses a time window-based approach to divide the traffic flow of activity into segments so that in-app activities can be identified just by observing only a fraction of the activity-related traffic. Our tests have shown that the DNN-based framework has demonstrated an accuracy of 90% or above in identifying previously trained in-app activities and an average accuracy of 79% in identifying previously untrained in-app activity traffic as unknown data when this framework is employed.

Electromagnetic field computation of multilayer vacuum chambers with physics-informed neural networks

The electromagnetic interaction of a charged particle beam with multilayer vacuum chambers is of particular interest in accelerator physics. This paper presents a deep learning-based approach for calculating electromagnetic fields generated by the beam in infinitely long multilayer vacuum chambers with arbitrary cross section. The presented approach is based on physics-informed neural networks and the surface impedance boundary condition of a multilayer structure derived from the transmission line theory. Deep neural networks (DNNs) are utilized to approximate the solution of partial differential equations (PDEs) describing the physics of electromagnetic fields self-generated by a charged particle beam traveling in a particle accelerator. A residual network is constructed from the output of DNNs, the PDEs and boundary conditions are embedded into the loss function and differential operators are calculated using the automatic differentiation. As a result, the presented approach is regarded to be mesh-free. The approach is applied to circular and elliptical vacuum chambers with a three-layer structure. It is verified in comparison with the recently proposed boundary element method. The effects of chamber geometries and multilayer structure on the beam coupling impedance are demonstrated.

Learning-Based Resource Allocation for Ultra-Reliable V2X Networks With Partial CSI

In this paper, we study the resource allocation in high mobility vehicle-to-everything (V2X) networks with only slowly varying large-scale channel parameters. For satisfying the diversity requirements of different types of links, i.e., low delay for vehicle-to-infrastructure (V2I) connections and ultra-reliability for vehicle-to-vehicle (V2V) connections, we formulate a joint power, spectrum and vehicle local computing ratio allocation problem to minimize the delay of V2I links whilst satisfying the V2V reliability constraint. For solving the formulated problem, a Feasible Region Transformation Method is firstly developed to convert the probabilistic V2V reliability requirement into a computable constraint. In addition, a Robust Signal to Interference Plus Noise Ratio (SINR) Modified Method is proposed to give the computable expression for the V2I throughput. Then, a parallel Deep Neural Network (DNN) framework is designed for the resource allocation in V2X networks, where one is the transmit power control unit and the other is the local computing ratio allocation unit. After that, a Feedback-oriented Learning Method is proposed to train the parallel DNN-based resource allocation framework, in which the output of DNN is used as feedback to dynamically revise the training loss function along with the training process. Afterwards, the Hungarian method is employed to obtain the optimal spectrum matching. Finally, we conduct the simulations to show that the proposed learning-based algorithm has better performance compared with other general algorithms.

Towards Neural Routing with Verified Bounds on Performance

When data-driven algorithms, especially the ones based on deep neural networks (DNNs), replace classical ones, their superior performance often comes with difficulty in their analysis. On the way to compensate for this drawback, formal verification techniques, which can provide reliable guarantees on program behavior, were developed for DNNs. These techniques, however, usually consider DNNs alone, excluding real-world environments in which they operate, and the applicability of techniques that do account for such environments is often limited. In this work, we consider the problem of formally verifying a neural controller for the routing problem in a conveyor network. Unlike in known problem statements, our DNNs are executed in a distributed context, and the performance of the routing algorithm, which we measure as the mean delivery time, depends on multiple executions of these DNNs. Under several assumptions, we reduce the problem to a number of DNN output reachability problems, which can be solved with existing tools. Our experiments indicate that sound-and-complete formal verification in such cases is feasible, although it is notably slower than the gradient-based search of adversarial examples.The paper is structured as follows. Section 1 introduces basic concepts. Then, Section 2 introduces the routing problem and DQN-Routing, the DNN-based algorithm that solves it. Section 3 proposes the contribution of this paper: a novel sound and complete approach to formally check an upper bound on the mean delivery time of DNN-based routing. This approach is experimentally evaluated in Section 4. The paper is concluded with some discussion of the results and outline of possible future work.

GreedyFool: Multi-factor imperceptibility and its application to designing a black-box adversarial attack

Adversarial examples are well-designed input samples that include perturbations that are imperceptible to the human eye but easily mislead the output of deep neural networks (DNNs). Existing studies synthesize adversarial examples by leveraging simple metrics to penalize perturbations that lack sufficient consideration of the human visual system (HVS), producing noticeable artifacts. To explore why these perturbations are visible,four primary factors that affect the perceptibility of human eyes are summarized in this paper. Based on this investigation, we design a multi-factor metric MulFactorLoss for measuring the perceptual loss between benign examples and adversarial examples. To test the imperceptibility of the multi-factor metric, we propose a novel black-box adversarial attack that is referred to as GreedyFool. GreedyFool applies differential evolution to evaluate the effects of perturbed pixels on the confidence of a target DNN and introduces greedy approximation to automatically generate adversarial perturbations. We conduct extensive experiments on the ImageNet and CIFRA-10 datasets and a comprehensive user study with 60 participants. The experimental results demonstrate that MulFactorLoss is a more imperceptible metric than the existing pixelwise metrics, and GreedyFool achieves a 100% success rate in a black-box manner.

Combating Label Noise in Image Data Using MultiNET Flexible Confident Learning

Deep neural networks (DNNs) have been used successfully for many image classification problems. One of the most important factors that determines the final efficiency of a DNN is the correct construction of the training set. Erroneously labeled training images can degrade the final accuracy and additionally lead to unpredictable model behavior, reducing reliability. In this paper, we propose MultiNET, a novel method for the automatic detection of noisy labels within image datasets. MultiNET is an adaptation of the current state-of-the-art confident learning method. In contrast to the original, our method aggregates the outputs of multiple DNNs and allows for the adjustment of detection sensitivity. We conduct an exhaustive evaluation, incorporating four widely used datasets (CIFAR10, CIFAR100, MNIST, and GTSRB), eight state-of-the-art DNN architectures, and a variety of noise scenarios. Our results demonstrate that MultiNET significantly outperforms the confident learning method.

Towards harnessing feature embedding for robust learning with noisy labels

The memorization effect of deep neural networks (DNNs) plays a pivotal role in recent label noise learning methods. To exploit this effect, the model prediction-based methods have been widely adopted, which aim to exploit the outputs of DNNs in the early stage of learning to correct noisy labels. However, we observe that the model will make mistakes during label prediction, resulting in unsatisfactory performance. By contrast, the produced features in the early stage of learning show better robustness. Inspired by this observation, in this paper, we propose a novel feature embedding-based method for deep learning with label noise, termed Lab El Noise Dilution (LEND). To be specific, we first compute a similarity matrix based on current embedded features to capture the local structure of training data. Then, the noisy supervision signals carried by mislabeled data are overwhelmed by nearby correctly labeled ones (i.e., label noise dilution), of which the effectiveness is guaranteed by the inherent robustness of feature embedding. Finally, the training data with diluted labels are further used to train a robust classifier. Empirically, we conduct extensive experiments on both synthetic and real-world noisy datasets by comparing our LEND with several representative robust learning approaches. The results verify the effectiveness of our LEND.

Numerical solution for elliptical orbit pursuit-evasion game via deep neural networks and pseudospectral method

This paper presents an efficient and stable DNNs-based Radau pseudospectral method for the free-time elliptical orbit pursuit-evasion game based on the equivalent reconstruction of the game model. Firstly, the relative dynamics equations are established by adding the nonlinear terms caused by the eccentricity to the Hill–Clohessy–Wilshire equations. Then the original pursuit-evasion problem is deduced to a 4-dimensional one-sided optimal control problem (OCP) based on the equivalent reconstruction. Secondly, in order to apply the deep neural networks (DNNs) to map the relationship between the OCP and the solution, the normalization of costates is introduced to eliminate the non-uniqueness of solutions when generating samples for training DNNs. Thirdly, the DNNs-based Radau pseudospectral method is proposed where the DNNs output the guesses of solutions to the derived OCP and the Radau pseudospectral method optimizes the histories of controls obtained by the guesses to the convergence. The simulation results demonstrate that the proposed method converges more stably and decreases the calculation time greatly compared with the traditional indirect method.

Deep Feature Compression Using Spatio-Temporal Arrangement Toward Collaborative Intelligent World

Collaborative Intelligence is a new paradigm that splits a deep neural network (DNN) into an edge and cloud for deploying a DNN-based image recognition application. In this paradigm, deep features, which are the outputs of the edge DNN, are compressed and transmitted to the cloud DNN. Because the deep features have a number of responses that are similar to each other, for efficient compression, previous methods spatially arrange and compress the deep features as an image to utilize the similarity as a spatial correlation. However, if the deep features are arranged in not only spatial but also temporal directions like those in a video, it may be possible to compress them more efficiently by increasing a temporal correlation. To explore this possibility, we propose a “spatio-temporal arrangement”. This method spatially arranges the deep features as images and temporally arranges them as a video with a novel ordering search algorithm. Our method effectively increases the spatial and temporal correlations hidden in the deep features and achieves high compression efficiency compared with the previous methods. Experimental results demonstrate the compression efficiency of our method is better than that of the previous methods (1.50% to 4.98% on BD-Rate evaluation in a lossy setting). Our analysis shows that our method effectively increases the correlation when the input is an image with rich edges and textures.

Feature-filter: Detecting adversarial examples by filtering out recessive features

Deep neural networks (DNNs) have achieved state-of-the-art performance in numerous tasks involving complex analysis of raw data, such as self-driving systems and biometric recognition systems. However, recent works have shown that DNNs are under threat from adversarial example attacks. The adversary can easily change the outputs of DNNs by adding small well-designed perturbations to inputs. Adversarial example detection is fundamental for robust DNN-based services. From a human-centric perspective, this paper divides image features into dominant features comprehensible to humans and recessive features incomprehensible to humans yet exploited by DNNs. Based on this perspective, the paper proposes a new viewpoint that imperceptible adversarial examples are the product of recessive features misleading neural networks, and that the adversarial attack enriches these recessive features. The imperceptibility of the adversarial examples indicates that the perturbations enrich recessive features but hardly affect dominant features. Therefore, adversarial examples are sensitive to filtering out recessive features, while benign examples are immune to such operations. Inspired by this idea, we propose a label-only adversarial detector that is referred to as a feature-filter. The feature-filter utilizes the discrete cosine transform (DCT) to approximately separate recessive features from dominant features and obtain a filtered image. A comprehensive user study demonstrates that the DCT-based filter can reliably filter out recessive features from the test image. By comparing only the DNN’s prediction labels on the input and its filtered version, the feature-filter can detect imperceptible adversarial examples in real time with high accuracy and few false-positives.

A deep neural network based SMAP soil moisture product

In this paper, it is demonstrated that while satellite soil moisture (SM) retrievals often have minimum biases, reanalysis data can capture more temporal variability of SM, especially for non-cropland areas – when validated against in situ measurements. Accordingly, this paper presents a deep neural network (DNN) that utilizes the merits of a suite of existing satellite and reanalysis products to produce a new SM product with minimum (maximum) bias (correlation) – using NASA's Soil Moisture Active Passive (SMAP) data and ERA5 reanalysis. The benchmark of the network is a bias-adjusted SM with maximum correlation with in situ data over each land-cover type. The mean of the benchmark data is adjusted to the product that exhibits a minimum bias over each land-cover type. Consistent with the laws of L-band microwave propagation in soil and canopy, the input variables of DNN include polarized SMAP brightness temperatures, incidence angle, vegetation scattering albedo, surface roughness parameter, surface water fraction, effective soil temperatures, bulk density, clay fraction, and vegetation optical depth from the normalized difference vegetation index (NDVI) climatology. The DNN is trained and validated using two years (04/2015–03/2017) of global data and deployed for assessment of its performance from 04/2017 to 03/2021. The testing results against in situ measurements demonstrate that the DNN outputs typically exhibit improved error quality metrics over most land-cover types and climate regimes and can properly capture SM temporal dynamics, beyond each SMAP product across regional to continental scales.

Simple Black-Box Universal Adversarial Attacks on Deep Neural Networks for Medical Image Classification

Universal adversarial attacks, which hinder most deep neural network (DNN) tasks using only a single perturbation called universal adversarial perturbation (UAP), are a realistic security threat to the practical application of a DNN for medical imaging. Given that computer-based systems are generally operated under a black-box condition in which only input queries are allowed and outputs are accessible, the impact of UAPs seems to be limited because well-used algorithms for generating UAPs are limited to white-box conditions in which adversaries can access model parameters. Nevertheless, we propose a method for generating UAPs using a simple hill-climbing search based only on DNN outputs to demonstrate that UAPs are easily generatable using a relatively small dataset under black-box conditions with representative DNN-based medical image classifications. Black-box UAPs can be used to conduct both nontargeted and targeted attacks. Overall, the black-box UAPs showed high attack success rates (40–90%). The vulnerability of the black-box UAPs was observed in several model architectures. The results indicate that adversaries can also generate UAPs through a simple procedure under the black-box condition to foil or control diagnostic medical imaging systems based on DNNs, and that UAPs are a more serious security threat.

Deep-Neural-Network-Based Modelling of Longitudinal-Lateral Dynamics to Predict the Vehicle States for Autonomous Driving.

Multibody models built in commercial software packages, e.g., ADAMS, can be used for accurate vehicle dynamics, but computational efficiency and numerical stability are very challenging in complex driving environments. These issues can be addressed by using data-driven models, owing to their robust generalization and computational speed. In this study, we develop a deep neural network (DNN) based model to predict longitudinal-lateral dynamics of an autonomous vehicle. Dynamic simulations of the autonomous vehicle are performed based on a semirecursive multibody method for data acquisition. The data are used to train and test the DNN model. The DNN inputs include the torque applied on wheels and the vehicle’s initial speed that imitates a double lane change maneuver. The DNN outputs include the longitudinal driving distance, the lateral driving distance, the final longitudinal velocities, the final lateral velocities, and the yaw angle. The predicted vehicle states based on the DNN model are compared with the multibody model results. The accuracy of the DNN model is investigated in detail in terms of error functions. The DNN model is verified within the framework of a commercial software package CarSim. The results demonstrate that the DNN model predicts accurate vehicle states in real time. It can be used for real-time simulation and preview control in autonomous vehicles for enhanced transportation safety.

Interbeat interval-based sleep staging: work in progress toward real-time implementation

Objective. Cardiac activity changes during sleep enable real-time sleep staging. We developed a deep neural network (DNN) to detect sleep stages using interbeat intervals (IBIs) extracted from electrocardiogram signals. Approach. Data from healthy and apnea subjects were used for training and validation; 2 additional datasets (healthy and sleep disorders subjects) were used for testing. R-peak detection was used to determine IBIs before resampling at 2 Hz; the resulting signal was segmented into 150 s windows (30 s shift). DNN output approximated the probabilities of a window belonging to light, deep, REM, or wake stages. Cohen’s Kappa, accuracy, and sensitivity/specificity per stage were determined, and Kappa was optimized using thresholds on probability ratios for each stage versus light sleep. Main results. Mean (SD) Kappa and accuracy for 4 sleep stages were 0.44 (0.09) and 0.65 (0.07), respectively, in healthy subjects. For 3 sleep stages (light+deep, REM, and wake), Kappa and accuracy were 0.52 (0.12) and 0.76 (0.07), respectively. Algorithm performance on data from subjects with REM behavior disorder or periodic limb movement disorder was significantly worse, with Kappa of 0.24 (0.09) and 0.36 (0.12), respectively. Average processing time by an ARM microprocessor for a 300-sample window was 19.2 ms. Significance. IBIs can be obtained from a variety of cardiac signals, including electrocardiogram, photoplethysmography, and ballistocardiography. The DNN algorithm presented is 3 orders of magnitude smaller compared with state-of-the-art algorithms and was developed to perform real-time, IBI-based sleep staging. With high specificity and moderate sensitivity for deep and REM sleep, small footprint, and causal processing, this algorithm may be used across different platforms to perform real-time sleep staging and direct intervention strategies. Novelty & Significance (92/100 words) This article describes the development and testing of a deep neural network-based algorithm to detect sleep stages using interbeat intervals, which can be obtained from a variety of cardiac signals including photoplethysmography, electrocardiogram, and ballistocardiography. Based on the interbeat intervals identified in electrocardiogram signals, the algorithm architecture included a group of convolution layers and a group of long short-term memory layers. With its small footprint, fast processing time, high specificity and good sensitivity for deep and REM sleep, this algorithm may provide a good option for real-time sleep staging to direct interventions.

A Meta-Model to Predict the Drag Coefficient of a Particle Translating in Viscoelastic Fluids: A Machine Learning Approach.

This study presents a framework based on Machine Learning (ML) models to predict the drag coefficient of a spherical particle translating in viscoelastic fluids. For the purpose of training and testing the ML models, two datasets were generated using direct numerical simulations (DNSs) for the viscoelastic unbounded flow of Oldroyd-B (OB-set containing 12,120 data points) and Giesekus (GI-set containing 4950 data points) fluids past a spherical particle. The kinematic input features were selected to be Reynolds number, , Weissenberg number, , polymeric retardation ratio, , and shear thinning mobility parameter, . The ML models, specifically Random Forest (RF), Deep Neural Network (DNN) and Extreme Gradient Boosting (XGBoost), were all trained, validated, and tested, and their best architecture was obtained using a 10-Fold cross-validation method. All the ML models presented remarkable accuracy on these datasets; however the XGBoost model resulted in the highest and the lowest root mean square error (RMSE) and mean absolute percentage error (MAPE) measures. Additionally, a blind dataset was generated using DNSs, where the input feature coverage was outside the scope of the training set or interpolated within the training sets. The ML models were tested against this blind dataset, to further assess their generalization capability. The DNN model achieved the highest and the lowest RMSE and MAPE measures when inferred on this blind dataset. Finally, we developed a meta-model using stacking technique to ensemble RF, XGBoost and DNN models and output a prediction based on the individual learner’s predictions and a DNN meta-regressor. The meta-model consistently outperformed the individual models on all datasets.

Mitigating Black-Box Adversarial Attacks via Output Noise Perturbation

In black-box adversarial attacks, attackers query the deep neural network (DNN) and use the query results to optimize the adversarial samples iteratively. In this paper, we study the method of adding white noise to the DNN output to mitigate such attacks. One of our unique contributions is a theoretical analysis of gradient signal-to-noise ratio (SNR), which shows the trade-off between the defense noise level and the attack query cost. The attacker’s query count (QC) is derived mathematically as a function of noise standard deviation. This will guide the defender to find the appropriate noise level for mitigating attacks to the desired security level specified by QC and DNN performance loss. Our analysis shows that the added noise is drastically magnified by the small variation of DNN outputs, which makes the reconstructed gradient have an extremely low SNR. Adding slight white noise with a very small standard deviation, e.g., less than 0.01, is enough to increase QC by many orders of magnitude yet without introducing any noticeable classification accuracy reduction. Our experiments demonstrate that this method can effectively mitigate both soft-label and hard-label black-box attacks under realistic QC constraints. We also prove that this method outperforms many other defense methods and is robust to the attacker’s countermeasures.

An ensemble learning method based on deep neural network and group decision making

Ensemble learning (EL) method which has high potential to improve the performance of single image classification model can be constructed in two steps: one is the generation of weak learners; the other is the combination of these learners. In this paper, an ensemble learning method based on deep neural network and group decision making (DNN-GDM-EL) is proposed, which uses deep neural networks (DNNs) to generate individual learners and exploits group decision making (GDM) to combine these learners. DNNs have demonstrated remarkable ability for image classification due to the powerful feature extraction ability. To ensure the diversity and accuracy, many different DNNs are used to generate individual learners. Furthermore, the individual learners are regarded as decision-makers (DMs), the categories are seen as alternatives, and the GDM aims to find an optimal alternative considering various suggestions of DMs. Specifically, a GDM model is established based on Bayesian theory which can reflect the complex relationship among the class of image, prior knowledge and output of DNN, and a GDM method based on TOPSIS is applied to solve this problem. Next, the index matrix consisted of DM's attributes is proposed, and an aggregation method based on 2-additive generalized Shapley AIVIFCA (2AGSAIVIFCA) operator is used to calculate the weights of DMs by fusing these matrixes. Further, state transition algorithm (STA) is applied to obtain the optimal weights of alternative's attributes. The effectiveness and superiority are verified in three public data sets and a real industrial problem by comparing DNN-GDM-EL method with other typical EL methods.