Original Deep Neural Networks Research Articles

KDGAN: Knowledge distillation‐based model copyright protection for secure and communication‐efficient model publishing

AbstractDeep learning‐based models have become ubiquitous across a wide range of applications, including computer vision, natural language processing, and robotics. Despite their efficacy, one of the significant challenges associated with deep neural network (DNN) models is the potential risk of copyright leakage due to the inherent vulnerability of the entire model architecture and the communication burden of the large models during publishing. So far, it is still challenging for us to safeguard the intellectual property rights of these DNN models while reducing the communication time during model publishing. To this end, this paper introduces a novel approach using knowledge distillation techniques aimed at training a surrogate model to stand in for the original DNN model. To be specific, a knowledge distillation generative adversarial network (KDGAN) model is proposed to train a student model capable of achieving remarkable performance levels while simultaneously safeguarding the copyright integrity of the original large teacher model and improving communication efficiency during model publishing. Herein, comprehensive experiments are conducted to showcase the efficacy of model copyright protection, communication‐efficient model publishing, and the superiority of the proposed KDGAN model over other copyright protection mechanisms.

Self-Checking Deep Neural Networks for Anomalies and Adversaries in Deployment

Deep Neural Networks (DNNs) have been widely adopted, yet DNN models are surprisingly unreliable, which raises significant concerns about their use in critical domains. In this work, we propose that runtime DNN mistakes can be quickly detected and properly dealt with <i>in deployment</i>, especially in settings like self-driving vehicles. Just as software engineering (SE) community has developed effective mechanisms and techniques to monitor and check programmed components, our previous work, SelfChecker, is designed to monitor and correct DNN predictions given unintended abnormal test data. SelfChecker triggers an alarm if the decisions given by the internal layer features of the model are inconsistent with the final prediction and provides <i>advice</i> in the form of an alternative prediction. In this paper, we extend SelfChecker to the security domain. Specifically, we describe SelfChecker++, which we designed to target both <i>unintended</i> abnormal test data and <i>intended</i> adversarial samples. Technically, we develop a technique which can transform any runtime inputs triggering alarms into semantically equivalent inputs, then we feed those transformed inputs to the model. Such runtime transformation nullifies any intended crafted samples, making the model immune to adversarial attacks that craft adversarial samples. We evaluated the alarm accuracy of SelfChecker++ on three DNN models and four popular image datasets, and found that SelfChecker++ triggers correct alarms on 63.10% of wrong DNN predictions, and triggers false alarms on 5.77% of correct DNN predictions. We also evaluated the effectiveness of SelfChecker++ in detecting adversarial examples and found it detects on average 70.09% of such samples with advice accuracy that is 20.89% higher than the original DNN model and 18.37% higher than SelfChecker.

Augmented deep neural network architecture for assessing damage severity in 3D concrete buildings under temperature fluctuations based on K-means optimization

This research paper introduces a novel approach for forecasting the severity of damage in column elements within a complex 3D concrete structure. The paper presents two innovative methods that have not been previously proposed. Firstly, a new equation is suggested to enhance the sensitivity of the dataset used for training and learning in machine learning techniques. This equation incorporates two crucial dynamic characteristics, namely frequencies and mode shapes. The second method involves a new technique for optimizing the architecture of a deep neural network (DNN) using K-means optimizer (KO), referred to KODNN. By employing KO, the optimal values for the DNN architecture and learning rate are determined. To assess the effectiveness of the proposed method, KODNN is employed to develop a model for predicting the compressive strength of concrete using benchmark datasets. A comparison is made between the obtained results from this example and those obtained using the original DNN model to demonstrate the performance improvement achieved. Finally, KODNN is utilized to predict the severity of damage in elements of a 3D concrete building under various temperature conditions, considering two specific damage cases. The results indicate that the proposed method achieves a high level of accuracy and reliability.

Detection of Cancer Recurrence Using Systemic Inflammatory Markers and Machine Learning after Concurrent Chemoradiotherapy for Head and Neck Cancers.

Pretreatment values of the neutrophil-to-lymphocyte ratio (NLR) and the platelet-to-lymphocyte ratio (PLR) are well-established prognosticators in various cancers, including head and neck cancers. However, there are no studies on whether temporal changes in the NLR and PLR values after treatment are related to the development of recurrence. Therefore, in this study, we aimed to develop a deep neural network (DNN) model to discern cancer recurrence from temporal NLR and PLR values during follow-up after concurrent chemoradiotherapy (CCRT) and to evaluate the model's performance compared with conventional machine learning (ML) models. Along with conventional ML models such as logistic regression (LR), random forest (RF), and gradient boosting (GB), the DNN model to discern recurrences was trained using a dataset of 778 consecutive patients with primary head and neck cancers who received CCRT. There were 16 input features used, including 12 laboratory values related to the NLR and the PLR. Along with the original training dataset (N = 778), data were augmented to split the training dataset (N = 900). The model performance was measured using ROC-AUC and PR-AUC values. External validation was performed using a dataset of 173 patients from an unrelated external institution. The ROC-AUC and PR-AUC values of the DNN model were 0.828 ± 0.032 and 0.663 ± 0.069, respectively, in the original training dataset, which were higher than the ROC-AUC and PR-AUC values of the LR, RF, and GB models in the original training dataset. With the recursive feature elimination (RFE) algorithm, five input features were selected. The ROC-AUC and PR-AUC values of the DNN-RFE model were higher than those of the original DNN model (0.883 ± 0.027 and 0.778 ± 0.042, respectively). The ROC-AUC and PR-AUC values of the DNN-RFE model trained with a split dataset were 0.889 ± 0.032 and 0.771 ± 0.044, respectively. In the external validation, the ROC-AUC values of the DNN-RFE model trained with the original dataset and the same model trained with the split dataset were 0.710 and 0.784, respectively. The DNN model with feature selection using the RFE algorithm showed the best performance among the ML models to discern a recurrence after CCRT in patients with head and neck cancers. Data augmentation by splitting training data was helpful for model performance. The performance of the DNN-RFE model was also validated with an external dataset.

Knowledge Sharing in AI Services: A Market-Based Approach

Today’s deep neural networks (DNNs) are very accurate when trained on a large amount of data. However, suitable input might not be available or may require extensive data collection. Data sharing is one option to address these issues, but it is generally impractical because of privacy concerns or due to the problematic process of finding a sharing agreement. Instead, this work considers knowledge sharing by first exchanging the weights of pretrained DNNs and then applying transfer learning (TL). Specifically, it addresses the economics of knowledge sharing in AI services by taking a market-based approach. In detail, a model based on Fisher’s market is devised for optimal knowledge sharing, defined as the gain in inference accuracy from exchanging DNN weights. The proposed approach is shown to reach a market equilibrium and to satisfy important economic properties, including Pareto optimality. A technique for weight fusion is also introduced to merge acquired knowledge with the existing one. Finally, an extensive evaluation is conducted in a distributed intelligence scenario. The obtained results show that the proposed solution is efficient and that weight fusion with TL significantly increases inference accuracy compared to the original DNN, without the overhead of federated learning.

Distributed Assignment With Load Balancing for DNN Inference at the Edge

Inference carried out on pretrained deep neural networks (DNNs) is particularly effective as it does not require retraining and entails no loss in accuracy. Unfortunately, resource-constrained devices such as those in the Internet of Things may need to offload the related computation to more powerful servers, particularly, at the network edge. However, edge servers have limited resources compared to those in the cloud; therefore, inference offloading generally requires dividing the original DNN into different pieces that are then assigned to multiple edge servers. Related approaches in the state-of-the-art either make strong assumptions on the system model or fail to provide strict performance guarantees. This article specifically addresses these limitations by applying distributed assignment to DNN inference at the edge. In particular, it devises a detailed model of DNN-based inference, suitable for realistic scenarios involving edge computing. Optimal inference offloading with load balancing is also defined as a multiple assignment problem that maximizes proportional fairness. Moreover, a distributed algorithm for DNN inference offloading is introduced to solve such a problem in polynomial time with strong optimality guarantees. Finally, extensive simulations employing different data sets and DNN architectures establish that the proposed solution significantly improves upon the state-of-the-art in terms of inference time (1.14 to 2.62 times faster), load balance (with Jain’s fairness index of 0.9), and convergence (one order of magnitude less iterations).

DiverGet: a Search-Based Software Testing approach for Deep Neural Network Quantization assessment

Quantization is one of the most applied Deep Neural Network (DNN) compression strategies, when deploying a trained DNN model on an embedded system or a cell phone. This is owing to its simplicity and adaptability to a wide range of applications and circumstances, as opposed to specific Artificial Intelligence (AI) accelerators and compilers that are often designed only for certain specific hardware (e.g., Google Coral Edge TPU). With the growing demand for quantization, ensuring the reliability of this strategy is becoming a critical challenge. Traditional testing methods, which gather more and more genuine data for better assessment, are often not practical because of the large size of the input space and the high similarity between the original DNN and its quantized counterpart. As a result, advanced assessment strategies have become of paramount importance. In this paper, we present DiverGet, a search-based testing framework for quantization assessment. DiverGet defines a space of metamorphic relations that simulate naturally-occurring distortions on the inputs. Then, it optimally explores these relations to reveal the disagreements among DNNs of different arithmetic precision. We evaluate the performance of DiverGet on state-of-the-art DNNs applied to hyperspectral remote sensing images. We chose the remote sensing DNNs as they're being increasingly deployed at the edge (e.g., high-lift drones) in critical domains like climate change research and astronomy. Our results show that DiverGet successfully challenges the robustness of established quantization techniques against naturally-occurring shifted data, and outperforms its most recent concurrent, DiffChaser, with a success rate that is (on average) four times higher.

DNN Patching: Progressive Fixing and Augmenting the Functionalities of DNNs for Autonomous Vehicles

Autonomous vehicles have become one of the most-awaited technologies of the century. In the recent years the advances in the area of deep learning and artificial intelligence enabled rapid development of the autonomous driving sector. Some car companies have already deployed autonomous driving software into their vehicles and millions of customers are using them. Although the autonomous driving modules are typically extensively tested prior to its release on the market, the customers can either still encounter certain previously overlooked failure cases or identify missing functionalities of the model. In this letter we focus on developing the methodology for repairing the deep neural network (DNN) steering the self-driving vehicle in response to these two scenarios via patching such that the following conditions are satisfied: i) the patch is added on the top of the original DNN to extend or fix its functionality for the particular failure case, ii) the patched model keeps the original functionalities of the DNN unchanged, and iii) the size of the patch is significantly smaller than the size of the original DNN. Our approach avoids training and evaluating the network from the scratch to improve or fix it, effectively enabling fast response of the autonomous vehicle manufacturers to the customer needs. We demonstrate the plausibility of the proposed approach on the task of autonomous driving in the Carla simulator A. Dosovitskiy <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">et al</i> .

Partially pruned DNN coupled with parallel Monte-Carlo algorithm for path loss prediction in underwater wireless optical channels.

In this paper, we propose a new approach to solve the radiative transfer equation (RTE) and determine the path loss for line-of-sight (LOS) propagation with laser diode sources in underwater wireless optical channels, which severely suffers from attenuation due to inevitable absorption and scattering. The scheme is based on an effective combination of Monte-Carlo (MC) simulation employed for dataset generation and a partially pruned deep neural network (PPDNN) utilized to predict the received optical power. First, a parallel MC algorithm is newly introduced and applied to speed up the dataset-generation process. Compared with the conventional single-step MC, the dataset-generation time of the parallel MC can be reduced by at least 95%. Meanwhile, a deep neural network (DNN) is partially pruned to acquire a compact structure and adopted to predict the path loss in three typical water types. The simulation results yield that the mean square errors (MSEs) between the predictive and the reference ones are all lower than 0.2, while the sparsity of the original DNN's weights can be appropriately increased to 0.9, 0.7, and 0.5 for clear water, coastal water, and harbor water, respectively. Finally, the occupied storage space of the original DNN can be dramatically compressed by at least 40% with a small performance penalty. In view of this, the received optical power under certain parameters could be instantly obtained by employing the proposed PPDNN, which can effectively help design underwater wireless optical communication systems in future work.

A DNN Fingerprint for Non-Repudiable Model Ownership Identification and Piracy Detection

A high-performance Deep Neural Network (DNN) model is a valuable intellectual property (IP) since designing and training such a model from scratch is very costly. Model transfer learning, compression and retraining are commonly used by pirates to evade detection or even redeploy the pirated models for new applications without compromising performance. This paper presents a novel non-intrusive DNN IP fingerprinting method that can detect pirated models and provide a non-repudiable and irrevocable ownership proof simultaneously. The fingerprint is derived from projecting a subset of front-layer weights onto a model owner identity defined random space to enable a distinguisher to differentiate pirated models that are used in the same application or retrained for a different task from originally designed DNN models. The proposed method generates compact and irrevocable fingerprints against model IP misappropriation and ownership fraud. It requires no retraining and makes no modification to the original model. The proposed fingerprinting method is evaluated on nine original DNN models trained on CIFAR-10, CIFAR-100, and ImageNet-10. It is demonstrated to have the highest discriminative power among existing fingerprinting methods in detecting pirated models deployed for the same and different applications, and fraudulent model IP ownership claims.

Hierarchical Attention-Based Machine Learning Model for Radiation Prediction of WB-BGA Package

Rapid increase in operating frequency of integrated chips and intricacy of electronic packages outpaces the ability of conventional methods in coping with the growing complexity of electromagnetic interference (EMI) issues. To address it, several machine learning (ML) methods––deep neural network (DNN), convolutional neural network, support vector regression, <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">K</i> -nearest neighbor, and linear regression are constructed to acquire the best ML model to accurately and rapidly predict the maximum 3-m radiated electric field of a wire-bond ball grid array package. The key hyperparameters of different ML models are tuned respectively to attain the least prediction error for each model. Among the optimized ML models, the prediction accuracy of the DNN model is the highest. In this article, a hierarchical attention-based DNN model is proposed and discussed in depth to reduce the number of training datasets, and identify the structural parameters with large contributions to radiation prediction. These structural parameters with contributions can guide the packaging design. The DNN model with attention-weight input requires fewer training datasets than the original DNN model. Furthermore, the experimental measurement for EMI radiation of a test package board is implemented, and the far-field results show the effectiveness and feasibility of the DNN model.

Enhanced DNNs for malware classification with GAN-based adversarial training

Deep learning based malware classification gains momentum recently. However, deep learning models are vulnerable to adversarial perturbation attacks especially when applied in network security application. Deep neural network (DNN)-based malware classifiers by eating the whole bit sequences are also vulnerable despite their satisfactory performance and less feature-engineering job. Therefore, this paper proposes a DNN-based malware classifier on the raw bit sequences of programs in Windows. We then propose two adversarial attacks targeting our trained DNNs to generate adversarial malware. A defensive mechanism is proposed by treating perturbations as noise added on bit sequences. In our defensive mechanism, a generative adversary network (GAN)-based model is designed to filter out the perturbation noise and those that with the highest probability to fool the target DNNs are chosen for adversarial training. The experiments show that GAN with filter-based model produced the highest quality adversarial samples with medium cost. The evasion ratio under GAN with filter-based model is as high as 50.64% on average. While incorporating GAN-based adversarial samples into training, the enhanced DNN achieves satisfactory with 90.20% accuracy while the evasion ratio is below 9.47%. GAN helps in secure the DNN-based malware classifier with negligible performance degradation when compared with the original DNN. The evasion ratio is remarkably minimized when faced with powerful adversarial attacks, including $${\textit{FGSM}}^r$$ and $${\textit{FGSM}}^k$$ .

Reducing deep learning network structure through variable reduction methods in crop modeling

Crop models are widely used to predict plant growth, water input requirements, and yield. However, existing models are very complex and require hundreds of variables to perform accurately. Due to these shortcomings, large-scale applications of crop models are limited. In order to address these limitations, reliable crop models were developed using a deep neural network (DNN) – a new approach for predicting crop yields. In addition, the number of required input variables was reduced using three common variable selection techniques: namely Bayesian variable selection, Spearman's rank correlation, and Principal Component Analysis Feature Extraction. The reduced-variable DNN models were capable of estimating future crop yields for 10,000,000 different weather and irrigation scenarios while maintaining comparable accuracy levels to the original model that used all input variables. To establish clear superiority of the methodology, the results were also compared with a very recent feature selection algorithm called min-redundancy max-relevance (mRMR). The results of this study showed that the Bayesian variable selection was the best method for achieving the aforementioned goals. Specifically, the final Bayesian-based DNN model with a structure of 10 neurons in 5 layers performed very similarly (78.6% accuracy) to the original DNN crop model with 400 neurons in 10 layers, even though the size of the neural network was reduced by 80-fold. This effort can help promote sustainable agricultural intensifications through the large-scale application of crop models.

Using a Deep Neural Network and Transfer Learning to Bridge Scales for Seismic Phase Picking

AbstractThe important task of tracking seismic activity requires both sensitive detection and accurate earthquake location. Approximate earthquake locations can be estimated promptly and automatically; however, accurate locations depend on precise seismic phase picking, which is a laborious and time‐consuming task. We adapted a deep neural network (DNN) phase picker trained on local seismic data to mesoscale hydraulic fracturing experiments. We designed a novel workflow, transfer learning‐aided double‐difference tomography, to overcome the 3 orders of magnitude difference in both spatial and temporal scales between our data and data used to train the original DNN. Only 3,500 seismograms (0.45% of the original DNN data) were needed to retrain the original DNN model successfully. The phase picks obtained with transfer‐learned model are at least as accurate as the analyst's and lead to improved event locations. Moreover, the effort required for picking once the DNN is trained is a small fraction of the analyst's.

Block-term tensor neural networks

Deep neural networks (DNNs) have achieved outstanding performance in a wide range of applications, e.g., image classification, natural language processing, etc. Despite the good performance, the huge number of parameters in DNNs brings challenges to efficient training of DNNs and also their deployment in low-end devices with limited computing resources. In this paper, we explore the correlations in the weight matrices, and approximate the weight matrices with the low-rank block-term tensors. We name the new corresponding structure as block-term tensor layers (BT-layers), which can be easily adapted to neural network models, such as CNNs and RNNs. In particular, the inputs and the outputs in BT-layers are reshaped into low-dimensional high-order tensors with a similar or improved representation power. Sufficient experiments have demonstrated that BT-layers in CNNs and RNNs can achieve a very large compression ratio on the number of parameters while preserving or improving the representation power of the original DNNs.

Improving deep neural network performance by integrating kernelized Min-Max objective

Deep neural networks (DNN), such as convolutional neural networks (CNN) have been widely used for object recognition. However, they are usually unable to ensure the required intra-class compactness and inter-class separability in the kernel space. These are known to be important in pattern recognition for achieving both robustness and accuracy. In this paper, we propose to integrate a kernelized Min-Max objective in the DNN training in order to explicitly enforce both kernelized within-class compactness and between-class margin. The involved kernel space is implicitly mapped from the feature space associated with a certain upper layer of DNN by exploiting a kernel trick, while the Min-Max objective in this space is interpolated with the original DNN loss function and finally optimized in the training phase. With a very small additional computation cost, the proposed strategy can be easily integrated in different DNN models without changing any other part of the original model. The comparative recognition accuracy of the proposed method is evaluated with multiple DNN models (including shallow CNN, deep CNN and deep residual neural network models) on two benchmark datasets: CIFAR-10 and CIFAR-100. Extensive experimental results demonstrate that the integration of kernelized Min-Max objective in the training of DNN models can achieve better results compared to state-of-the-art models, without incurring additional model complexity.

DeepVID: Deep Visual Interpretation and Diagnosis for Image Classifiers via Knowledge Distillation.

Deep Neural Networks (DNNs) have been extensively used in multiple disciplines due to their superior performance. However, in most cases, DNNs are considered as black-boxes and the interpretation of their internal working mechanism is usually challenging. Given that model trust is often built on the understanding of how a model works, the interpretation of DNNs becomes more important, especially in safety-critical applications (e.g., medical diagnosis, autonomous driving). In this paper, we propose DeepVID, a Deep learning approach to Visually Interpret and Diagnose DNN models, especially image classifiers. In detail, we train a small locally-faithful model to mimic the behavior of an original cumbersome DNN around a particular data instance of interest, and the local model is sufficiently simple such that it can be visually interpreted (e.g., a linear model). Knowledge distillation is used to transfer the knowledge from the cumbersome DNN to the small model, and a deep generative model (i.e., variational auto-encoder) is used to generate neighbors around the instance of interest. Those neighbors, which come with small feature variances and semantic meanings, can effectively probe the DNN's behaviors around the interested instance and help the small model to learn those behaviors. Through comprehensive evaluations, as well as case studies conducted together with deep learning experts, we validate the effectiveness of DeepVID.

New Improved Training for Deep Neural Networks Based on Intrusion Detection System

All Network intrusion detection is designed for detecting, preventing, and repelling network security breaches and it has become an urgent issue. Maintaining a safe and secure network requires an efficient and flexible solution called an intrusion detection system. This paper reports an advanced intrusion detection method created with a deep learning approach. Evolutionary operators can reduce the probability of stagnation in local solutions due to high local optima avoidance and have thus superseded conventional training algorithms, such as back propagation (BP). Combining a deep neural network (DNN) and an evolutionary algorithm (EA) may solve problems or outperform DNN in solving existing problems. We develop a hybrid training method that combines simulated annealing (SA) and BP to improve the performance of DNN (SABP-DNN). The NSL-KDD dataset is used to verify the accuracy and efficiency of the proposed method. The proposed method is also compared with the original DNN based on PB (PB-DNN) and DNN based on SA (SA-DNN). We confirm that the proposed method presents a strong potential to become an alternative solution to IDS through experiments and comparisons with existing methods.

A unified approach to transfer learning of deep neural networks with applications to speaker adaptation in automatic speech recognition

In this paper, we present a unified approach to transfer learning of deep neural networks (DNNs) to address performance degradation issues caused by a potential acoustic mismatch between the training and testing conditions due to inter-speaker variability in state-of-the-art connectionist (a.k.a., hybrid) automatic speech recognition (ASR) systems. Different schemes to transfer knowledge of deep neural networks related to speaker adaptation can be developed with ease under such a unifying concept as demonstrated in the three frameworks investigated in this study. In the first solution, knowledge is transferred between homogeneous domains, namely the source and the target domains. Moreover the transfer takes place in a sequential manner from the target to the source speaker to boost the ASR accuracy on spoken utterances from a surprise target speaker. In the second solution, a multi-task approach is adopted to adjust the connectionist parameters to improve the ASR system performance on the target speaker. Knowledge is transferred simultaneously among heterogeneous tasks, and that is achieved by adding one or more smaller auxiliary output layers to the original DNN structure. In the third solution, DNN output classes are organised into a hierarchical structure in order to adjust the connectionist parameters and close the gap between training and testing conditions by transferring prior knowledge from the root node to the leaves in a structural maximum a posteriori fashion. Through a series of experiments on the Wall Street Journal (WSJ) speech recognition task, we show that the proposed solutions result in consistent and statistically significant word error rate reductions. Most importantly, we show that transfer learning is an enabling technology for speaker adaptation, since it outperforms both the transformation-based adaptation algorithms usually adapted in the speech community, and the multi-condition training (MCT) schemes, which is a data combination methods often adopted to cover more acoustic variabilities in speech when data from the source and target domains are both available at the training time. Finally, experimental evidence demonstrates that all proposed solutions are robust to negative transfer even when only a single sentence from the target speaker is available.