Layers Of Deep Neural Network Research Articles

A bidirectional DNN partition mechanism for efficient pipeline parallel training in cloud

Recently, deep neural networks (DNNs) have shown great promise in many fields while their parameter sizes are rapidly expanding. To break through the computation and memory limitation of a single machine, pipeline model parallelism is proposed for large-scale DNN training by fully utilizing the computation and storage power of the distributed cluster. Cloud data centers can also provide sufficient computing, storage and bandwidth resources. However, most existing approaches apply layer-wise partitioning, which is difficult to obtain an even model partition result because of the large computational overhead discrepancy between DNN layers, resulting in degraded efficiency. To tackle this issue, we propose “Bi-Partition”, a novel partitioning method based on bidirectional partitioning for forward propagation (FP) and backward propagation (BP), which improves the efficiency of the pipeline model parallelism system. By deliberated designing distinct cut positions for FP and BP of DNN training, workers in the pipeline get nearly equal computational loads, and the balanced pipeline fully utilizes the computing resources. Experiments on various DNN models and datasets validate the efficiency of our mechanism, e.g., the training efficiency achieving up to 1.9times faster than the state-of-the-art method PipeDream.

An Advanced Decision Tree-Based Deep Neural Network in Nonlinear Data Classification

Deep neural networks (DNNs), the integration of neural networks (NNs) and deep learning (DL), have proven highly efficient in executing numerous complex tasks, such as data and image classification. Because the multilayer in a nonlinearly separable data structure is not transparent, it is critical to develop a specific data classification model from a new and unexpected dataset. In this paper, we propose a novel approach using the concepts of DNN and decision tree (DT) for classifying nonlinear data. We first developed a decision tree-based neural network (DTBNN) model. Next, we extend our model to a decision tree-based deep neural network (DTBDNN), in which the multiple hidden layers in DNN are utilized. Using DNN, the DTBDNN model achieved higher accuracy compared to the related and relevant approaches. Our proposal achieves the optimal trainable weights and bias to build an efficient model for nonlinear data classification by combining the benefits of DT and NN. By conducting in-depth performance evaluations, we demonstrate the effectiveness and feasibility of the proposal by achieving good accuracy over different datasets.

Few-Shot Remote Sensing Image Scene Classification Based on Metric Learning and Local Descriptors

Scene classification is a critical technology to solve the challenges of image search and image recognition. It has become an indispensable and challenging research topic in the field of remote sensing. At present, most scene classifications are solved by deep neural networks. However, existing methods require large-scale training samples and are not suitable for actual scenarios with only a few samples. For this reason, a framework based on metric learning and local descriptors (MLLD) is proposed to enhance the classification effect of remote sensing scenes on the basis of few-shot. Specifically, MLLD adopts task-level training that is carried out through meta-learning, and meta-knowledge is learned to improve the model’s ability to recognize different categories. Moreover, Manifold Mixup is introduced by MLLD as a feature processor for the hidden layer of deep neural networks to increase the low confidence space for smoother decision boundaries and simpler hidden layer representations. In the end, a learnable metric is introduced; the nearest category of the image is matched by measuring the similarity of local descriptors. Experiments are conducted on three public datasets: UC Merced, WHU-RS19, and NWPU-RESISC45. Experimental results show that the proposed scene classification method can achieve the most advanced results on limited datasets.

Accelerating Deep Learning Inference via Model Parallelism and Partial Computation Offloading

With the rapid development of Internet-of-Things (IoT) and the explosive advance of deep learning, there is an urgent need to enable deep learning inference on IoT devices in Mobile Edge Computing (MEC). To address the computation limitation of IoT devices in processing complex Deep Neural Networks (DNNs), computation offloading is proposed as a promising approach. Recently, partial computation offloading is developed to dynamically adjust task assignment strategy in different channel conditions for better performance. In this paper, we take advantage of intrinsic DNN computation characteristics and propose a novel Fused-Layer-based (FL-based) DNN model parallelism method to accelerate inference. The key idea is that a DNN layer can be converted to several smaller layers in order to increase partial computation offloading flexibility, and thus further create the better computation offloading solution. However, there is a trade-off between computation offloading flexibility as well as model parallelism overhead. Then, we investigate the optimal DNN model parallelism and the corresponding scheduling and offloading strategies in partial computation offloading. In particular, we propose a Particle Swarm Optimization with Minimizing Waiting (PSOMW) method, which explores and updates the FL strategy, path scheduling strategy, and path offloading strategy to reduce time complexity and avoid invalid solutions. Finally, we validate the effectiveness of the proposed method in commonly used DNNs. The results show that the proposed method can reduce the DNN inference time by an average of 12.75 times compared to the legacy No FL (NFL) algorithm, and is very close to the optimal solution achieved by the Brute Force (BF) algorithm with the difference of less than 0.04%.

SENTIMENT ANALYSIS ON COVID-19 RELATED TWEETS

1. Introduction COVID-19[1] is a pandemic going on during the current time. COVID-19 poses a new challenge to humankind. Meanwhile, It is essential to know what the majority of people thinking about this pandemic. Sentiment analysis[2] is a widespread technique applied in field of marketing, customer feedback and consumer research. This project performs sentiment analysis on COVID-19 related tweets[3] and getting some higher idea about people's sentiment toward pandemic. After performing sentimental analysis on COVID-19 related tweets, this project also draws some interesting findings and conclusions in the last section. Performing sentiment analysis on social media data like tweets, Facebook comments are challenging task for some reasons .1) Due to high volume, data may or may not be in csv, excel ,or other text-rich formats. Performing operations on data needs 2) To draw human sentiment from raw text itself complex task for a machine. However, recently there are advanced deep learning based methods available for performing such a complex task. For example, Answering the question by chatbot or auto-compilation of a statement in email.[4]. This project takes NLP(Natural Language Processing) approach with the help of Google's pre-trained deep neural network called BERT[5] for identifying sentiment from raw text data. The layers of deep neural network implemented in PyTorch[6].

Multiobjective End-to-End Design Space Exploration of Parameterized DNN Accelerators

Deep neural network (DNN) hardware accelerators enable the execution of complex DNN inferences on resource-constrained IoT devices. Inference performance and energy figures depend on how the DNN layers are mapped into the accelerator and how the architecture of the accelerator fits the variety of layers’ shapes of the actual DNN. The mapping determines the execution order of the operations, both temporally and spatially. Thus, selecting the best mapping that allows fitting the DNN model to the specific accelerator is of paramount importance to meet the strong constraints imposed by resource-scarce IoT platforms. Although several mapping space exploration techniques have been proposed in the literature, they are focused on determining the best mapping for a given layer, for a given architecture, and for optimizing a single objective. This article largely extends the scope of the exploration by considering the huge design space spanned by mapping related and architectural parameters, considering all the layers of the DNN, and optimizing multiple objectives simultaneously. We present EPOCA, end-to-end Pareto optimization of DNN accelerators, whose goal is to determine the accelerator’s architecture and the mapping for each layer that optimizes end-to-end and in a multiobjective fashion a set of conflicting design criteria. We assess EPOCA on different DNN models on a parameterized hardware accelerator designed for IoT applications and compare them with a state-of-the-art mapping space explorer, considering the area, inference latency, and inference energy as optimization metrics. We show that the set of Pareto solutions found by EPOCA provides the designer with a range of choices from which to select the best tradeoff with respect to the specific application.

L-SecNet: Towards Secure and Lightweight Deep Neural Network Inference

The advances in machine learning technology has promoted its great potential for deep neural network (DNN) inference powered applications of Internet of Things (IoT), such as facial verification cameras and speech recognition assistants. The current deployment of these applications also raises serious privacy concerns, especially when sensitive individual information is accessed easily by various IoT devices. Fortunately, the cryptography-based solutions are able to execute secure inference without infringing the user’s raw data and the model owner’s proprietary model. However, existing works suffer from impractically high latency and low accuracy, stemming primarily from the evaluations of the non-linear layers in DNN. In this paper, we propose L-SecNet, a lightweight secure neural network inference system that provides efficient inference services without sacrificing accuracy and privacy. Specifically, to reduce latency caused by comparison operations in non-linear layers, we subtly combine additive secret sharing and multiplicative secret sharing to design a lightweight secure comparison protocol. Further, we approximate the commonly used and time-consuming activation functions (including Sigmoid and Tanh functions) with the non-linear sin function instead of the linear polynomial approximation functions in the existing works. In order to maintain low running latency while meeting the requirement of high accuracy, a secure and lightweight protocol for a sin function is proposed. Our theoretical analysis and empirical experiments evaluate the security and efficiency of the L-SecNet system. Compared with the state-of-the-art works, L-SecNet saves up to about 80 times bandwidth and about 53 times runtime.

Normalization effects on deep neural networks

We study the effect of normalization on the layers of deep neural networks of feed-forward type. A given layer $ i $ with $ N_{i} $ hidden units is allowed to be normalized by $ 1/N_{i}^{\gamma_{i}} $ with $ \gamma_{i}\in[1/2,1] $ and we study the effect of the choice of the $ \gamma_{i} $ on the statistical behavior of the neural network's output (such as variance) as well as on the test accuracy on the MNIST data set. We find that in terms of variance of the neural network's output and test accuracy the best choice is to choose the $ \gamma_{i} $'s to be equal to one, which is the mean-field scaling. We also find that this is particularly true for the outer layer, in that the neural network's behavior is more sensitive in the scaling of the outer layer as opposed to the scaling of the inner layers. The mechanism for the mathematical analysis is an asymptotic expansion for the neural network's output. An important practical consequence of the analysis is that it provides a systematic and mathematically informed way to choose the learning rate hyperparameters. Such a choice guarantees that the neural network behaves in a statistically robust way as the $ N_i $ grow to infinity.

Performance Improvement on Traditional Chinese Task-Oriented Dialogue Systems With Reinforcement Learning and Regularized Dropout Technique

The development of conversational voice assistant applications has been in full swing around the world. This paper aims to develop traditional Chinese multi-domain task-oriented dialogue (TOD) systems. It is typically implemented using pipeline approach, where submodules are optimized independently, resulting in inconsistencies with each other. Instead, this paper implements end-to-end multi-domain TOD models using pre-trained deep neural networks (DNNs). This allows us to integrate all the submodules into one single DNN model to solve the inconsistencies. Data shortages are common in conversational natural language processing (NLP) tasks using DNN models. In this regard, dropout regularization has been widely used to improve overfitting caused by insufficient training dataset. However, the randomness it introduces leads to non-negligible discrepancies between training and inference. On the other hand, pre-trained language models have successfully provided effective regularization for NLP tasks. An inherent disadvantage is that fine-tuning the pre-trained language model suffers from exposure bias and loss-evaluation mismatch. To this end, we propose a reinforcement learning (RL) approach to address both issues. Furthermore, we adopt a method called regularized dropout (R-Drop) to improve the inconsistency in dropout layers of DNNs. Experimental results show that both our proposed RL approach and the R-Drop technique can significantly improve the joint target accuracy (JGA) score and combined score of traditional Chinese TOD system in tasks of dialogue state tracking (DST) and end-to-end sentence prediction, respectively.

A Layer Selection Optimizer for Communication-Efficient Decentralized Federated Deep Learning

Federated Learning (FL) systems orchestrate the cooperative training of a shared machine learning (ML) model across connected devices. Recently, decentralized FL architectures driven by consensus have been proposed to enable the devices to share and aggregate the ML parameters via direct sidelink communications. The approach has the advantage of promoting the federation among the agents even in the absence of a server, but may require an intensive use of communication resources compared to vanilla FL methods. This paper proposes a communication-efficient design of consensus-driven FL optimized for training of deep neural networks (DNNs). Devices independently select fragments of the DNN to be shared with neighbors on each training round. Selection is based on a local optimizer that trades model quality improvement with sidelink communication resource savings. The proposed technique is validated on a vehicular cooperative sensing use case characterized by challenging real-world datasets and complex DNNs typically employed in autonomous driving with up to 40 trainable layers. The impact of layer selection is analyzed under different distributed coordination configurations. The results show that it is better to prioritize the DNN layers possessing few parameters, while the selection policy should optimally balance gradient sorting and randomization. Latency, accuracy and communication tradeoffs are analyzed in detail targeting sustainable federation policies.

Joint Optimization of DNN Partition and Continuous Task Scheduling for Digital Twin-Aided MEC Network With Deep Reinforcement Learning

Nowadays, mobile application services face the challenges of high speed, low latency and high reliability. The combination of digital twin (DT) technology and mobile edge computing (MEC) network can effectively solve these challenges. DT technology can help MEC network monitor and predict the network states. In this paper, we propose a DT-aided MEC network scenario with deep neural network (DNN) inference as the computing task of end devices (EDs). ED can offload part of DNN layers to MEC server. To allocate communication resources, we propose an algorithm based on asynchronous advantage actor-critic (A3C), which manages the transmission power and channel selection of EDs. Since DNN inference is continuous in real scenes, we consider the continuous DNN inference tasks. We convert the DNN optimal partition point solving problem to a min st-cut problem, and propose a graph theory based DNN optimal partition point solving algorithm to minimize the inference latency. Simulation results show that the proposed algorithm can effectively reduce the inference latency. Compared with actor-critic (AC) and deep Q network (DQN), the proposed algorithm has faster convergence speed and better convergence value. Compared with the traditional one-time DNN model partition algorithm, the proposed algorithm is more suitable for DNN continuous task arrival scenario.

Effect of DNN Approximation for Channel Estimation and Signal Detection on OFDM Applications

This paper offers a deep learning approximation to realize channel estimation and signal detection that creates the main communication structure skeleton for the orthogonal frequency-division multiplexing (OFDM) system known as an efficient modulation type on 5G. This letter offers an application of deep learning to handle the wireless OFDM channels' end-to-end conduct. First, channel state information (CSI) is predicted explicitly that differs from existing OFDM receivers, then detected the transmitted symbols utilizing the predicted CSI. In the end, CSI is predicted by the suggested deep learning approximation indirectly and transmitted symbols are directly recovered. The structure of the designed receiver occurs of a layer of DNN and soft decisions, which resolves the issues channel estimation error, time delay, and limitation of decoding between users in classic detection techniques. In the simulation results, it is observed that the receiver has powerful stability on the power distribution of user, not only convenient for the linear channel, but also for nonlinear channel when enhancement the number of users, also detection can be well on the receiver. Generally, the efficiency of the modulation system decreases with the features of the multipath channel utilized for transmission. Channel estimation and detection of symbols utilize to reduce the impacts of the channel, which needs high computation and bandwidth conventionally. This paper is used deep neural networks (DNN) for detecting the signal, in this way much effort in detecting the channel is prevented. The proposed method saves priceless bandwidth via used CP in OFDM with a big increase in SNR.

AccHashtag : Accelerated Hashing for Detecting Fault-Injection Attacks on Embedded Neural Networks

We propose AccHashtag , the first framework for high-accuracy detection of fault-injection attacks on Deep Neural Networks (DNNs) with provable bounds on detection performance. Recent literature in fault-injection attacks shows the severe DNN accuracy degradation caused by bit flips. In this scenario, the attacker changes a few DNN weight bits during execution by injecting faults to the dynamic random-access memory (DRAM). To detect bit flips, AccHashtag extracts a unique signature from the benign DNN prior to deployment. The signature is used to validate the model’s integrity and verify the inference output on the fly. We propose a novel sensitivity analysis that identifies the most vulnerable DNN layers to the fault-injection attack. The DNN signature is constructed by encoding the weights in vulnerable layers using a low-collision hash function. During DNN inference, new hashes are extracted from the target layers and compared against the ground-truth signatures. AccHashtag incorporates a lightweight methodology that allows for real-time fault detection on embedded platforms. We devise a specialized compute core for AccHashtag on field-programmable gate arrays (FPGAs) to facilitate online hash generation in parallel to DNN execution. Extensive evaluations with the state-of-the-art bit-flip attack on various DNNs demonstrate the competitive advantage of AccHashtag in terms of both attack detection and execution overhead.

Layerwise Security Protection for Deep Neural Networks in Industrial Cyber Physical Systems

Although deep neural networks (DNNs) have been increasingly applied in industrial cyber physical systems (ICPSs), they are vulnerable to security attacks due to the tight interaction between cyber elements and physical elements. In this article, we aim to protect the core IP of DNNs, i.e., the model weights, against security attacks. Different from conventional approaches, a layerwise protection framework is proposed to ensure the confidentiality of DNN model weights during the inference procedure such that the security quality is maximized, while satisfying the latency constraint of the DNN task. Based on the layerwise execution characteristics of DNN tasks, the encrypted layer-related weights are decrypted and fed to the next layer of DNN in plaintext. CPU-field programmable gate array (FPGA) coscheduling is considered to accelerate the execution of confidentiality protection, where CPU is utilized to conduct the decryption of weights and FPGA is used to perform the layer execution of DNN. Considering to provide optimal confidential protection for each layer, the problem is transformed into a quality of security maximization problem subject to layerwise execution constraint and deadline constraint of the DNN application. Due to the problem being NP-hard, a fast approximation algorithm is proposed to obtain the near-optimal solution under given real-time and security constraints. Extensive experiments and a real-life ICPS application evaluate the efficiency of the proposed techniques.

Decoder Design for Massive-MIMO Systems Using Deep Learning

Designing a massive multiple-input multiple-output (M-MIMO) system is complex, time-consuming, and challenging. Conventional MIMO decoders are either inefficient or complex to implement for M-MIMO scenarios. Deep learning (DL) has recently emerged to perform many complex operations more efficiently within a shorter time. In this article, a DL-based iterative network (DLNet) has been proposed for signal decoding in M-MIMO systems. The proposed DLNet decoder is a 40 layer deep neural network (DNN) that can decode M-MIMO signals for uplink Rayleigh and correlated MIMO channels. With the knowledge of the received signals and the M-MIMO channels, the proposed DLNet decoder decodes the messages of all the users. The DLNet considers complex-valued Rayleigh and correlated MIMO channels while decoding. Its training converges faster than other available deep learning (DL) based MIMO decoders. During simulations, the required number of DNN layers, training iterations, and other network parameters are estimated to improve the proposed decoder's performance. In the M-MIMO perspective, the proposed DLNet has been evaluated for symbol-error-rate (SER) performance, algorithm complexity, and runtime requirement. Simulation results show that the SER of the proposed DLNet decoder performs better by at least 2 dB than other M-MIMO decoding techniques. The DLNet consumes 800 times less time during decoding than other MIMO decoders for equal transmitting and receiving antenna configurations. At the same time, it has 13 times fewer trainable parameters than the available detection-network algorithm.

A New Approach to Descriptors Generation for Image Retrieval by Analyzing Activations of Deep Neural Network Layers.

In this brief, we consider the problem of descriptors construction for the task of content-based image retrieval using deep neural networks. The idea of neural codes, based on fully connected layers' activations, is extended by incorporating the information contained in convolutional layers. It is known that the total number of neurons in the convolutional part of the network is large and the majority of them have little influence on the final classification decision. Therefore, in this brief, we propose a novel algorithm that allows us to extract the most significant neuron activations and utilize this information to construct effective descriptors. The descriptors consisting of values taken from both the fully connected and convolutional layers perfectly represent the whole image content. The images retrieved using these descriptors match semantically very well to the query image, and also, they are similar in other secondary image characteristics, such as background, textures, or color distribution. These features of the proposed descriptors are verified experimentally based on the IMAGENET1M dataset using the VGG16 neural network. For comparison, we also test the proposed approach on the ResNet50 network.

NeuroMap: Efficient Task Mapping of Deep Neural Networks for Dynamic Thermal Management in High-Bandwidth Memory

High-bandwidth memory (HBM) offers breakthrough memory bandwidth through its vertically stacked memory architecture and through-silicon via (TSV)-based fast interconnect. However, the stacked architecture leads to high-power density causing thermal issues when running modern memory-hungry workloads such as deep neural networks (DNNs). Prior works on dynamic thermal management (DTM) of 3-D DRAM do not consider the physical structure of HBM and often lead to heavy DTM-induced performance penalty. We propose an application-aware efficient task mapping and migration-based DTM policy that maps DNN instances to cores through exploiting the channel layout of HBM and leveraging the significant temperature gradient across DRAM dies while making thermal decisions. We utilize the variation in the memory access behavior of DNN layers and attempt to minimize stalling due to thermal hotspots in the HBM stack. We also use application-aware dynamic voltage and frequency scaling (DVFS) and DRAM low-power states to further improve performance. Experimental results on workloads comprising seven popular DNNs show that NeuroMap results in an average execution time and memory energy reduction of 39% and 40%, respectively, over state-of-the-art DTM mechanisms.

Increasing Throughput of In-Memory DNN Accelerators by Flexible Layerwise DNN Approximation

Approximate computing and mixed-signal in-memory accelerators are promising paradigms to significantly reduce computational requirements of deep neural network (DNN) inference without accuracy loss. In this work, we present a novel in-memory design for layerwise approximate computation at different approximation levels. A sensitivity-based high-dimensional search is performed to explore the optimal approximation level for each DNN layer. Our new methodology offers high flexibility and optimal tradeoff between accuracy and throughput, which we demonstrate by an extensive evaluation on various DNN benchmarks for medium- and large-scale image classification with CIFAR10, CIFAR100, and ImageNet. With our novel approach, we reach an average of 5× and up to 8× speedup without accuracy loss.

Assessment of Transfer Learning Capabilities for Fatigue Damage Classification and Detection in Aluminum Specimens with Different Notch Geometries

Fatigue damage detection and its classification in metallic materials are persistently challenging the structural health monitoring community. The mechanics of fatigue damage is difficult to analyze and is further complicated because of the presence of notches of different geometries. These notches act as possible crack-nucleation sites resulting in failure mechanisms that are drastically different from one another. Often, sensor-based tools are used to monitor and detect fatigue damage in critical metallic materials such as aluminum alloys. Through deep neural networks (DNNs), such a sensor-based approach can be ubiquitously extended for a variety of geometries as appropriate for different applications. To that end, this paper presents a DNN-based transfer learning framework that can be used to classify and detect fatigue damage across candidate notch geometries. The DNNs are built upon ultrasonic time-series data obtained during fatigue testing of Al7075-T6 specimens with two types of notch geometries, namely, a U-notch and a V-notch. The baseline U-notch DNN is shown to achieve an accuracy of 96.1% while the baseline V-notch DNN has an accuracy of 95.8%. Both baseline DNNs are, thereafter, subjected to a transfer learning process by keeping a certain number of layers frozen and retraining only the remaining layers with a small volume of data obtained from the other notch geometry. When a layer of the baseline U-notch DNN is retrained with just 10% of the total V-notch data, an accuracy above 90% is observed for fatigue damage detection of V-notch specimens. Similar results are also obtained when the baseline V-notch DNN is retrained and interrogated to detect damage for U-notch specimens. These results, in summary, demonstrate the data-thrifty quality of combining the concepts of transfer learning and DNN for fatigue damage detection in different geometries of specimens made of high-performance aluminum alloys.

Block Walsh–Hadamard Transform-based Binary Layers in Deep Neural Networks

Convolution has been the core operation of modern deep neural networks. It is well known that convolutions can be implemented in the Fourier Transform domain. In this article, we propose to use binary block Walsh–Hadamard transform (WHT) instead of the Fourier transform. We use WHT-based binary layers to replace some of the regular convolution layers in deep neural networks. We utilize both one-dimensional (1D) and 2D binary WHTs in this article. In both 1D and 2D layers, we compute the binary WHT of the input feature map and denoise the WHT domain coefficients using a nonlinearity that is obtained by combining soft-thresholding with the tanh function. After denoising, we compute the inverse WHT. We use 1D-WHT to replace the 1 × 1 convolutional layers, and 2D-WHT layers can replace the 3 × 3 convolution layers and Squeeze-and-Excite layers. 2D-WHT layers with trainable weights can be also inserted before the Global Average Pooling layers to assist the dense layers. In this way, we can reduce the number of trainable parameters significantly with a slight decrease in trainable parameters. In this article, we implement the WHT layers into MobileNet-V2, MobileNet-V3-Large, and ResNet to reduce the number of parameters significantly with negligible accuracy loss. Moreover, according to our speed test, the 2D-FWHT layer runs about 24 times as fast as the regular 3 × 3 convolution with 19.51% less RAM usage in an NVIDIA Jetson Nano experiment.