Implementation Of Deep Neural Networks Research Articles

Challenges in the Path Toward a Scalable Silicon Photonics Implementation of Deep Neural Networks

The wide utility provided by the computational algorithms known as “deep learning” (or artificial neural networks) has prompted the development of specialized (electronic) hardware that can efficiently meet the speed and energy requirements that they demand. In the last few years, several proposals have also been made to build integrated photonic platforms to implement these algorithms by harnessing the light's ability to efficiently perform several kinds of computational operations with low latency times and low power consumption. In this work, we present a proposal of a photonic device that is designed to implement one basic deep learning architecture, the multilayer perceptron, which consists of the sequential application of matrix-vector multiplications and element-wise nonlinear functions. We analyze how the speed and energy efficiency performance metrics, as well as system requirements, scale with the number of nodes in the layers of the network. Finally, we discuss the possible routes that may enable the implementation of larger networks, which opens a wide variety of research paths that can contribute to the creation of more scalable and applicable photonic devices.

Fertility-GRU: Identifying Fertility-Related Proteins by Incorporating Deep-Gated Recurrent Units and Original Position-Specific Scoring Matrix Profiles.

Protein function prediction is one of the well-known problems in proteome research, attracting the attention of numerous researchers. However, the implementation of deep neural networks, which helps to increase the protein function prediction, still poses a big challenge. This study proposes a deep learning approach namely Fertility-GRU that incorporates gated recurrent units and position-specific scoring matrix profiles to predict the function of fertility-related protein, which is a highly crucial biological function. Fertility-related proteins also have been proven to be important in many biological entities (i.e., bone marrow and peripheral blood, postnatal mammalian ovary) and parameters (i.e., daily sperm production). As a result, our model can achieve a cross-validation accuracy of 85.8% and an independent accuracy of 91.1%. We also solve the problem of overfitting in the data set by adding dropout layers in the deep learning model. The independent testing results showed sensitivity, specificity, and Matthews correlation coefficient (MCC) values of 90.5%, 91.7%, and 0.82, respectively. Fertility-GRU demonstrates superiority in performance against the state-of-the-art predictor on the same data set. In our proposed study, we provided a method that enables more proteins to be discovered, especially proteins associated with fertility. Moreover, our achievement could promote the use of recurrent networks and gated recurrent units in proteome research. The source code and data set are freely accessible via https://github.com/khanhlee/fertility-gru .

Intelligent nanophotonics: merging photonics and artificial intelligence at the nanoscale.

Nanophotonics has been an active research field over the past two decades, triggered by the rising interests in exploring new physics and technologies with light at the nanoscale. As the demands of performance and integration level keep increasing, the design and optimization of nanophotonic devices become computationally expensive and time-inefficient. Advanced computational methods and artificial intelligence, especially its subfield of machine learning, have led to revolutionary development in many applications, such as web searches, computer vision, and speech/image recognition. The complex models and algorithms help to exploit the enormous parameter space in a highly efficient way. In this review, we summarize the recent advances on the emerging field where nanophotonics and machine learning blend. We provide an overview of different computational methods, with the focus on deep learning, for the nanophotonic inverse design. The implementation of deep neural networks with photonic platforms is also discussed. This review aims at sketching an illustration of the nanophotonic design with machine learning and giving a perspective on the future tasks.

3-D Stacked Synapse Array Based on Charge-Trap Flash Memory for Implementation of Deep Neural Networks

This paper proposes a synaptic device based on charge-trap flash memory that has good CMOS compatibility and superior reliability characteristics compared with other synaptic devices. Using hot-electron injection and hot-hole injection, we designed operation methods to implement gradual conductance modulation and spike-timing-dependent plasticity. We demonstrate the feasibility of the device for neuromorphic applications through both a device-level technology computer-aided design simulation and a system-level MATLAB simulation. For the first time, we also propose a 3-D stacked synapse array and present the structure, operation, and process methods. The proposed array architecture features a small area and low process cost and could be a novel solution for neuromorphic systems for implementing deep neural networks.

Classification Performance of Violence Content by Deep Neural Network with Monarch Butterfly Optimization

Violence is self-sufficient, it is perplexing due to visibility of content dissimilarities among the positive instances that been displayed on media. Besides, the ever-increasing demand on internet, with various types of videos and genres, causes difficulty for a proper search of these videos to ensure the contents is humongous. It involves in aiding users to choose movies or web videos suitable for audience, in terms of classifying violence content. Nevertheless, this is a cumbersome job since the definition of violence is broad and subjective. Detecting such nuances from videos becomes technical without a human’s supervision that can lead to conceptual problem. Generally, violence classification is performed based on text, audio, and visual features; to be precise, it is more relevant to use of audio and visual base. However, from this perspective, deep neural network is the current build-up in machine learning approach to solve classification problems. In this research, audio and visual features are learned by the deep neural network for more specific violence content classification. This study has explored the implementation of deep neural network with monarch butterfly optimization (DNNMBO) to effectively perform the classification of the violence content in web videos. Hence, the experiments are conducted using YouTube videos from VSD2014 dataset that are publicly available by Technicolor group. The results are compared with similar modified approaches such as DNNPSO and the original DNN. The outcome has shown 94% of violence classification rate by DNNMBO.

Memory-Efficient Deep Learning on a SpiNNaker 2 Prototype.

The memory requirement of deep learning algorithms is considered incompatible with the memory restriction of energy-efficient hardware. A low memory footprint can be achieved by pruning obsolete connections or reducing the precision of connection strengths after the network has been trained. Yet, these techniques are not applicable to the case when neural networks have to be trained directly on hardware due to the hard memory constraints. Deep Rewiring (DEEP R) is a training algorithm which continuously rewires the network while preserving very sparse connectivity all along the training procedure. We apply DEEP R to a deep neural network implementation on a prototype chip of the 2nd generation SpiNNaker system. The local memory of a single core on this chip is limited to 64 KB and a deep network architecture is trained entirely within this constraint without the use of external memory. Throughout training, the proportion of active connections is limited to 1.3%. On the handwritten digits dataset MNIST, this extremely sparse network achieves 96.6% classification accuracy at convergence. Utilizing the multi-processor feature of the SpiNNaker system, we found very good scaling in terms of computation time, per-core memory consumption, and energy constraints. When compared to a X86 CPU implementation, neural network training on the SpiNNaker 2 prototype improves power and energy consumption by two orders of magnitude.

Compression of Deep Neural Networks with Structured Sparse Ternary Coding

Deep neural networks (DNNs) contain large number of weights, and usually require many off-chip memory accesses for inference. Weight size compression is a major requirement for on-chip memory based implementation of DNNs, which not only increases inference speed but also reduces power consumption. We propose a weight compression method for deep neural networks by combining pruning and quantization. The proposed method allows weights to have values of + 1 or − 1 only at predetermined positions. Then, a look-up table stores all possible combinations of sub-vectors of weight matrices. Encoding and decoding structured sparse weights can be conducted easily with the table. This method not only allows multiplication-free DNN implementations but also compresses the weight storage by as much as x32 times more than that in floating-point networks and with only a tiny performance loss. Weight distribution normalization and gradual pruning techniques are applied to lower performance degradation. Experiments are conducted with fully connected DNNs and convolutional neural networks.

Deep Learning With Spiking Neurons: Opportunities and Challenges.

Spiking neural networks (SNNs) are inspired by information processing in biology, where sparse and asynchronous binary signals are communicated and processed in a massively parallel fashion. SNNs on neuromorphic hardware exhibit favorable properties such as low power consumption, fast inference, and event-driven information processing. This makes them interesting candidates for the efficient implementation of deep neural networks, the method of choice for many machine learning tasks. In this review, we address the opportunities that deep spiking networks offer and investigate in detail the challenges associated with training SNNs in a way that makes them competitive with conventional deep learning, but simultaneously allows for efficient mapping to hardware. A wide range of training methods for SNNs is presented, ranging from the conversion of conventional deep networks into SNNs, constrained training before conversion, spiking variants of backpropagation, and biologically motivated variants of STDP. The goal of our review is to define a categorization of SNN training methods, and summarize their advantages and drawbacks. We further discuss relationships between SNNs and binary networks, which are becoming popular for efficient digital hardware implementation. Neuromorphic hardware platforms have great potential to enable deep spiking networks in real-world applications. We compare the suitability of various neuromorphic systems that have been developed over the past years, and investigate potential use cases. Neuromorphic approaches and conventional machine learning should not be considered simply two solutions to the same classes of problems, instead it is possible to identify and exploit their task-specific advantages. Deep SNNs offer great opportunities to work with new types of event-based sensors, exploit temporal codes and local on-chip learning, and we have so far just scratched the surface of realizing these advantages in practical applications.

Lightening the Load with Highly Accurate Storage- and Energy-Efficient LightNNs

Hardware implementations of deep neural networks (DNNs) have been adopted in many systems because of their higher classification speed. However, while they may be characterized by better accuracy, larger DNNs require significant energy and area, thereby limiting their wide adoption. The energy consumption of DNNs is driven by both memory accesses and computation. Binarized neural networks (BNNs), as a tradeoff between accuracy and energy consumption, can achieve great energy reduction and have good accuracy for large DNNs due to their regularization effect. However, BNNs show poor accuracy when a smaller DNN configuration is adopted. In this article, we propose a new DNN architecture, LightNN, which replaces the multiplications to one shift or a constrained number of shifts and adds. Our theoretical analysis for LightNNs shows that their accuracy is maintained while dramatically reducing storage and energy requirements. For a fixed DNN configuration, LightNNs have better accuracy at a slight energy increase than BNNs, yet are more energy efficient with only slightly less accuracy than conventional DNNs. Therefore, LightNNs provide more options for hardware designers to trade off accuracy and energy. Moreover, for large DNN configurations, LightNNs have a regularization effect, making them better in accuracy than conventional DNNs. These conclusions are verified by experiment using the MNIST and CIFAR-10 datasets for different DNN configurations. Our FPGA implementation for conventional DNNs and LightNNs confirms all theoretical and simulation results and shows that LightNNs reduce latency and use fewer FPGA resources compared to conventional DNN architectures.

Implementation of deep neural networks to count dopamine neurons in substantia nigra.

Unbiased estimates of neuron numbers within substantia nigra are crucial for experimental Parkinson's disease models and gene‐function studies. Unbiased stereological counting techniques with optical fractionation are successfully implemented, but are extremely laborious and time‐consuming. The development of neural networks and deep learning has opened a new way to teach computers to count neurons. Implementation of a programming paradigm enables a computer to learn from the data and development of an automated cell counting method. The advantages of computerized counting are reproducibility, elimination of human error and fast high‐capacity analysis. We implemented whole‐slide digital imaging and deep convolutional neural networks (CNN) to count substantia nigra dopamine neurons. We compared the results of the developed method against independent manual counting by human observers and validated the CNN algorithm against previously published data in rats and mice, where tyrosine hydroxylase (TH)‐immunoreactive neurons were counted using unbiased stereology. The developed CNN algorithm and fully cloud‐embedded Aiforia™ platform provide robust and fast analysis of dopamine neurons in rat and mouse substantia nigra.

OA214] An experience with open source machine learning software deepmedic

Purpose The aim of this study was to investigate if 3D convolutional deep neural network implementation DeepMedic [1] could be applied to computed tomography angiography (CTA) images in acute stroke lesion detection. DeepMedic has previously been successful in lesion segmentation from magnetic resonance images [2] . Methods Preprocessing steps included robust intracranial space segmentation and intensity normalization. Hypoattenuated regions visible in the CTAs were manually delineated by two neuroradiologists and considered as ground truths. Data augmentation was performed with left–right flipped images. Training was done with scans from five infarct patients and subsequent testing with five separate cases. 35 epochs with 15 subepochs were used in the training. Results The initial results were promising as the infarcts in all the test cases could be correctly lateralized. The achieved voxel-wise segmentation performance was moderate (considering small training set) with Sorensen-Dice similarity coefficient 0.52 for the test data. Conclusions The significance of open science and freely available scientific software will only increase in the future. Convolutional neural networks demonstrate promising prospects for medical imaging. DeepMedic can be applied to CT images with proper data preprocessing. In a future study the findings will be verified with a larger cohort. Improving the performance with additional input features will also be investigated.

A computational framework for the detection of subcortical brain dysmaturation in neonatal MRI using 3D Convolutional Neural Networks

Deep neural networks are increasingly being used in both supervised learning for classification tasks and unsupervised learning to derive complex patterns from the input data. However, the successful implementation of deep neural networks using neuroimaging datasets requires adequate sample size for training and well-defined signal intensity based structural differentiation. There is a lack of effective automated diagnostic tools for the reliable detection of brain dysmaturation in the neonatal period, related to small sample size and complex undifferentiated brain structures, despite both translational research and clinical importance. Volumetric information alone is insufficient for diagnosis. In this study, we developed a computational framework for the automated classification of brain dysmaturation from neonatal MRI, by combining a specific deep neural network implementation with neonatal structural brain segmentation as a method for both clinical pattern recognition and data-driven inference into the underlying structural morphology. We implemented three-dimensional convolution neural networks (3D-CNNs) to specifically classify dysplastic cerebelli, a subset of surface-based subcortical brain dysmaturation, in term infants born with congenital heart disease. We obtained a 0.985 ± 0. 0241-classification accuracy of subtle cerebellar dysplasia in CHD using 10-fold cross-validation. Furthermore, the hidden layer activations and class activation maps depicted regional vulnerability of the superior surface of the cerebellum, (composed of mostly the posterior lobe and the midline vermis), in regards to differentiating the dysplastic process from normal tissue. The posterior lobe and the midline vermis provide regional differentiation that is relevant to not only to the clinical diagnosis of cerebellar dysplasia, but also genetic mechanisms and neurodevelopmental outcome correlates. These findings not only contribute to the detection and classification of a subset of neonatal brain dysmaturation, but also provide insight to the pathogenesis of cerebellar dysplasia in CHD. In addition, this is one of the first examples of the application of deep learning to a neuroimaging dataset, in which the hidden layer activation revealed diagnostically and biologically relevant features about the clinical pathogenesis. The code developed for this project is open source, published under the BSD License, and designed to be generalizable to applications both within and beyond neonatal brain imaging.

Towards Ultra-High Performance and Energy Efficiency of Deep Learning Systems: An Algorithm-Hardware Co-Optimization Framework

Hardware accelerations of deep learning systems have been extensively investigated in industry and academia. The aim of this paper is to achieve ultra-high energy efficiency and performance for hardware implementations of deep neural networks (DNNs). An algorithm-hardware co-optimization framework is developed, which is applicable to different DNN types, sizes, and application scenarios. The algorithm part adopts the general block-circulant matrices to achieve a fine-grained tradeoff of accuracy and compression ratio. It applies to both fully-connected and convolutional layers and contains a mathematically rigorous proof of the effectiveness of the method. The proposed algorithm reduces computational complexity per layer from O(n2) to O(n log n) and storage complexity from O(n2) to O(n), both for training and inference. The hardware part consists of highly efficient Field Programmable Gate Array (FPGA)-based implementations using effective reconfiguration, batch processing, deep pipelining, resource re-using, and hierarchical control. Experimental results demonstrate that the proposed framework achieves at least 152X speedup and 71X energy efficiency gain compared with IBM TrueNorth processor under the same test accuracy. It achieves at least 31X energy efficiency gain compared with the reference FPGA-based work.

An Architecture to Accelerate Convolution in Deep Neural Networks

In the past few years, the demand for real-time hardware implementations of deep neural networks (DNNs), especially convolutional neural networks (CNNs), has dramatically increased, thanks to their excellent performance on a wide range of recognition and classification tasks. When considering real-time action recognition and video/image classification systems, latency is of paramount importance. Therefore, applications strive to maximize the accuracy while keeping the latency under a given application-specific maximum: in most cases, this threshold cannot exceed a few hundred milliseconds. Until now, the research on DNNs has mainly focused on achieving a better classification or recognition accuracy, whereas very few works in literature take in account the computational complexity of the model. In this paper, we propose an efficient computational method, which is inspired by a computational core of fully connected neural networks, to process convolutional layers of state-of-the-art deep CNNs within strict latency requirements. To this end, we implemented our method customized for VGG and VGG-based networks which have shown state-of-the-art performance on different classification/recognition data sets. The implementation results in 65-nm CMOS technology show that the proposed accelerator can process convolutional layers of VGGNet up to 9.5 times faster than state-of-the-art accelerators reported to-date while occupying 3.5 mm2.

Deep Learning Based Intrusion Detection With Adversaries.

Deep neural networks have demonstrated their effectiveness for most machine learning tasks, with Intrusion Detection included. Unfortunately, recent research found that deep neural networks are vulnerable to adversarial examples in the image classification domain, i.e., they leave some opportunities for an attacker to fool the networks into misclassification by introducing imperceptible changes to the original pixels in an image. The vulnerability raise some concerns in applying deep neural networks in security-critical areas such as Intrusion Detection. In this paper, we investigate the performances of the state-of-the-art attack algorithms against deep learning based Intrusion Detection on the NSL-KDD dataset. Based on the implementation of deep neural networks using TensorFlow, we examine the vulnerabilities of neural networks under attacks on the IDS. To gain insights into the nature of Intrusion Detection and its attacks, we also explore the roles of individual features in generating adversarial examples.

Recent advances in efficient computation of deep convolutional neural networks

Deep neural networks have evolved remarkably over the past few years and they are currently the fundamental tools of many intelligent systems. At the same time, the computational complexity and resource consumption of these networks also continue to increase. This will pose a significant challenge to the deployment of such networks, especially in real-time applications or on resource-limited devices. Thus, network acceleration has become a hot topic within the deep learning community. As for hardware implementation of deep neural networks, a batch of accelerators based on FPGA/ASIC have been proposed in recent years. In this paper, we provide a comprehensive survey of recent advances in network acceleration, compression and accelerator design from both algorithm and hardware points of view. Specifically, we provide a thorough analysis of each of the following topics: network pruning, low-rank approximation, network quantization, teacher-student networks, compact network design and hardware accelerators. Finally, we will introduce and discuss a few possible future directions.

Proteus: Exploiting precision variability in deep neural networks

Abstract This work investigates how using reduced precision data in Deep Neural Networks (DNNs) affects network accuracy during classification. We observe that the tolerance of DNNs to reduced precision data not only varies across networks, but also within networks. We study how error tolerance across layers varies and propose a method for finding a low precision configuration for a network while maintaining high accuracy. To exploit these low precision configurations, this work proposes Proteus , which reduces the data traffic and storage footprint needed by DNNs, resulting in reduced energy for DNN implementations. Proteus uses a different representation per layer for both the data (neuron activations) and the weights (synapses) processed by DNNs. Proteus is a layered extension over existing DNN implementations that maintains the native precision of the compute engine by converting to and from a fixed-point reduced precision format used in memory. Proteus uses a novel memory layout, enabling a simple, low-cost and low-energy conversion unit. We evaluate Proteus as an extension to a state-of-the-art accelerator [1] which uses a uniform 16-bit fixed-point representation. On five popular Convolutional Neural Networks (CNNs) and during inference Proteus reduces data traffic by 43% on average while maintaining accuracy within 1% compared to the full precision baseline. As a result, Proteus improves energy by 15% with no performance loss.

VLSI Implementation of Deep Neural Network Using Integral Stochastic Computing

The hardware implementation of deep neural networks (DNNs) has recently received tremendous attention: many applications in fact require high-speed operations that suit a hardware implementation. However, numerous elements and complex interconnections are usually required, leading to a large area occupation and copious power consumption. Stochastic computing has shown promising results for low-power area-efficient hardware implementations, even though existing stochastic algorithms require long streams that cause long latencies. In this paper, we propose an integer form of stochastic computation and introduce some elementary circuits. We then propose an efficient implementation of a DNN based on integral stochastic computing. The proposed architecture has been implemented on a Virtex7 FPGA, resulting in 45% and 62% average reductions in area and latency compared to the best reported architecture in literature. We also synthesize the circuits in a 65 nm CMOS technology and we show that the proposed integral stochastic architecture results in up to 21% reduction in energy consumption compared to the binary radix implementation at the same misclassification rate. Due to fault-tolerant nature of stochastic architectures, we also consider a quasi-synchronous implementation which yields 33% reduction in energy consumption w.r.t. the binary radix implementation without any compromise on performance.