Implementation Of Deep Neural Networks Research Articles

Implementation of Deep Neural Networks for performance prediction and optimization of a porous volumetric solar receiver considering mechanical safety

The volumetric solar receiver is a major component of concentrating solar thermal systems. Proper selection of design and operating parameters is necessary to obtain the best receiver performance. The current work models and optimizes the hydraulic, thermal, and mechanical performance of a porous volumetric solar receiver using the Deep Neural Network (DNN) technique. The dataset for DNN training is generated by numerical simulations and porosity and pore size of foam structure, inlet fluid velocity and absorber length are selected as the input parameters. The pressure drop, outlet fluid temperature and maximum failure index are selected as the output variables. The trained model demonstrates high accuracy in performance prediction for any combination of input variables in the trained range. Further, the present work proposes the integration of the DNN technique with Genetic Algorithm for the constrained multi-objective optimization of the absorber performance with a novel constraint of maximum failure index to select configurations preventing mechanical failure. Such integration is found to be nearly 524 times faster than the conventional direct approach, demonstrating a significant decrease in the time needed for optimization. Furthermore, the potential of the Technique for Order of Preference by Similarity to Ideal Solution in selecting the desired configuration from the Pareto optimal solutions as per individual requirements is also investigated. The optimization results indicate that the optimum values for porosity, pore size, and absorber length are in the range of 0.9–0.93, 2–4 mm, and 0.01–0.06 m, respectively, when all variables are optimized simultaneously. Furthermore, the optimal input velocity is observed to be close to the lower bound of 0.4 m/s while the corresponding maximum failure index varies from 0.346 to 0.836, indicating mechanically safe configurations. The high accuracy and faster optimization results obtained in the present work demonstrate the ability of the proposed approach in implementing quick and computationally cheap optimization studies for improved and mechanically safe absorber designs.

An Empirical Approach to Enhance Performance for Scalable CORDIC-Based Deep Neural Networks

Practical implementation of deep neural networks (DNNs) demands significant hardware resources, necessitating high computational power and memory bandwidth. While existing field-programmable gate array (FPGA)–based DNN accelerators are primarily optimized for fast single-task performance, cost, energy efficiency, and overall throughput are crucial considerations for their practical use in various applications. This article proposes a performance-centric pipeline Coordinate Rotation Digital Computer (CORDIC)–based MAC unit and implements a scalable CORDIC-based DNN architecture that is area- and power-efficient and has high throughput. The CORDIC-based neuron engine uses bit-rounding to maintain input-output precision and minimal hardware resource overhead. The results demonstrate the versatility of the proposed pipelined MAC, which operates at 460 MHz and allows for higher network throughput. A software-based implementation platform evaluates the proposed MAC operation’s accuracy for more extensive neural networks and complex datasets. The DNN accelerator with parameterized and modular layer-multiplexed architecture is designed. Empirical evaluation through Pareto analysis is used to improve the efficiency of DNN implementations by fixing the arithmetic precision and optimal pipeline stages. The proposed architecture utilizes layer-multiplexing, a technique that effectively reuses a single DNN layer to enhance efficiency while maintaining modularity and adaptability for integrating various network configurations. The proposed CORDIC MAC-based DNN architecture is scalable for any bit-precision network size, and the DNN accelerator is prototyped using the Xilinx Virtex-7 VC707 FPGA board, operating at 66 MHz. The proposed design does not use any Xilinx macros, making it easily adaptable for ASIC implementation. Compared with state-of-the-art designs, the proposed design reduces resource use by 45% and power consumption by 4× without sacrificing performance. The accelerator is validated using the MNIST dataset, achieving 95.06% accuracy, only 0.35% less than other cutting-edge implementations.

A Survey on Fault-Tolerant Methodologies for Deep Neural Networks

Asignificant rise in Artificial Intelligence (AI) has impacted many applications around us, so much so that AI has now been increasingly used in safety-critical applications. AI at the edge is the reality, which means performing the data computation closer to the source of the data, as opposed to performing it on the cloud. Safety-critical applications have strict reliability requirements; therefore, it is essential that AI models running on the edge (i.e., hardware) must fulfill the required safety standards. In the vast field of AI, Deep Neural Networks (DNNs) are the focal point of this survey as it has continued to produce extraordinary outcomes in various applications i.e. medical, automotive, aerospace, defense, etc. Traditional reliability techniques for DNNs implementation are not always practical, as they fail to exploit the unique characteristics of the DNNs. Furthermore, it is also essential to understand the targeted edge hardware because the impact of the faults can be different in ASICs and FPGAs. Therefore, in this survey, first, we have examined the impact of the fault in ASICs and FPGAs, and then we seek to provide a glimpse of the recent progress made towards the fault-tolerant DNNs. We have discussed several factors that can impact the reliability of the DNNs. Further, we have extended this discussion to shed light on many state-of-the-art fault mitigation techniques for DNNs.

FPGA-based implementation of deep neural network using stochastic computing

A serious challenge in artificial real-time applications is the hardware implementation of deep neural networks (DNN). Among various methods, stochastic computing (SC)-based implementations received tremendous attention due to the low hardware overhead. However, the slow convergence rate is a major problem in SC-based neural networks’ implementation, and millions of clock cycles are required to generate a relatively accurate output. The reconfigurability and parallel nature of field programmable gate array (FPGA) chips make them a preferable platform for SC-based DNN implementation. A fully or semi-parallel implementation of DNNs requires extensive hardware resources. In this paper, an efficient method for DNN implementation on an FPGA chip is presented to address these problems. The FPGA chip reconfiguration feature allows a DNN with several different neurons and topologies to be implemented on a single chip. Convergence time is significantly reduced by limiting the length of the stochastic bitstreams and establishing synchronization between the processing units based on precise timing. Furthermore, due to the limited number of input–output pins in the FPGA chip, a sequential architecture is proposed wherein the DNN inputs enter through only three 8-bit ports. This makes it possible to implement DNNs for image-processing applications. The proposed method is implemented using the Verilog hardware description language on the Xilinx FPGA Virtex-7 xc7v2000t chip. The results show a more than 82% reduction in hardware resources and the minimum rate of power consumption compared to state-of-the-art methods. In addition, the average error rate of the implemented DNN is reduced by 2%.

Optimization of Deep Neural Networks Using DAPP (DNN Acceleration Using Ping-Pong) Approach

Abstract: Deep Neural Networks (DNNs) are one of the leading classification algorithms. Deep Learning has achieved remarkable milestones such as Google-Net and Alpha-Go. They have shown promising results in pattern recognition for images and text, language translation, sound recognition, and many more. DNN has been widely accepted and also employed for pattern matching and image recognition. All these applications are possible as these networks emulate functioning of human brain and hence name “Neural” Network.However, to provide competent results, these millions of neurons in neural networks needs to be trained. For which billions of operations are to be carried out. Training of these many neurons and with these many operations is a time-consuming affair. Hence choice of network and its parameter play an important role both in providing accurate trained network and time taken for training. If the network is deep and has plethora of neurons, the time taken is considerably high as training works sequentially on batches of dataset using Sequential Back-propagation Algorithm.To accelerate the training there are many hardware solutions like use of GPU, FPGA and ASICs. However because of popularity of DNN there is increase in demand in mobile and IoT platform devices. These are resource constrained devices, where power and size of these device, restricts usage and implementation of deep NN (Neural Network).Simulation of DAPP is done on MNIST and CIFAR-10 datasets using System-C. Additionally, this technique has been adapted for multi-core architectures. The design shows a reduction in time by 38% for 3 layers of CNN and 92% for 10 layers of CNN, while maintaining the accuracy of networks. This generic methodology has been implemented for Vanilla RNN and LSTM networks. An improvement of 38% for Vanilla RNN and 40% for LSTM has been demonstrated by this methodology.

Large scale and quantum accurate molecular dynamics simulation: Liquid iron under extreme condition

Liquid iron is the major component of planetary cores. Its structure and dynamics under high pressure and temperature is of great significance in studying geophysics and planetary science. However, for experimental techniques, it is still difficult to generate and probe such a state of matter under extreme conditions, while for theoretical method like molecular dynamics simulation, the reliable estimation of dynamic properties requires both large simulation size and <i>ab initio</i> accuracy, resulting in unaffordable computational costs for traditional method. Owing to the technical limitation, the understanding of such matters remains limited. In this work, combining molecular dynamics simulation, we establish a neural network potential energy surface model to study the static and dynamic properties of liquid iron at its extreme thermodynamic state close to core-mantle boundary. The implementation of deep neural network extends the simulation scales from one hundred atoms to millions of atoms within quantum accuracy. The estimated static and dynamic structure factor show good consistency with all available X-ray diffraction and inelastic X-ray scattering experimental observations, while the empirical potential based on embedding-atom-method fails to give a unified description of liquid iron across a wide range of thermodynamic conditions. We also demonstrate that the transport property like diffusion coefficient exhibits a strong size effect, which requires more than at least ten thousands of atoms to give a converged value. Our results show that the combination of deep learning technology and molecular modelling provides a way to describe matter realistically under extreme conditions.

Neural network architectures using min-plus algebra for solving certain high-dimensional optimal control problems and Hamilton–Jacobi PDEs

Solving high-dimensional optimal control problems and corresponding Hamilton–Jacobi PDEs are important but challenging problems in control engineering. In this paper, we propose two abstract neural network architectures which are, respectively, used to compute the value function and the optimal control for certain class of high-dimensional optimal control problems. We provide the mathematical analysis for the two abstract architectures. We also show several numerical results computed using the deep neural network implementations of these abstract architectures. A preliminary implementation of our proposed neural network architecture on FPGAs shows promising speedup compared to CPUs. This work paves the way to leverage efficient dedicated hardware designed for neural networks to solve high-dimensional optimal control problems and Hamilton–Jacobi PDEs.

A Framework for Neural Network Architecture and Compile Co-optimization

The efficiency of deep neural network (DNN) solutions on real hardware devices are mainly decided by the DNN architecture and the compiler-level scheduling strategy on the hardware. When we try to fully exploit the underlying hardware and obtain the optimal tradeoff between DNN accuracy and runtime performance, we discovered that the two optimization goals of DNN architecture and scheduling policy are intimately related to each other. However, current hardware-aware Neural Architecture Search (NAS) methods primarily focus on the DNN architecture search process, ignoring the effects of various compiler-level scheduling strategies (e.g., graph-level optimization, loop transformations, parallelization, etc.) on network candidates being evaluated in the search process. As a result, they may overlook the true-optimal DNN implementations on hardware, which can only be discovered by trying-out different combinations of scheduling strategies and DNN architectures. This work proposes a NAS framework (CHaNAS) that searches for not only the network architecture but also the dedicated compiler-level scheduling policy, as the optimal co-design solution on the target hardware. We propose to use a block-based pre-scheduling methodology to reduce the co-design search space and enable the automatic generation of the optimal co-design, including the network architecture and the tensor programs that practice the scheduling policy. Further, we introduce a new search objective function based on the generalization gap to prevent the selection of architectures that are prone to overfitting. We evaluate CHaNAS on Imagenet on different hardware back-ends against the state-of-the-art hardware-aware search method based on the MobileNet-v3 search space. Experimental results show that the co-design solutions obtained by ChaNAS show up to 1.6×, 1.9×, and 1.7×, 24 performance boost on NVIDIA P100 GPU, Intel Xeon 8163 CPU, and Samsung Note 10 Mobile, respectively, over the baselines of the same-level accuracy.

Signal recovery in optical wireless communication using photonic convolutional processor.

Deep neural networks (DNNs) have been applied to recover signals in optical communication systems and have shown competence of mitigating linear and nonlinear distortions. However, as the data throughput increases, the heavy computational cost of DNNs impedes them from rapid and power-efficient processing. In this paper, we propose an optical communication signal recovery technology based on a photonic convolutional processor, which is realized by dispersion delay unit and wavelength division multiplexing. Based on the photonic convolutional processor, we implement an optoelectronic convolutional neural network (OECNN) for signal post-equalization and experimentally demonstrate on 16QAM and 32QAM of an optical wireless communication system. With system parameters optimization, we verify that the OECNN can achieve accurate signal recovery where the bit error ratio (BER) is below the 7% forward error correction threshold of 3.8×10-3 at 2Gbps. With adding the OECNN-based nonlinear compensation, compared with only linear compensation, we improve the quality (Q) factor by 3.35 dB at 16QAM and 3.30 dB at 32QAM, which is comparable to that of an electronic neural network. This work proves that the photonic implementation of DNN is promising to provide a fast and power-efficient solution for optical communication signal processing.

Improvising limitations of DNN based ultrasound image reconstruction.

Ultrasound modalities are cost-effective and radiation-free technology for real-time medical imaging. These modalities require image reconstruction to obtain the actual ultrasound images from ultrasound raw data. The ultrasound raw data is obtained in the form of echo after scanning an imaging plane through ultrasound waves. The most commonly used image reconstruction beamforming technique is Delay and Sum (DAS). Other sophisticated beamforming techniques are Delay Multiply and Sum (DMAS) and Minimum Variance Distortionless Response (MVDR). DAS has limited image quality, and the employment of sophisticated techniques increases the computational complexity and computational time with improvement in image quality. To overcome these problems, various DNN (Deep Neural Networks) based techniques have been proposed which can reconstruct ultrasound images directly from ultrasound raw data. But DNN implementation has two limitations: accuracy of reconstruction and generalizability of the model. To overcome these limitations, we are proposing methodologies with a DNN model which was able to reduce these limitations. Firstly, we generated the datasets which include multiple shapes such as line, circle, ellipse, and parabola. After that, we have implemented a CNN-DNN (Convolution Neural Network and Deep Neural Network) hybrid model which has significantly improved computational time as well as image quality. We have trained our model with different sets of data to validate the reconstruction of the image matrix. We achieved a significant improvement in computational time of around 100 times (from around 0.6s to 0.0059s) as compared to DAS beamforming technique. At the same time, we also achieved a significant improvement in image quality with 37.19dB average and 41.37dB maximum improved Peak Signal to Noise Ratio (PSNR), and 87.41% average and 95% maximum Structural Similarity Index Matrix (SSIM) value. We also achieved generalizability and precise image reconstruction by using the proposed model.

A Brief Review of Deep Neural Network Implementations for ARM Cortex-M Processor

Deep neural networks have recently become increasingly used for a wide range of applications, (e.g., image and video processing). The demand for edge inference is growing, especially in the areas of relevance to the Internet-of-Things. Low-cost microcontrollers as edge devices are a promising solution for optimal application systems from several points of view such as: cost, power consumption, latency, or real-time execution. The implementation of these systems has become feasible due to the advanced development of hardware architectures and DSP capabilities, while the cost and power consumption have been maintained at a low level. The aim of the paper is to provide a literature review on the implementation of deep neural networks using ARM Cortex-M core-based low-cost microcontrollers. As an emerging research direction, there are a limited number of publications that address this topic at the moment. Therefore, the research papers that stand out have been analyzed in greater detail, to promote further interest of researchers to bring AI techniques to low power standard ARM Cortex-M microcontrollers. The article addresses a niche research domain. Despite the increasing interest manifested toward both (1) edge AI applications and (2) theoretical contributions in DNN optimization and compression, the number of existing publications dedicated to the current topic is rather limited. Therefore, a comprehensive literature survey using systematic mapping is not possible. The presentation focuses on systems that have shown increased efficiency in resource-constrained applications, as well as the predominant impediments that still hinder their implementation. The reader will take away the following concepts from this paper: (1) an overview of applications, DNN architectures, and results obtained using ARM Cortex-M core-based microcontrollers, (2) an overview of low-cost hardware devices and SW development solutions, and (3) understanding recent trends and opportunities.

NEAT: Nonlinearity Aware Training for Accurate, Energy-Efficient, and Robust Implementation of Neural Networks on 1T-1R Crossbars

In this era of IoT, energy-efficient and adversarially secure implementation of deep neural networks (DNNs) on hardware has become imperative. Memristive crossbars have emerged as an energy-efficient component of deep learning hardware accelerators due to their compact and efficient matrix-vector multiplication (MVM) implementation. However, they suffer from nonidealities (such as, interconnect parasitics, device variations, and sneak paths) introduced by their circuit topology that degrades computational accuracy. A 1T-1R synapse, adding a transistor (1T) in series with the memristive synapse (1R), has been proposed to mitigate sneak paths in a crossbar. However, we observe that the nonlinear characteristics of the transistor affect the overall conductance of the 1T-1R cell which in turn affects the MVM operation. This 1T-1R nonlinearity arising from the input voltage-dependent nonlinearity is not only difficult to model or formulate, but also causes a drastic performance degradation of DNNs when mapped to such crossbars. In this article, we first analyses the nonlinearity in ideal 1T-1R crossbars (excluding nonidealities, such as device variations and interconnect parasitics) and propose a novel nonlinearity aware training (NEAT) method to address the nonlinearities. Specifically, we first identify the range of network weights, which can be mapped into the 1T-1R cell within the linear operating region of the transistor. After that, we regularize the weights of neural networks to exist within the linear operating range by using an iterative training algorithm. Our iterative training significantly recovers the classification accuracy drop caused by the nonlinearity. Moreover, we find that each layer has a different weight distribution and in turn requires different gate voltage of transistor to guarantee linear operation. Based on this observation, we achieve energy efficiency while preserving classification accuracy by applying heterogeneous gate-voltage control to the 1T-1R cells across different layers. Finally, we conduct various experiments on CIFAR10 and CIFAR100 benchmark datasets to demonstrate the effectiveness of our NEAT. Overall, NEAT yields <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\sim 20\%$ </tex-math></inline-formula> energy gain with less than 1% accuracy loss (with homogeneous gate control) when mapping ResNet18 networks on 1T-1R crossbars. Thereafter, we integrate the 1T-1R crossbars with various nonidealities. We show that NEAT leads to more adversarially robust mappings of DNNs onto nonideal 1T-1R crossbars than standard DNNs mapped directly onto 1R crossbars. In case of a VGG11 network on CIFAR100 dataset, we obtain <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\sim 17$ </tex-math></inline-formula> % improvement in clean accuracy and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\sim 2$ </tex-math></inline-formula> %–8% & <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\sim 5$ </tex-math></inline-formula> %–6% improvements in adversarial accuracies, respectively, for fast gradient sign method (FGSM) and projected gradient descent (PGD)-based adversarial attacks via NEAT on nonideal 64 <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$\times $ </tex-math></inline-formula> 64 crossbars, in comparison to standard DNNs.

Compression of deep neural networks: bridging the gap between conventional-based pruning and evolutionary approach

Recently, many studies have been carried out on model compression to handle the high computational cost and high memory footprint brought by the implementation of deep neural networks. In this paper, model compression of convolutional neural networks is constructed as a multiobjective optimization problem with two conflicting objectives, reducing the model size and improving the performance. A novel structured pruning method called Conventional-based and Evolutionary Approaches Guided Multiobjective Pruning (CEA-MOP) is proposed to address this problem, where the power of conventional pruning methods is effectively exploited for the evolutionary process. A delicate balance in pruning rate and model accuracy has been automated achieved by a multiobjective optimization evolutionary model. First, an ensemble framework integrates pruning metrics to establish a codebook for further evolutionary operations. Then, an efficient coding method is developed to shorten the length of chromosome, thus ensuring its superior scalability. Finally, sensitivity analysis is automatically carried out to determine the upper bound of pruning rate for each layer. Notably, on CIFAR-10, CEA-MOP reduces more than 50% FLOPs on ResNet-110 and improves the relative accuracy. Moreover, on ImageNet, CEA-MOP reduces more than 50% FLOPs on ResNet-101 with negligible top-1 accuracy drop.

Ternary Neural Networks Based on on/off Memristors: Set-Up and Training

Neuromorphic systems based on hardware neural networks (HNNs) are expected to be an energy and time-efficient computing architecture for solving complex tasks. In this paper, we consider the implementation of deep neural networks (DNNs) using crossbar arrays of memristors. More specifically, we considered the case where such devices can be configured in just two states: the low-resistance state (LRS) and the high-resistance state (HRS). HNNs suffer from several non-idealities that need to be solved when mapping our software-based models. A clear example in memristor-based neural networks is conductance variability, which is inherent to resistive switching devices, so achieving good performance in an HNN largely depends on the development of reliable weight storage or, alternatively, mitigation techniques against weight uncertainty. In this manuscript, we provide guidelines for a system-level designer where we take into account several issues related to the set-up of the HNN, such as what the appropriate conductance value in the LRS is or the adaptive conversion of current outputs at one stage to input voltages for the next stage. A second contribution is the training of the system, which is performed via offline learning, and considering the hardware imperfections, which in this case are conductance fluctuations. Finally, the resulting inference system is tested in two well-known databases from MNIST, showing that is competitive in terms of classification performance against the software-based counterpart. Additional advice and insights on system tuning and expected performance are given throughout the paper.

User Demographic Information and Deep Neural Network in Film Recommendation System based on Collaborative Filtering

Research about implementation of deep neural network in recommender system based on collaborative filtering received many attentions recently. One of the major problems in deep neural network based collaborative filtering recommendation system was coldstart problem. Some recent work tried to improve model performance by modifying how the model modelled the interaction between user and item features to generate TOP-N recommendations. This work proposed DNCF (Demographic Neural Collaborative Filtering) model that utilized user demographic information and deep neural network architecture to generate film recommendation system based on collaborative filtering in cold-start problem. NCF model was used as baseline model for model performance comparison. Hit Ratio and Normalized Discounted Cumulative Gain for TOP-10 recommendations were used as evaluation metrics for model performance. Experiment results showed that the proposed DNCF model outperform baseline NCF model by 23,61% in HR@10 and 22,40% in NDCG@10 evaluation metrics. Keywords — collaborative filtering, deep neural network, demographic information, neural collaborative filtering, recommendation system

A Long Short-Term Memory for AI Applications in Spike-based Neuromorphic Hardware

Spike-based neuromorphic hardware holds promise for more energy-efficient implementations of deep neural networks (DNNs) than standard hardware such as GPUs. But this requires us to understand how DNNs can be emulated in an event-based sparse firing regime, as otherwise the energy advantage is lost. In particular, DNNs that solve sequence processing tasks typically employ long short-term memory units that are hard to emulate with few spikes. We show that a facet of many biological neurons, slow after-hyperpolarizing currents after each spike, provides an efficient solution. After-hyperpolarizing currents can easily be implemented in neuromorphic hardware that supports multi-compartment neuron models, such as Intel’s Loihi chip. Filter approximation theory explains why after-hyperpolarizing neurons can emulate the function of long short-term memory units. This yields a highly energy-efficient approach to time-series classification. Furthermore, it provides the basis for an energy-efficient implementation of an important class of large DNNs that extract relations between words and sentences in order to answer questions about the text. Deep learning could be less energy intensive when implemented on spike-based neuromorphic chips. An approach inspired by a characteristic feature of biological neurons, the presence of slowly changing internal currents, is developed to emulate long short-term memory units in a sparse spiking regime for neuromorphic implementation.

MegaD: Deep Learning for Rapid and Accurate Disease Status Prediction of Metagenomic Samples.

The diversity within different microbiome communities that drive biogeochemical processes influences many different phenotypes. Analyses of these communities and their diversity by countless microbiome projects have revealed an important role of metagenomics in understanding the complex relation between microbes and their environments. This relationship can be understood in the context of microbiome composition of specific known environments. These compositions can then be used as a template for predicting the status of similar environments. Machine learning has been applied as a key component to this predictive task. Several analysis tools have already been published utilizing machine learning methods for metagenomic analysis. Despite the previously proposed machine learning models, the performance of deep neural networks is still under-researched. Given the nature of metagenomic data, deep neural networks could provide a strong boost to growth in the prediction accuracy in metagenomic analysis applications. To meet this urgent demand, we present a deep learning based tool that utilizes a deep neural network implementation for phenotypic prediction of unknown metagenomic samples. (1) First, our tool takes as input taxonomic profiles from 16S or WGS sequencing data. (2) Second, given the samples, our tool builds a model based on a deep neural network by computing multi-level classification. (3) Lastly, given the model, our tool classifies an unknown sample with its unlabeled taxonomic profile. In the benchmark experiments, we deduced that an analysis method facilitating a deep neural network such as our tool can show promising results in increasing the prediction accuracy on several samples compared to other machine learning models.

NN-Lock : A Lightweight Authorization to Prevent IP Threats of Deep Learning Models

The prevalent usage and unparalleled recent success of Deep Neural Network (DNN) applications have raised the concern of protecting their Intellectual Property (IP) rights in different business models to prevent the theft of trade secrets. In this article, we propose a lightweight, generic, key-based DNN IP protection methodology, NN-Lock , to defend against unauthorized usage of stolen DNN models. NN-Lock utilizes SBox, a cryptographic primitive, with good security properties to encrypt each parameter of a trained DNN model with the secret keys derived from a master key through a key-scheduling algorithm. The method ensures that only an authorized user with a correct master key can accurately use the locked DNN model. Evaluation results of NN-Lock on a Google Coral edge device for various DNN architectures on several datasets show that for an incorrect master key, the accuracy of a locked model is that of a random classifier. The dense network of encrypted parameters makes the method robust against the model fine-tuning attack and a novel approximation attack using the Genetic Algorithm, which achieves reasonable success against another recent IP protection scheme called HPNN Chakraborty et al. 2020 . The security evaluation of NN-Lock against other families of attacks demonstrates its soundness in practical scenarios. NN-Lock does not modify any internal structure of a DNN model, making it scalable for all of the existing DNN implementations without adversely affecting their performance.

Perceptual Hash of Neural Networks

In recent years, advances in deep learning have boosted the practical development, distribution and implementation of deep neural networks (DNNs). The concept of symmetry is often adopted in a deep neural network to construct an efficient network structure tailored for a specific task, such as the classic encoder-decoder structure. Massive DNN models are diverse in category, quantity and open source frameworks for implementation. Therefore, the retrieval of DNN models has become a problem worthy of attention. To this end, we propose a new idea of generating perceptual hashes of DNN models, named HNN-Net (Hash Neural Network), to index similar DNN models by similar hash codes. The proposed HNN-Net is based on neural graph networks consisting of two stages: the graph generator and the graph hashing. In the graph generator stage, the target DNN model is first converted and optimized into a graph. Then, it is assigned with additional information extracted from the execution of the original model. In the graph hashing stage, it learns to construct a compact binary hash code. The constructed hash function can well preserve the features of both the topology structure and the semantics information of a neural network model. Experimental results demonstrate that the proposed scheme is effective to represent a neural network with a short hash code, and it is generalizable and efficient on different models.

Fixed-Point Analysis and FPGA Implementation of Deep Neural Network Based Equalizers for High-Speed PON

Adeep neural network based equalizer is proposed to mitigate the intersymbol interference observed in next generation high speed passive optical network (PON) links. The DNN based equalizer is shown to outperform the best known conventional equalizer, the maximum likelihood sequence estimator (MLSE) both in back-back and through fiber experiments. To reduce the hardware complexity of DNN based equalizer for PON systems, we investigate the use of embedded parallelization within a DNN structure having multiple symbol outputs from one DNN. We further investigate using a classification output stage with cross entropy cost to perform joint decision on multiple symbol outputs and demonstrated that the sensitivity gain of such scheme over regression output. To understand the complexity of hardware implementation, the fixed-point DNN based equalizers are developed and implemented in FPGA. The impact of fixed-point resolution on the receiver sensitivity and hardware resource utilization in FPGA implementation is analyzed and reported in detail. We show that a reduction of over 40% in LUTs (look up table) utilization is possible by reducing the DNN's weight resolution from 8-bit to 4-bit while incurring a small penalty in receiver sensitivity.