Implementation Of Deep Neural Networks Research Articles

Review of Advanced Methods in Hardware Acceleration for Deep Neural Networks

Abstract: Convolutional neural networks have become very efficient in performing tasks like Object Detection providing human like accuracy. However, their practical implementation needs significant hardware resources and memory bandwidth. In recent past a lot of research is being carried out for achieving higher efficiency in implementing such neural networks in hardware. We talk about FPGAs for hardware implementation due to their flexibility for customisation for such neural network architectures. In this paper we will discuss the metrics for efficient hardware accelerator and general methods available for achieving an efficient design. Further, we will discuss the actual methods used by recent research for implementation of deep neural networks particularly for object detection related applications. These methods range from actual ASIC design like TPUs [1] for on chip acceleration, state of the art open source designs like Gemini to methods like hardware reuse, re-configurable nodes and approximation in computations as a trade-off between speed and accuracy. This paper will be a valuable summary for the researchers starting in the field of hardware accelerators design for neural networks

2bit Nonlinear Diffractive Deep Neural Network (2bit ND2NN): A quantized nonlinear all-optical diffractive deep neural network implementation

To alleviate the problems of long training time and computational resource consumption in diffractive deep neural networks (D2NNs), we present the 2bit nonlinear diffractive deep neural network (2bit ND2NN) model, which employs the concept of quantized neural networks. 2bit ND2NN converts the phase of each diffraction layer pixel from continuous to discrete values, i.e., the five regions of 0, π/2, π, 3π/2 and 2π. We use the formulas for phase and relative thickness to determine the thickness values of the pixels. Furthermore, the phase and amplitude of neurons in the 2bit ND2NN model are both considered as learnable parameters. We use a revised formula to determine the size of neurons in the model, and we determine the diffraction grating spacing, the number of grating layers, the pixel size and the pixel counts through ablation experiments. Experimental results for image classification indicate that the highest accuracy achieved by 2bit ND2NN on the MNIST and Fashion-MNIST datasets is 97.88 % and 89.28 %, respectively.

Voltage Scaled Low Power DNN Accelerator Design on Reconfigurable Platform

The exponential emergence of Field-Programmable Gate Arrays (FPGAs) has accelerated research on hardware implementation of Deep Neural Networks (DNNs). Among all DNN processors, domain-specific architectures such as Google’s Tensor Processor Unit (TPU) have outperformed conventional GPUs (Graphics Processing Units) and CPUs (Central Processing Units). However, implementing low-power TPUs in reconfigurable hardware remains a challenge in this field. Voltage scaling, a popular approach for energy savings, can be challenging in FPGAs, as it may lead to timing failures if not implemented appropriately. This work presents an ultra-low-power FPGA implementation of a TPU for edge applications. We divide the systolic array of a TPU into different FPGA partitions based on the minimum slack value of different design paths of Multiplier Accumulators (MACs). Each partition uses different near-threshold (NTC) biasing voltages to run its FPGA cores. The biasing voltage for each partition is roughly calculated by the proposed static schemes. However, further calibration of biasing voltage is performed by the proposed runtime scheme. To overcome the timing failure caused by NTC, the MACs with higher minimum slack are placed in lower-voltage partitions, while the MACs with lower minimum slack paths are placed in higher-voltage partitions. The proposed architecture is implemented in a commercial platform, namely Vivado with Xilinx Artix-7 FPGA and academic platform VTR with 22 nm, 45 nm and 130 nm FPGAs. Any timing error caused by NTC can be caught by the Razor flipflop used in each MAC. The proposed voltage-scaled, partitioned systolic array can save 3.1% to 11.6% of dynamic power in Vivado and VTR tools, respectively, depending on the FPGA technology, partition size, number of partitions and biasing voltages. The normalized performance and accuracy of benchmark models running on our low-power TPU are very competitive compared to existing literature.

Retraction: “Implementation of deep neural networks for classifying electroencephalogram signal using fractional S‐transform for epileptic seizure detection”

Retraction: Ashokkumar SR, Anupallavi S, Premkumar M, Jeevanantham V. Implementation of deep neural networks for classifying electroencephalogram signal using fractional S‐transform for epileptic seizure detection. Int J Imaging Syst Technol. 2021;31:895–908. https://doi.org/10.1002/ima.22565The above article, published online on 02 March 2021 in Wiley Online Library (wileyonlinelibrary.com), has been retracted by agreement between the journal Editor‐in‐Chief Dr. Mohamed Seghier, and Wiley Periodicals, LLC. The retraction has been agreed due to major overlap between this article and the following article: Liu G, Zhou W, Geng M. Automatic Seizure Detection Based on S‐Transform and Deep Convolutional Neural Network. Int. J. Neural Syst. 2020;30:1950024. https://doi.org/10.1142/S0129065719500242. Following a complaint from a third party, an investigation by the editorial office and Wiley concluded that Figures 5 and 10 were duplicated from the previously‐published article and that the degree of overlap between the articles' topics, preprocessing methods, and proposed AI tools constitutes plagiarism. The authors responded to an inquiry by the publisher, but did not provide original data. The authors were notified about the retraction but did not respond.

Implementation of deep neural networks learning on unmanned aerial vehicle based remote-sensing

Due to efficient and adaptable data collecting, unmanned aerial vehicle (UAV) has been a popular topic in computer vision (CV) and remote sensing (RS) in recent years. Inspiring by the recent success of deep learning (DL), several enhanced object identification and tracking methods have been broadly applied to a variety of UAV-related applications, including environmental monitoring, precision agriculture, and traffic management. In this research, we present efficient neural network (ENet), a unique deep neural network architecture designed exclusively for jobs demanding low latency operation. ENet is up to quicker, takes fewer floating-point operations per second (FLOPs), has fewer parameters, and offers accuracy comparable to or superior to that of previous models. We have tested it on the street and cityscapes reports on comparisons with current state-of-the-art approaches and the tradeoffs between a network's processing speed and accuracy. We give measurements of the proposed architecture's performance on embedded devices and offer software enhancements that might make ENet even quicker.

A Real-time P-SFA hardware implementation of Deep Neural Networks using FPGA

Machine Learning (ML) algorithms, specifically Artificial Neural Networks (ANNs), have proved their effectiveness in solving complex problems in many different applications and multiple fields. This paper focuses on optimizing the activation function (AF) block of the NN hardware architecture. The AF block used is based on a probability-based sigmoid function approximation block (P-SFA) combined with a novel real-time probability module (PRT) that calculates the probability of the input data. The proposed NN design aims to use the least amount of hardware resources and area while maintaining a high recognition accuracy. The proposed AF module in this work consists of two P-SFA blocks and the PRT component. The architecture proposed for implementing NNs is evaluated on Field Programmable Gate Arrays (FPGAs). The proposed design has achieved a recognition accuracy of 97.84 % on a 6-layer Deep Neural Network (DNN) for the MNIST dataset and a recognition accuracy of 88.58% on a 6-layer DNN for the FMNIST dataset. The proposed AF module has a total area of 1136 LUTs and 327 FFs, a logical critical path delay of 8.853 ns. The power consumption of the P-SFA block is 6 mW and the PRT block is 5 mW.

A Heterogeneous Inference Framework for a Deep Neural Network

Artificial intelligence (AI) is one of the most promising technologies based on machine learning algorithms. In this paper, we propose a workflow for the implementation of deep neural networks. This workflow attempts to combine the flexibility of high-level compilers (HLS)-based networks with the architectural control features of hardware description languages (HDL)-based flows. The architecture consists of a convolutional neural network, SqueezeNet v1.1, and a hard processor system (HPS) that coexists with acceleration hardware to be designed. This methodology allows us to compare solutions based solely on software (PyTorch 1.13.1) and propose heterogeneous inference solutions, taking advantage of the best options within the software and hardware flow. The proposed workflow is implemented on a low-cost field programmable gate array system-on-chip (FPGA SOC) platform, specifically the DE10-Nano development board. We have provided systolic architectural solutions written in OpenCL that are highly flexible and easily tunable to take full advantage of the resources of programmable devices and achieve superior energy efficiencies working with a 32-bit floating point. From a verification point of view, the proposed method is effective, since the reference models in all tests, both for the individual layers and the complete network, have been readily available using packages well known in the development, training, and inference of deep networks.

Image Signal Processing in the Context of Deep Learning Applications

Deep learning accelerators are a specialized sort of hardware architecture designed to enhance the computational efficiency of computers engaged in deep neural networks (DNNs) training. The implementation of DNNs in embedded vision applications might potentially be facilitated by the integration of energy-effective accelerators of deep learning into sensors. The lack of recognition for their significant impact on accuracy is a notable oversight. In previous iterations of deep learning accelerators integrated inside sensors, a common approach was bypassing the image signal processor (ISP). This deviation from the traditional vision pipelines had a detrimental impact on the performance of machine learning models trained on data that had undergone post-ISP processing. In this study, we establish a set of energy-efficient techniques that allow ISP to maximize their advantages while also limiting the covariate shift between the target dataset (RAW images) and the training dataset (ISP-analyzed images). This approach enables the practical use of in-sensor accelerators. To clarify, our results do not minimize the relevance of in-sensor accelerators. Instead, we highlight deficiencies in the methodology used in prior research and propose methodologies that empower in-sensor accelerators to fully exploit their capabilities.

State-Aware Deep Item Response Theory using student facial features.

This paper introduces a novel approach to Item Response Theory (IRT) by incorporating deep learning to analyze student facial expressions to enhance the prediction and understanding of student responses to test items. This research is based on the assertion that students' facial expressions offer crucial insights into their cognitive and affective states during testing, subsequently influencing their item responses. The proposed State-Aware Deep Item Response Theory (SAD-IRT) model introduces a new parameter, the student state parameter, which can be viewed as a relative subjective difficulty parameter. It is latent-regressed from students' facial features while solving test items using state-of-the-art deep learning techniques. In an experiment with 20 students, SAD-IRT boosted prediction performance in students' responses compared to prior models without the student state parameter, including standard IRT and its deep neural network implementation, while maintaining consistent predictions of student ability and item difficulty parameters. The research further illustrates the model's early prediction ability in predicting the student's response result before the student answered. This study holds substantial implications for educational assessment, laying the groundwork for more personalized and effective learning and assessment strategies that consider students' emotional and cognitive states.

Attacks on Recent DNN IP Protection Techniques and Their Mitigation

With the rapid increase in the development of Deep Learning methodologies, Deep Neural Networks (DNNs) are now being commonly deployed in smart systems (e.g. autonomous vehicles) and high-end security applications (e.g. face recognition, biometric authentication, etc.). The training of such DNN models often requires exclusive valuable training datasets, enormous computational resources, and expert fine-tuning skills. Hence, a trained DNN model can be regarded as valuable proprietary Intellectual Property (IP). Piracy of such DNN IPs has emerged as a major concern, with increasing trends of illegal copying and redistribution. A number of mitigation approaches targeting DNN IP protection have been proposed in recent years. In this work, we target two recently proposed DNN IP protection schemes: (a) Chaotic Map theory based encryption of the weight parameters, and (b) traditional block cipher based encryption of the weights. We demonstrate attacks on two recent DNN IP protection techniques, with one technique each belonging to the above-mentioned schemes, under a pragmatic attack model. We also propose a novel DNN IP protection technique based on selective encryption of the weight parameters, termed LEWIP to mitigate the exposed weaknesses, while having low implementation and performance overheads. Finally, we demonstrate the effectiveness of the LEWIP technique against state-of-the-art DNN implementations.

Deep Neural Network Model for Determination of Coal Cutting Resistance and Performance of Bucket-Wheel Excavator Based on the Environmental Properties and Excavation Parameters

In the present paper, we develop a new model, based on the implementation of deep neural networks, for the estimation of a series of excavation parameters, depending on the main environmental and excavation properties. The developed model, with high statistical accuracy (R > 0.79) and small RMSE (<17% of the actual output values), enables the simultaneous assessment of the following excavation parameters: effective capacity Qef, maximum current consumption Imax, maximum power consumption Nmax, maximum force consumption Pmax, maximum energy consumption Emax, and maximum linear and areal cutting resistance, KLmax and KFmax, respectively, based on the impact of the following environmental properties and excavation parameters: coal unit weight, coal compression strength, coal cohesion, friction angle, excavator movement angle in the left and right direction, slice height and thickness, and wheel velocity. The data analyzed in the present paper were previously collected from three neighboring open-pit coal mines in Serbia: Tamnava Western Field, Tamnava Eastern Field, and Field D. These mines have similar geological conditions and coal properties. Additionally, for each output factor, a complex analysis is provided on the impact of the examined input factors, by applying the multiple linear regression method. As far as we are aware, this is the first time such a comprehensive estimation model has been suggested for the operation of a bucket-wheel excavator in the Neogene coal basins. The deep neural network (DNN) model, trained over 300 epochs, shows an MSE range of 6.7–16.5% across various input factors, with distinct evaluations for Imax due to its unique values (4.8–12.5%).

Hardware implementation of deep neural network for seizure prediction

Epilepsy is a neurological disorder characterized by seizures, which are caused by a sudden, uncontrollable electrical disturbance in the brain. Recently, machine learning and deep learning techniques have been used in seizure prediction algorithms, which greatly aids epilepsy patients. Moreover, the usage of hardware that is available for both medical practitioners and epilepsy patients would help enhance the quality of life for epilepsy patients. For this purpose, a Field-programmable Gate Array (FPGA) is used in this work to implement a hardware model of a neural network that is able to predict seizures. In this research, the main aim is to implement an FPGA-based general and patient-specific seizure prediction algorithm that detects seizures for epilepsy patients using Multilayer Perceptron (MLP) neural network models. Moreover, the work will also tackle ways to optimize the FPGA resources and the computational time of the seizure prediction models. The data available for training and testing are raw electroencephalogram (EEG) signal samples provided by the Melbourne-NeuroVista seizure trial and the Melbourne-University AES-MathWorks-NIH Seizure Prediction Challenge. The results show that the general model had an Area under curve (AUC) score of 0.72 while the patient-specific model had an average AUC score of 0.8.

Implementation of Deep Neural Network in the Design of Ethereum Blockchain Scam Token Detection Applications

Implementation of Deep Neural Network in the Design of Ethereum Blockchain Scam Token Detection Applications

Implementation of deep neural networks and statistical methods to predict the resilient modulus of soils

ABSTRACT The Resilient Modulus (Mr) is perhaps the most relevant and widely used parameter to characterise the soil behaviour under repetitive loading for pavement applications. Accordingly, it is a crucial parameter controlling the mechanistic-empirical pavement design. Nonetheless, determining the Mr by laboratory tests is not always possible due to the high consumption of time and financial resources. Thus, developing new indirect approaches for estimating the MR is necessary. Precisely, this article investigates the application of Deep Neural Networks (DNNs) and statistical methods to predict the Mr of soils. For that purpose, the Long-Term Pavement Performance (LTPP) database was implemented. It includes 64 701 datasets resulting from coarse-grained and fine-grained soil samples considering a wide range of grain size distribution and subjected to different stress levels. The input parameters were the bulk stress, octahedral shear stress, and the percentage of soil particles passing through the different sieves (3”, 2”, 3/2”, 1”, 3/4”, 1/2”, 3/8”, No. 4, No. 10, No. 40, No. 80, and No. 200) and the output was the Mr. The results suggest that while conventional mathematical models are unable to predict the influence of the grain size distribution and stress level on the Mr, the proposed DNNs were able to reproduce very accurate predictions. Notably, the proposed computational models have been uploaded to a GitHub repository and have become a valuable tool for forecasting the Mr when experimental measurements are not feasible.

Integration of Ag-CBRAM crossbars and Mott ReLU neurons for efficient implementation of deep neural networks in hardware

In-memory computing with emerging non-volatile memory devices (eNVMs) has shown promising results in accelerating matrix-vector multiplications. However, activation function calculations are still being implemented with general processors or large and complex neuron peripheral circuits. Here, we present the integration of Ag-based conductive bridge random access memory (Ag-CBRAM) crossbar arrays with Mott rectified linear unit (ReLU) activation neurons for scalable, energy and area-efficient hardware (HW) implementation of deep neural networks. We develop Ag-CBRAM devices that can achieve a high ON/OFF ratio and multi-level programmability. Compact and energy-efficient Mott ReLU neuron devices implementing ReLU activation function are directly connected to the columns of Ag-CBRAM crossbars to compute the output from the weighted sum current. We implement convolution filters and activations for VGG-16 using our integrated HW and demonstrate the successful generation of feature maps for CIFAR-10 images in HW. Our approach paves a new way toward building a highly compact and energy-efficient eNVMs-based in-memory computing system.

Advancements in On-Device Deep Neural Networks

In recent years, rapid advancements in both hardware and software technologies have resulted in the ability to execute artificial intelligence (AI) algorithms on low-resource devices. The combination of high-speed, low-power electronic hardware and efficient AI algorithms is driving the emergence of on-device AI. Deep neural networks (DNNs) are highly effective AI algorithms used for identifying patterns in complex data. DNNs, however, contain many parameters and operations that make them computationally intensive to execute. Accordingly, DNNs are usually executed on high-resource backend processors. This causes an increase in data processing latency and energy expenditure. Therefore, modern strategies are being developed to facilitate the implementation of DNNs on devices with limited resources. This paper presents a detailed review of the current methods and structures that have been developed to deploy DNNs on devices with limited resources. Firstly, an overview of DNNs is presented. Next, the methods used to implement DNNs on resource-constrained devices are explained. Following this, the existing works reported in the literature on the execution of DNNs on low-resource devices are reviewed. The reviewed works are classified into three categories: software, hardware, and hardware/software co-design. Then, a discussion on the reviewed approaches is given, followed by a list of challenges and future prospects of on-device AI, together with its emerging applications.

Implementation of Deep Neural Networks for performance prediction and optimization of a porous volumetric solar receiver considering mechanical safety

The volumetric solar receiver is a major component of concentrating solar thermal systems. Proper selection of design and operating parameters is necessary to obtain the best receiver performance. The current work models and optimizes the hydraulic, thermal, and mechanical performance of a porous volumetric solar receiver using the Deep Neural Network (DNN) technique. The dataset for DNN training is generated by numerical simulations and porosity and pore size of foam structure, inlet fluid velocity and absorber length are selected as the input parameters. The pressure drop, outlet fluid temperature and maximum failure index are selected as the output variables. The trained model demonstrates high accuracy in performance prediction for any combination of input variables in the trained range. Further, the present work proposes the integration of the DNN technique with Genetic Algorithm for the constrained multi-objective optimization of the absorber performance with a novel constraint of maximum failure index to select configurations preventing mechanical failure. Such integration is found to be nearly 524 times faster than the conventional direct approach, demonstrating a significant decrease in the time needed for optimization. Furthermore, the potential of the Technique for Order of Preference by Similarity to Ideal Solution in selecting the desired configuration from the Pareto optimal solutions as per individual requirements is also investigated. The optimization results indicate that the optimum values for porosity, pore size, and absorber length are in the range of 0.9–0.93, 2–4 mm, and 0.01–0.06 m, respectively, when all variables are optimized simultaneously. Furthermore, the optimal input velocity is observed to be close to the lower bound of 0.4 m/s while the corresponding maximum failure index varies from 0.346 to 0.836, indicating mechanically safe configurations. The high accuracy and faster optimization results obtained in the present work demonstrate the ability of the proposed approach in implementing quick and computationally cheap optimization studies for improved and mechanically safe absorber designs.

An Empirical Approach to Enhance Performance for Scalable CORDIC-Based Deep Neural Networks

Practical implementation of deep neural networks (DNNs) demands significant hardware resources, necessitating high computational power and memory bandwidth. While existing field-programmable gate array (FPGA)–based DNN accelerators are primarily optimized for fast single-task performance, cost, energy efficiency, and overall throughput are crucial considerations for their practical use in various applications. This article proposes a performance-centric pipeline Coordinate Rotation Digital Computer (CORDIC)–based MAC unit and implements a scalable CORDIC-based DNN architecture that is area- and power-efficient and has high throughput. The CORDIC-based neuron engine uses bit-rounding to maintain input-output precision and minimal hardware resource overhead. The results demonstrate the versatility of the proposed pipelined MAC, which operates at 460 MHz and allows for higher network throughput. A software-based implementation platform evaluates the proposed MAC operation’s accuracy for more extensive neural networks and complex datasets. The DNN accelerator with parameterized and modular layer-multiplexed architecture is designed. Empirical evaluation through Pareto analysis is used to improve the efficiency of DNN implementations by fixing the arithmetic precision and optimal pipeline stages. The proposed architecture utilizes layer-multiplexing, a technique that effectively reuses a single DNN layer to enhance efficiency while maintaining modularity and adaptability for integrating various network configurations. The proposed CORDIC MAC-based DNN architecture is scalable for any bit-precision network size, and the DNN accelerator is prototyped using the Xilinx Virtex-7 VC707 FPGA board, operating at 66 MHz. The proposed design does not use any Xilinx macros, making it easily adaptable for ASIC implementation. Compared with state-of-the-art designs, the proposed design reduces resource use by 45% and power consumption by 4× without sacrificing performance. The accelerator is validated using the MNIST dataset, achieving 95.06% accuracy, only 0.35% less than other cutting-edge implementations.

A Survey on Fault-Tolerant Methodologies for Deep Neural Networks

Asignificant rise in Artificial Intelligence (AI) has impacted many applications around us, so much so that AI has now been increasingly used in safety-critical applications. AI at the edge is the reality, which means performing the data computation closer to the source of the data, as opposed to performing it on the cloud. Safety-critical applications have strict reliability requirements; therefore, it is essential that AI models running on the edge (i.e., hardware) must fulfill the required safety standards. In the vast field of AI, Deep Neural Networks (DNNs) are the focal point of this survey as it has continued to produce extraordinary outcomes in various applications i.e. medical, automotive, aerospace, defense, etc. Traditional reliability techniques for DNNs implementation are not always practical, as they fail to exploit the unique characteristics of the DNNs. Furthermore, it is also essential to understand the targeted edge hardware because the impact of the faults can be different in ASICs and FPGAs. Therefore, in this survey, first, we have examined the impact of the fault in ASICs and FPGAs, and then we seek to provide a glimpse of the recent progress made towards the fault-tolerant DNNs. We have discussed several factors that can impact the reliability of the DNNs. Further, we have extended this discussion to shed light on many state-of-the-art fault mitigation techniques for DNNs.

FPGA-based implementation of deep neural network using stochastic computing

A serious challenge in artificial real-time applications is the hardware implementation of deep neural networks (DNN). Among various methods, stochastic computing (SC)-based implementations received tremendous attention due to the low hardware overhead. However, the slow convergence rate is a major problem in SC-based neural networks’ implementation, and millions of clock cycles are required to generate a relatively accurate output. The reconfigurability and parallel nature of field programmable gate array (FPGA) chips make them a preferable platform for SC-based DNN implementation. A fully or semi-parallel implementation of DNNs requires extensive hardware resources. In this paper, an efficient method for DNN implementation on an FPGA chip is presented to address these problems. The FPGA chip reconfiguration feature allows a DNN with several different neurons and topologies to be implemented on a single chip. Convergence time is significantly reduced by limiting the length of the stochastic bitstreams and establishing synchronization between the processing units based on precise timing. Furthermore, due to the limited number of input–output pins in the FPGA chip, a sequential architecture is proposed wherein the DNN inputs enter through only three 8-bit ports. This makes it possible to implement DNNs for image-processing applications. The proposed method is implemented using the Verilog hardware description language on the Xilinx FPGA Virtex-7 xc7v2000t chip. The results show a more than 82% reduction in hardware resources and the minimum rate of power consumption compared to state-of-the-art methods. In addition, the average error rate of the implemented DNN is reduced by 2%.