Implementation Of Deep Neural Networks Research Articles

Skyrmion-mediated nonvolatile ternary memory

Multistate memory systems have the ability to store and process more data in the same physical space as binary memory systems, making them a potential alternative to existing binary memory systems. In the past, it has been demonstrated that voltage-controlled magnetic anisotropy (VCMA) based writing is highly energy-efficient compared to other writing methods used in non-volatile nano-magnetic binary memory systems. In this study, we introduce a new, VCMA-based and skyrmion-mediated non-volatile ternary memory system using a perpendicular magnetic tunnel junction (p-MTJ) in the presence of room temperature thermal perturbation. We have also shown that ternary states {− 1, 0, + 1} can be implemented with three magnetoresistance values obtained from a p-MTJ corresponding to ferromagnetic up, down, and skyrmion state, with 99% switching probability in the presence of room temperature thermal noise in an energy-efficient way, requiring ~ 2 fJ energy on an average for each switching operation. Additionally, we show that our proposed ternary memory demonstrates an improvement in area and energy by at least 2X and ~ 104X respectively, compared to state-of-the-art spin-transfer torque (STT)-based non-volatile magnetic multistate memories. Furthermore, these three states can be potentially utilized for energy-efficient, high-density in-memory quantized deep neural network implementation.

Unveiling the Phase Diagram and Reaction Paths of the Active Model B with the Deep Minimum Action Method.

Nonequilibrium phase transitions are notably difficult to analyze because their mechanisms depend on the system's dynamics in a complex way due to the lack of time-reversal symmetry. To complicate matters, the system's steady-state distribution is unknown in general. Here, the phase diagram of the active Model B is computed with a deep neural network implementation of the geometric minimum action method (gMAM). This approach unveils the unconventional reaction paths and nucleation mechanism in dimensions 1, 2, and 3, by which the system switches between the homogeneous and inhomogeneous phases in the binodal region. Our main findings are (i)the mean time to escape the phase-separated state is (exponentially) extensive in the system size L, but it increases nonmonotonically with L in dimension 1; (ii)the mean time to escape the homogeneous state is always finite, in line with the recent work of Cates and Nardini [Phys. Rev. Lett. 130, 098203 (2023)PRLTAO0031-900710.1103/PhysRevLett.130.098203]; (iii)at fixed L, the active term increases the stability of the homogeneous phase, eventually destroying the phase separation in the binodal for large but finite systems. Our results are particularly relevant for active matter systems in which the number of constituents hardly goes beyond 10^{7} and where finite-size effects matter.

Query-guided generalizable medical image segmentation

The practical implementation of deep neural networks in clinical settings faces hurdles due to variations in data distribution across different centers. While the incorporation of query-guided Transformer has improved performance across diverse tasks, the full scope of their generalization capabilities remains unexplored. Given the ability of the query-guided Transformer to dynamically adjust to individual samples, fulfilling the need for domain generalization, this paper explores the potential of query-based Transformer for cross-center generalization and introduces a novel Query-based Cross-Center medical image Segmentation mechanism (QuCCeS). By integrating a query-guided Transformer into a U-Net-like architecture, QuCCeS utilizes attribution modeling capability of query-guided Transformer decoder for segmentation in fluctuating scenarios with limited data. Additionally, QuCCeS incorporates an auxiliary task with adaptive sample weighting for coarse mask prediction. Experimental results demonstrate QuCCeS’s superior generalization on unseen domains.

Review of Advanced Methods in Hardware Acceleration for Deep Neural Networks

Abstract: Convolutional neural networks have become very efficient in performing tasks like Object Detection providing human like accuracy. However, their practical implementation needs significant hardware resources and memory bandwidth. In recent past a lot of research is being carried out for achieving higher efficiency in implementing such neural networks in hardware. We talk about FPGAs for hardware implementation due to their flexibility for customisation for such neural network architectures. In this paper we will discuss the metrics for efficient hardware accelerator and general methods available for achieving an efficient design. Further, we will discuss the actual methods used by recent research for implementation of deep neural networks particularly for object detection related applications. These methods range from actual ASIC design like TPUs [1] for on chip acceleration, state of the art open source designs like Gemini to methods like hardware reuse, re-configurable nodes and approximation in computations as a trade-off between speed and accuracy. This paper will be a valuable summary for the researchers starting in the field of hardware accelerators design for neural networks

Synapse device based neuromorphic system for biomedical applications.

Despite holding valuable information, unstructured data pose challenges for efficient recognition due to the difficulties in feature extraction using traditional Von-Neumann architecture systems, which are limited by power and time bottlenecks. Although biological neural signals offer crucial insights, they require more effective recognition solutions due to inherent noise and the vast volumes of data. Inspired by the human brain, neuromorphic systems have emerged as promising alternatives because of their parallelism, low power consumption, and error tolerance. By leveraging deep neural networks (DNNs), these systems can recognize imprecise data through two key processes: learning (feature extraction) and testing (feature matching and recognition). During the learning phase, DNNs extract and store unique features such as weight changes in synapse units. In the testing phase, new data are compared with the stored features for recognition. The parallelization of the neuromorphic system enables the efficient processing of large, imprecise datasets with minimal energy consumption. Nevertheless, the hardware implementation is essential for determining the full potential of DNNs. This paper focuses on synapse devices, which are the core units for hardware DNN implementations, and presents a biomedical application example: a rat neural signal recognition system implemented using a synapse device-based neuromorphic system.

2bit Nonlinear Diffractive Deep Neural Network (2bit ND2NN): A quantized nonlinear all-optical diffractive deep neural network implementation

To alleviate the problems of long training time and computational resource consumption in diffractive deep neural networks (D2NNs), we present the 2bit nonlinear diffractive deep neural network (2bit ND2NN) model, which employs the concept of quantized neural networks. 2bit ND2NN converts the phase of each diffraction layer pixel from continuous to discrete values, i.e., the five regions of 0, π/2, π, 3π/2 and 2π. We use the formulas for phase and relative thickness to determine the thickness values of the pixels. Furthermore, the phase and amplitude of neurons in the 2bit ND2NN model are both considered as learnable parameters. We use a revised formula to determine the size of neurons in the model, and we determine the diffraction grating spacing, the number of grating layers, the pixel size and the pixel counts through ablation experiments. Experimental results for image classification indicate that the highest accuracy achieved by 2bit ND2NN on the MNIST and Fashion-MNIST datasets is 97.88 % and 89.28 %, respectively.

Voltage Scaled Low Power DNN Accelerator Design on Reconfigurable Platform

The exponential emergence of Field-Programmable Gate Arrays (FPGAs) has accelerated research on hardware implementation of Deep Neural Networks (DNNs). Among all DNN processors, domain-specific architectures such as Google’s Tensor Processor Unit (TPU) have outperformed conventional GPUs (Graphics Processing Units) and CPUs (Central Processing Units). However, implementing low-power TPUs in reconfigurable hardware remains a challenge in this field. Voltage scaling, a popular approach for energy savings, can be challenging in FPGAs, as it may lead to timing failures if not implemented appropriately. This work presents an ultra-low-power FPGA implementation of a TPU for edge applications. We divide the systolic array of a TPU into different FPGA partitions based on the minimum slack value of different design paths of Multiplier Accumulators (MACs). Each partition uses different near-threshold (NTC) biasing voltages to run its FPGA cores. The biasing voltage for each partition is roughly calculated by the proposed static schemes. However, further calibration of biasing voltage is performed by the proposed runtime scheme. To overcome the timing failure caused by NTC, the MACs with higher minimum slack are placed in lower-voltage partitions, while the MACs with lower minimum slack paths are placed in higher-voltage partitions. The proposed architecture is implemented in a commercial platform, namely Vivado with Xilinx Artix-7 FPGA and academic platform VTR with 22 nm, 45 nm and 130 nm FPGAs. Any timing error caused by NTC can be caught by the Razor flipflop used in each MAC. The proposed voltage-scaled, partitioned systolic array can save 3.1% to 11.6% of dynamic power in Vivado and VTR tools, respectively, depending on the FPGA technology, partition size, number of partitions and biasing voltages. The normalized performance and accuracy of benchmark models running on our low-power TPU are very competitive compared to existing literature.

Retraction: “Implementation of deep neural networks for classifying electroencephalogram signal using fractional S‐transform for epileptic seizure detection”

Retraction: Ashokkumar SR, Anupallavi S, Premkumar M, Jeevanantham V. Implementation of deep neural networks for classifying electroencephalogram signal using fractional S‐transform for epileptic seizure detection. Int J Imaging Syst Technol. 2021;31:895–908. https://doi.org/10.1002/ima.22565The above article, published online on 02 March 2021 in Wiley Online Library (wileyonlinelibrary.com), has been retracted by agreement between the journal Editor‐in‐Chief Dr. Mohamed Seghier, and Wiley Periodicals, LLC. The retraction has been agreed due to major overlap between this article and the following article: Liu G, Zhou W, Geng M. Automatic Seizure Detection Based on S‐Transform and Deep Convolutional Neural Network. Int. J. Neural Syst. 2020;30:1950024. https://doi.org/10.1142/S0129065719500242. Following a complaint from a third party, an investigation by the editorial office and Wiley concluded that Figures 5 and 10 were duplicated from the previously‐published article and that the degree of overlap between the articles' topics, preprocessing methods, and proposed AI tools constitutes plagiarism. The authors responded to an inquiry by the publisher, but did not provide original data. The authors were notified about the retraction but did not respond.

Implementation of deep neural networks learning on unmanned aerial vehicle based remote-sensing

Due to efficient and adaptable data collecting, unmanned aerial vehicle (UAV) has been a popular topic in computer vision (CV) and remote sensing (RS) in recent years. Inspiring by the recent success of deep learning (DL), several enhanced object identification and tracking methods have been broadly applied to a variety of UAV-related applications, including environmental monitoring, precision agriculture, and traffic management. In this research, we present efficient neural network (ENet), a unique deep neural network architecture designed exclusively for jobs demanding low latency operation. ENet is up to quicker, takes fewer floating-point operations per second (FLOPs), has fewer parameters, and offers accuracy comparable to or superior to that of previous models. We have tested it on the street and cityscapes reports on comparisons with current state-of-the-art approaches and the tradeoffs between a network's processing speed and accuracy. We give measurements of the proposed architecture's performance on embedded devices and offer software enhancements that might make ENet even quicker.

A Real-time P-SFA hardware implementation of Deep Neural Networks using FPGA

Machine Learning (ML) algorithms, specifically Artificial Neural Networks (ANNs), have proved their effectiveness in solving complex problems in many different applications and multiple fields. This paper focuses on optimizing the activation function (AF) block of the NN hardware architecture. The AF block used is based on a probability-based sigmoid function approximation block (P-SFA) combined with a novel real-time probability module (PRT) that calculates the probability of the input data. The proposed NN design aims to use the least amount of hardware resources and area while maintaining a high recognition accuracy. The proposed AF module in this work consists of two P-SFA blocks and the PRT component. The architecture proposed for implementing NNs is evaluated on Field Programmable Gate Arrays (FPGAs). The proposed design has achieved a recognition accuracy of 97.84 % on a 6-layer Deep Neural Network (DNN) for the MNIST dataset and a recognition accuracy of 88.58% on a 6-layer DNN for the FMNIST dataset. The proposed AF module has a total area of 1136 LUTs and 327 FFs, a logical critical path delay of 8.853 ns. The power consumption of the P-SFA block is 6 mW and the PRT block is 5 mW.

A Heterogeneous Inference Framework for a Deep Neural Network

Artificial intelligence (AI) is one of the most promising technologies based on machine learning algorithms. In this paper, we propose a workflow for the implementation of deep neural networks. This workflow attempts to combine the flexibility of high-level compilers (HLS)-based networks with the architectural control features of hardware description languages (HDL)-based flows. The architecture consists of a convolutional neural network, SqueezeNet v1.1, and a hard processor system (HPS) that coexists with acceleration hardware to be designed. This methodology allows us to compare solutions based solely on software (PyTorch 1.13.1) and propose heterogeneous inference solutions, taking advantage of the best options within the software and hardware flow. The proposed workflow is implemented on a low-cost field programmable gate array system-on-chip (FPGA SOC) platform, specifically the DE10-Nano development board. We have provided systolic architectural solutions written in OpenCL that are highly flexible and easily tunable to take full advantage of the resources of programmable devices and achieve superior energy efficiencies working with a 32-bit floating point. From a verification point of view, the proposed method is effective, since the reference models in all tests, both for the individual layers and the complete network, have been readily available using packages well known in the development, training, and inference of deep networks.

Image Signal Processing in the Context of Deep Learning Applications

Deep learning accelerators are a specialized sort of hardware architecture designed to enhance the computational efficiency of computers engaged in deep neural networks (DNNs) training. The implementation of DNNs in embedded vision applications might potentially be facilitated by the integration of energy-effective accelerators of deep learning into sensors. The lack of recognition for their significant impact on accuracy is a notable oversight. In previous iterations of deep learning accelerators integrated inside sensors, a common approach was bypassing the image signal processor (ISP). This deviation from the traditional vision pipelines had a detrimental impact on the performance of machine learning models trained on data that had undergone post-ISP processing. In this study, we establish a set of energy-efficient techniques that allow ISP to maximize their advantages while also limiting the covariate shift between the target dataset (RAW images) and the training dataset (ISP-analyzed images). This approach enables the practical use of in-sensor accelerators. To clarify, our results do not minimize the relevance of in-sensor accelerators. Instead, we highlight deficiencies in the methodology used in prior research and propose methodologies that empower in-sensor accelerators to fully exploit their capabilities.

State-Aware Deep Item Response Theory using student facial features.

This paper introduces a novel approach to Item Response Theory (IRT) by incorporating deep learning to analyze student facial expressions to enhance the prediction and understanding of student responses to test items. This research is based on the assertion that students' facial expressions offer crucial insights into their cognitive and affective states during testing, subsequently influencing their item responses. The proposed State-Aware Deep Item Response Theory (SAD-IRT) model introduces a new parameter, the student state parameter, which can be viewed as a relative subjective difficulty parameter. It is latent-regressed from students' facial features while solving test items using state-of-the-art deep learning techniques. In an experiment with 20 students, SAD-IRT boosted prediction performance in students' responses compared to prior models without the student state parameter, including standard IRT and its deep neural network implementation, while maintaining consistent predictions of student ability and item difficulty parameters. The research further illustrates the model's early prediction ability in predicting the student's response result before the student answered. This study holds substantial implications for educational assessment, laying the groundwork for more personalized and effective learning and assessment strategies that consider students' emotional and cognitive states.

Attacks on Recent DNN IP Protection Techniques and Their Mitigation

With the rapid increase in the development of Deep Learning methodologies, Deep Neural Networks (DNNs) are now being commonly deployed in smart systems (e.g. autonomous vehicles) and high-end security applications (e.g. face recognition, biometric authentication, etc.). The training of such DNN models often requires exclusive valuable training datasets, enormous computational resources, and expert fine-tuning skills. Hence, a trained DNN model can be regarded as valuable proprietary Intellectual Property (IP). Piracy of such DNN IPs has emerged as a major concern, with increasing trends of illegal copying and redistribution. A number of mitigation approaches targeting DNN IP protection have been proposed in recent years. In this work, we target two recently proposed DNN IP protection schemes: (a) Chaotic Map theory based encryption of the weight parameters, and (b) traditional block cipher based encryption of the weights. We demonstrate attacks on two recent DNN IP protection techniques, with one technique each belonging to the above-mentioned schemes, under a pragmatic attack model. We also propose a novel DNN IP protection technique based on selective encryption of the weight parameters, termed LEWIP to mitigate the exposed weaknesses, while having low implementation and performance overheads. Finally, we demonstrate the effectiveness of the LEWIP technique against state-of-the-art DNN implementations.

Deep Neural Network Model for Determination of Coal Cutting Resistance and Performance of Bucket-Wheel Excavator Based on the Environmental Properties and Excavation Parameters

In the present paper, we develop a new model, based on the implementation of deep neural networks, for the estimation of a series of excavation parameters, depending on the main environmental and excavation properties. The developed model, with high statistical accuracy (R > 0.79) and small RMSE (<17% of the actual output values), enables the simultaneous assessment of the following excavation parameters: effective capacity Qef, maximum current consumption Imax, maximum power consumption Nmax, maximum force consumption Pmax, maximum energy consumption Emax, and maximum linear and areal cutting resistance, KLmax and KFmax, respectively, based on the impact of the following environmental properties and excavation parameters: coal unit weight, coal compression strength, coal cohesion, friction angle, excavator movement angle in the left and right direction, slice height and thickness, and wheel velocity. The data analyzed in the present paper were previously collected from three neighboring open-pit coal mines in Serbia: Tamnava Western Field, Tamnava Eastern Field, and Field D. These mines have similar geological conditions and coal properties. Additionally, for each output factor, a complex analysis is provided on the impact of the examined input factors, by applying the multiple linear regression method. As far as we are aware, this is the first time such a comprehensive estimation model has been suggested for the operation of a bucket-wheel excavator in the Neogene coal basins. The deep neural network (DNN) model, trained over 300 epochs, shows an MSE range of 6.7–16.5% across various input factors, with distinct evaluations for Imax due to its unique values (4.8–12.5%).

Hardware implementation of deep neural network for seizure prediction

Epilepsy is a neurological disorder characterized by seizures, which are caused by a sudden, uncontrollable electrical disturbance in the brain. Recently, machine learning and deep learning techniques have been used in seizure prediction algorithms, which greatly aids epilepsy patients. Moreover, the usage of hardware that is available for both medical practitioners and epilepsy patients would help enhance the quality of life for epilepsy patients. For this purpose, a Field-programmable Gate Array (FPGA) is used in this work to implement a hardware model of a neural network that is able to predict seizures. In this research, the main aim is to implement an FPGA-based general and patient-specific seizure prediction algorithm that detects seizures for epilepsy patients using Multilayer Perceptron (MLP) neural network models. Moreover, the work will also tackle ways to optimize the FPGA resources and the computational time of the seizure prediction models. The data available for training and testing are raw electroencephalogram (EEG) signal samples provided by the Melbourne-NeuroVista seizure trial and the Melbourne-University AES-MathWorks-NIH Seizure Prediction Challenge. The results show that the general model had an Area under curve (AUC) score of 0.72 while the patient-specific model had an average AUC score of 0.8.

Implementation of Deep Neural Network in the Design of Ethereum Blockchain Scam Token Detection Applications

Implementation of Deep Neural Network in the Design of Ethereum Blockchain Scam Token Detection Applications

Implementation of deep neural networks and statistical methods to predict the resilient modulus of soils

ABSTRACT The Resilient Modulus (Mr) is perhaps the most relevant and widely used parameter to characterise the soil behaviour under repetitive loading for pavement applications. Accordingly, it is a crucial parameter controlling the mechanistic-empirical pavement design. Nonetheless, determining the Mr by laboratory tests is not always possible due to the high consumption of time and financial resources. Thus, developing new indirect approaches for estimating the MR is necessary. Precisely, this article investigates the application of Deep Neural Networks (DNNs) and statistical methods to predict the Mr of soils. For that purpose, the Long-Term Pavement Performance (LTPP) database was implemented. It includes 64 701 datasets resulting from coarse-grained and fine-grained soil samples considering a wide range of grain size distribution and subjected to different stress levels. The input parameters were the bulk stress, octahedral shear stress, and the percentage of soil particles passing through the different sieves (3”, 2”, 3/2”, 1”, 3/4”, 1/2”, 3/8”, No. 4, No. 10, No. 40, No. 80, and No. 200) and the output was the Mr. The results suggest that while conventional mathematical models are unable to predict the influence of the grain size distribution and stress level on the Mr, the proposed DNNs were able to reproduce very accurate predictions. Notably, the proposed computational models have been uploaded to a GitHub repository and have become a valuable tool for forecasting the Mr when experimental measurements are not feasible.

Integration of Ag-CBRAM crossbars and Mott ReLU neurons for efficient implementation of deep neural networks in hardware

In-memory computing with emerging non-volatile memory devices (eNVMs) has shown promising results in accelerating matrix-vector multiplications. However, activation function calculations are still being implemented with general processors or large and complex neuron peripheral circuits. Here, we present the integration of Ag-based conductive bridge random access memory (Ag-CBRAM) crossbar arrays with Mott rectified linear unit (ReLU) activation neurons for scalable, energy and area-efficient hardware (HW) implementation of deep neural networks. We develop Ag-CBRAM devices that can achieve a high ON/OFF ratio and multi-level programmability. Compact and energy-efficient Mott ReLU neuron devices implementing ReLU activation function are directly connected to the columns of Ag-CBRAM crossbars to compute the output from the weighted sum current. We implement convolution filters and activations for VGG-16 using our integrated HW and demonstrate the successful generation of feature maps for CIFAR-10 images in HW. Our approach paves a new way toward building a highly compact and energy-efficient eNVMs-based in-memory computing system.

Advancements in On-Device Deep Neural Networks

In recent years, rapid advancements in both hardware and software technologies have resulted in the ability to execute artificial intelligence (AI) algorithms on low-resource devices. The combination of high-speed, low-power electronic hardware and efficient AI algorithms is driving the emergence of on-device AI. Deep neural networks (DNNs) are highly effective AI algorithms used for identifying patterns in complex data. DNNs, however, contain many parameters and operations that make them computationally intensive to execute. Accordingly, DNNs are usually executed on high-resource backend processors. This causes an increase in data processing latency and energy expenditure. Therefore, modern strategies are being developed to facilitate the implementation of DNNs on devices with limited resources. This paper presents a detailed review of the current methods and structures that have been developed to deploy DNNs on devices with limited resources. Firstly, an overview of DNNs is presented. Next, the methods used to implement DNNs on resource-constrained devices are explained. Following this, the existing works reported in the literature on the execution of DNNs on low-resource devices are reviewed. The reviewed works are classified into three categories: software, hardware, and hardware/software co-design. Then, a discussion on the reviewed approaches is given, followed by a list of challenges and future prospects of on-device AI, together with its emerging applications.