Deep Neural Network Topology Research Articles

Intelligent fault diagnosis of machinery based on hybrid deep learning with multi temporal correlation feature fusion

AbstractWith the advent of intelligent manufacturing era, higher requirements are put forward for the fault diagnosis technology of machinery. The existing data‐driven approaches either rely on specialized empirical knowledge for feature analysis, or adopt single deep neural network topology structure for automatic feature extraction with compromise of certain information loss especially the time‐series information's sacrifice, which both eventually affect the diagnosis accuracy. To address the issue, this paper proposes a novel multi‐temporal correlation feature fusion net (MTCFF‐Net) for intelligent fault diagnosis, which can capture and retain time‐series fault feature information from different dimensions. MTCFF‐Net contains four sub‐networks, which are long and short‐term memory (LSTM) sub‐network, Gramian angular summation field (GASF)‐GhostNet sub‐network and Markov transition field (MTF)‐GhostNet sub‐network and feature fusion sub‐network. Features of different dimensional are extracted through parallel LSTM sub‐network, GASF‐GhostNet sub‐network and MTF‐GhostNet sub‐network, and then fused by feature fusion sub‐network for accurate fault diagnosis. Two fault diagnosis experimental studies on bearings are implemented to validate the effectiveness and generalization of the proposed MTCFF‐Net. Experimental results demonstrate that the proposed model is superior to other comparative approaches.

AutoMap: Automatic Mapping of Neural Networks to Deep Learning Accelerators for Edge Devices

Emerging deep neural networks (DNNs) have been emerging in applications (object detection, automatic speech recognition, etc.) deployed on edge devices. To improve the energy efficiency of edge devices, domain specific deep learning accelerators (DLAs) are designed with limited on-chip resources. The manifold DLA designs and evolving DNN topologies bring challenges for applications mapping and scheduling on hardware resources. In this paper, we propose an automatic DNN mapping framework named AutoMap given the hardware backend information. Firstly, a computational graph representation called Extended Directed Weighted Graph (EDWG) is proposed, which realizes unified expression for both spatial and temporal network inter-layer connections. Secondly, an associated partitioner is implemented for splitting an EDWG into subEDWGs, which incorporates the on-chip memory constraint and facilitates weight data reuse on chip. Lastly, a dynamic memory allocation strategy is utilized to alleviate the feature storing burden introduced by the multivarious network sizes and connections. Compared to the baseline mapping methods, experimental results show that our proposed automatic mapping framework can help to speedup the execution of several DNNs on state-of-the-art DLAs, ranging from 1.27x to 3.45x. The utilization of the PE array can increase from 20% to 64%.

심층신경망 기반의 무릎 생체역학 변인 추정기

Knee contact forces and knee stiffness are biomechanical factors worth considering for walking in knee osteoarthritis patients. However, it is challenging to acquire these factors in real time; thus, making it difficult to use them in robotic rehabilitation and assistive systems. This study investigated whether trained deep neural networks (DNNs) can capture the biomechanical factors only using kinematics during gait, which is possible to measure via sensors in real time. A public dataset of walking on the ground was analyzed through biomechanical analysis to train and test DNNs. Using the training dataset, several DNN topologies were explored via Bayesian optimization to tune the hyperparameters. After optimization, DNNs were trained to estimate the biomechanical factors in a supervised manner. The trained DNNs were then evaluated using two new datasets, which were not used in the training process. The trained DNNs estimated the biomechanical factors with a high level of accuracy in both types of test datasets. Results confirmed that DNNs can estimate the biomechanical factors based on only kinematics during gait.

Design Space Exploration of FPGA-Based System With Multiple DNN Accelerators

Many emerging systems concurrently execute multiple applications that use deep neural network (DNN) as a key portion of the computation. To speedup the execution of such DNNs, various hardware accelerators have been proposed in recent works. Deep learning processor unit (DPU) from Xilinx is one such accelerator targeted for field programmable gate array (FPGA)-based systems. We study the runtime and energy consumption for different DNNs on a range of DPU configurations and derive useful insights. Using these insights, we formulate a design space exploration (DSE) strategy to explore tradeoffs in accuracy, runtime, cost, and energy consumption arising due to flexibility in choosing DNN topology, DPU configuration, and FPGA model. The proposed strategy provides a reduction of 28× in the number of design points to be simulated and 23× in the pruning time.

TD-SRAM: Time-Domain-Based In-Memory Computing Macro for Binary Neural Networks

In-Memory Computing (IMC), which takes advantage of analog multiplication-accumulation (MAC) insides memory, is promising to alleviate the Von-Neumann bottleneck and improve the energy efficiency of deep neural networks (DNNs). Since the time-domain (TD) computing is also an energy-efficient analog computing paradigm, we present an 8kb mixed-signal IMC macro, TD-SRAM, by combining IMC with TD computing. A dual-edge single input (DESI) TD computing topology is proposed, which can significantly improve the area and power efficiencies of TD cell. The TD-SRAM bitcell consisting of a 6T DESI based TD cell and a 6T-SRAM cell supports binary DNNs. In the IMC mode, 60 columns work in parallel and 96-input binary-MAC operations are processed in each column. Implemented in a standard 40-nm CMOS process, the TD-SRAM achieves the high energy efficiency of 537 TOPS/W at 0.9-V supply. With different DNN topologies, the test chips achieve the accuracy of 95.90%-98.00% with a dual 2-bit time-to-digital converter (TDC) in the MNIST dataset.

Dynamic parameters identification for sliding joints of surface grinder based on deep neural network modeling

Dynamic parameters of joints are indispensable factors affecting performance of machine tools. In order to obtain the stiffness and damping of sliding joints between the working platform and the machine tool body of the surface grinder, a new method of dynamic parameters identification is proposed that based on deep neural network (DNN) modeling. Firstly, the DNN model of dynamic parameters for working platform-machine tool body sliding joints is established by taking the stiffness and damping parameters as the input and the natural frequencies as the output. Secondly, the number of hidden layers in DNN topology is optimally selected in order to the optimal training results. Thirdly, combining the predicted results by DNN model with experimental results by modal test, the stiffness and damping are identified via cuckoo search algorithm. Finally, the relative error between the predicted and experimental results is less than 2.2%, which achieves extremely high prediction precision; and thereby indicates the feasibility and effectiveness of the proposed method.

Deep neural networks architecture driven by problem-specific information

Deep learning provides a variety of neural network-based models, known as deep neural networks (DNNs), which are being successfully used in several domains to build highly accurate predictors. A key factor which usually makes DNNs to outperform traditional machine learning models is the amount of data that is nowadays accessible and available. Nevertheless, there are other factors linked to DNNs topologies that may also have influence on the predictive performance of DNN models. In particular, fully connected deep neural networks (fc-DNNs) typically struggle in achieving good performance rates when applied to small datasets. This is due to the high number of parameters which need to be learned when training this kind of models, which makes them prone to over-fitting issues. In this paper, authors propose the use of problem-specific information in order to impose constraints to network architecture so that a fc-DNN is transformed into a partially connected DNN (pc-DNN), in such a way that network topology is driven by prior knowledge. This work compares two baseline models, the elastic net and fc-DNNs, to pc-DNNs applied on three synthetic datasets with different number of samples. Synthetic data was generated to estimate the goodness of using problem-specific information to drive network architectures. Furthermore, a similar analysis is performed herein on a real-world problem dataset to show the benefits of pc-DNN models in term of predictive performance. The results of the analysis showed that pc-DNNs with built-in problem-specific information clearly outperformed the elastic net and fc-DNNs in most of the datasets used, in either synthetic or real-world problems. The pc-DNNs turned out to be a useful model, especially when it is applied to small- or medium-size datasets, on which it significantly outperformed the baseline models considered in this study. Specifically, the pc-DNNs achieved AUC and MSE improvement rates of ( $$8.21\%$$ , $$19.79\%$$ ) and ( $$6.65\%$$ , $$20.54\%$$ ) in small- and medium-size datasets for both case studies analyzed, the synthetic and real-world problem, respectively.

User Driven FPGA-Based Design Automated Framework of Deep Neural Networks for Low-Power Low-Cost Edge Computing

Deep Learning techniques have been successfully applied to solve many Artificial Intelligence (AI) applications problems. However, owing to topologies with many hidden layers, Deep Neural Networks (DNNs) have high computational complexity, which makes their deployment difficult in contexts highly constrained by requirements such as performance, real-time processing, or energy efficiency. Numerous hardware/software optimization techniques using GPUs, ASICs, and reconfigurable computing (i.e, FPGAs), have been proposed in the literature. With FPGAs, very specialized architectures have been developed to provide an optimal balance between high-speed and low power. However, when targeting edge computing, user requirements and hardware constraints must be efficiently met. Therefore, in this work, we only focus on reconfigurable embedded systems based on the Xilinx ZYNQ SoC and popular DNNs that can be implemented on Embedded Edge improving performance per watt while maintaining accuracy. In this context, we propose an automated framework for the implementation of hardware-accelerated DNN architectures. This framework provides an end-to-end solution that facilitates the efficient deployment of topologies on FPGAs by combining custom hardware scalability with optimization strategies. Cutting-edge comparisons and experimental results demonstrate that the architectures developed by our framework offer the best compromise between performance, energy consumption, and system costs. For instance, the low power (0.266W) DNN topologies generated for the MNIST database achieved a high throughput of 3,626 FPS.

A Novel Deep Neural Network Topology for Parametric Modeling of Passive Microwave Components

Artificial neural network technique has gained recognition as a powerful technique in microwave modeling and design. This paper proposes a novel deep neural network topology for parametric modeling of microwave components. In the proposed deep neural network, the outputs are S-parameters. The inputs of the proposed model include geometrical variables and the frequency. We divide the hidden layers in the proposed deep neural network topology into two parts. Hidden layers in Part I handle both the geometrical inputs and the frequency inputs while hidden layers in Part II only handle the geometrical inputs. In this way, more training parameters are utilized to specifically learn the relationship between the S-parameters and the geometrical variables, which are more complicated than that between the S-parameters and the frequency. The purpose is to reduce the total number of training parameters in the deep neural network model. New formulations are derived to calculate the derivatives of the error function with respect to training parameters in the deep neural network. Taking advantage of the calculated derivatives, we propose an advanced two-stage training algorithm for the deep neural network. The two-stage training algorithm can determine the number of hidden layers in both parts during the training process and guarantee that the proposed deep neural network model can achieve the required model accuracy. The proposed deep neural network can achieve similar model accuracy with less training parameters compared to the commonly used fully connected neural network. The proposed technique is demonstrated by two microwave parametric modeling examples.

DNN-Based Surrogate Modeling-Based Feasible Performance Reliability Design Methodology for Aircraft Engine

The risks and costs of developing a new aeroengine are fundamentally depending on the performance design final proposal. Thus, this paper presents a novel aeroengine performance design methodology that is committed to managing the effectiveness and economic availability of the design proposal. To reach such a target, the presented methodology formulates the traditional thermal cycle design problem as a reliability-based fuzzy optimization. The performance reliability is predicted by the deep neural network (DNN)-based surrogate models while a hyperparameter tuning technique is proposed to find the optimal DNN topology for better implementing a particular deep learning task. The testing results imply the DNN models with optimized topology possess remarkable function approximation capability in global so that achieves significantly higher prediction accuracy. Moreover, the DNN-based surrogate models only cost nearly 0.003% as much computing time as MC simulation (2.3591 sec vs 64746 sec, for 20 samples). Such kind of remarkably higher computational efficiency facilitates the optimization for reliability-based fitness calculation. The efficiency of the presented methodology can be further verified by abundant feasible cycle proposals. The obtained cycle solutions can achieve expected reliability (>95%) in all reference states without unnecessary performance redundancy. Besides, the diversity of feasible cycle solutions contributes to the selection of best proposal associated with engineering situation. The presented effort is favorable to acquire a more cost-efficient design proposal and enrich thermodynamic system design theory.

An Experiment on the Use of Genetic Algorithms for Topology Selection in Deep Learning

The choice of a good topology for a deep neural network is a complex task, essential for any deep learning project. This task normally demands knowledge from previous experience, as the higher amount of required computational resources makes trial and error approaches prohibitive. Evolutionary computation algorithms have shown success in many domains, by guiding the exploration of complex solution spaces in the direction of the best solutions, with minimal human intervention. In this sense, this work presents the use of genetic algorithms in deep neural networks topology selection. The evaluated algorithms were able to find competitive topologies while spending less computational resources when compared to state-of-the-art methods.

The Art of Getting Deep Neural Networks in Shape

Training a deep neural network (DNN) involves selecting a set of hyperparameters that define the network topology and influence the accuracy of the resulting network. Often, the goal is to maximize prediction accuracy on a given dataset. However, non-functional requirements of the trained network -- such as inference speed, size, and energy consumption -- can be very important as well. In this article, we aim to automate the process of selecting an appropriate DNN topology that fulfills both functional and non-functional requirements of the application. Specifically, we focus on tuning two important hyperparameters, depth and width, which together define the shape of the resulting network and directly affect its accuracy, speed, size, and energy consumption. To reduce the time needed to search the design space, we train a fraction of DNNs and build a model to predict the performances of the remaining ones. We are able to produce tuned ResNets, which are up to 4.22 times faster than original depth-scaled ResNets on a batch of 128 images while matching their accuracy.

Spatio-Spectral Representation Learning for Electroencephalographic Gait-Pattern Classification.

The brain plays a pivotal role in locomotion by coordinating muscles through interconnections that get established by the peripheral nervous system. To date, many attempts have been made to reveal the underlying mechanisms of humans' gait. However, decoding cortical processes associated with different walking conditions using EEG signals for gait-pattern classification is a less-explored research area. In this paper, we design an EEG-based experiment with four walking conditions (i.e., free walking, and exoskeleton-assisted walking at zero, low, and high assistive forces by the use of a unilateral exoskeleton to right lower limb). We proposed spatio-spectral representation learning (SSRL), a deep neural network topology with shared weights to learn the spatial and spectral representations of multi-channel EEG signals during walking. Adoption of weight sharing reduces the number of free parameters, while learning spatial and spectral equivariant features. SSRL outperformed state-of-the-art methods in decoding gait patterns, achieving a classification accuracy of 77.8%. Moreover, the features extracted in the intermediate layer of SSRL were observed to be more discriminative than the hand-crafted features. When analyzing the weights of the proposed model, we found an intriguing spatial distribution that is consistent with the distribution found in well-known motor-activated cortical regions. Our results show that SSRL advances the ability to decode human locomotion and it could have important implications for exoskeleton design, rehabilitation processes, and clinical diagnosis.

ScaleDeep

Deep Neural Networks (DNNs) have demonstrated state-of-the-art performance on a broad range of tasks involving natural language, speech, image, and video processing, and are deployed in many real world applications. However, DNNs impose significant computational challenges owing to the complexity of the networks and the amount of data they process, both of which are projected to grow in the future. To improve the efficiency of DNNs, we propose ScaleDeep, a dense, scalable server architecture, whose processing, memory and interconnect subsystems are specialized to leverage the compute and communication characteristics of DNNs. While several DNN accelerator designs have been proposed in recent years, the key difference is that ScaleDeep primarily targets DNN training, as opposed to only inference or evaluation. The key architectural features from which ScaleDeep derives its efficiency are: (i) heterogeneous processing tiles and chips to match the wide diversity in computational characteristics (FLOPs and Bytes/FLOP ratio) that manifest at different levels of granularity in DNNs, (ii) a memory hierarchy and 3-tiered interconnect topology that is suited to the memory access and communication patterns in DNNs, (iii) a low-overhead synchronization mechanism based on hardware data-flow trackers, and (iv) methods to map DNNs to the proposed architecture that minimize data movement and improve core utilization through nested pipelining. We have developed a compiler to allow any DNN topology to be programmed onto ScaleDeep, and a detailed architectural simulator to estimate performance and energy. The simulator incorporates timing and power models of ScaleDeep's components based on synthesis to Intel's 14nm technology. We evaluate an embodiment of ScaleDeep with 7032 processing tiles that operates at 600 MHz and has a peak performance of 680 TFLOPs (single precision) and 1.35 PFLOPs (half-precision) at 1.4KW. Across 11 state-of-the-art DNNs containing 0.65M-14.9M neurons and 6.8M-145.9M weights, including winners from 5 years of the ImageNet competition, ScaleDeep demonstrates 6x-28x speedup at iso-power over the state-of-the-art performance on GPUs.