Deep Neural Network Structures Research Articles

Leveraging Deep Learning for Practical DoA Estimation: Experiments with Real Data Collected via USRP.

This paper presents an experimental validation of deep learning-based direction-of-arrival (DoA) estimation by using realistic data collected via universal software radio peripheral (USRP). Deep neural network (DNN) and convolutional neural network (CNN) structures are designed to estimate the DoA. Two types of data are used for training networks. One is the data synthesized by the signal model, and the other is the data collected by USRP. Here, the signal model considers both mutual coupling and multipath signals. Experimental results show that the estimation performance is most accurate when training DNN and CNN with the collected data. Furthermore, the estimation tends to be poor in the indoor environment, which suffers from the strong non-line-of-sight (NLoS) signals.

Optimal Model Placement and Online Model Splitting for Device-Edge Co-Inference

Device-edge co-inference opens up new possibilities for resource-constrained wireless devices (WDs) to execute deep neural network (DNN)-based applications with heavy computation workloads. In particular, the WD executes the first few layers of the DNN and sends the intermediate features to the edge server that processes the remaining layers of the DNN. By adapting the model splitting decision, there exists a tradeoff between local computation cost and communication overhead. In practice, the DNN model is re-trained and updated periodically at the edge server. Once the DNN parameters are regenerated, part of the updated model must be placed at the WD to facilitate on-device inference. In this paper, we study the joint optimization of the model placement and online model splitting decisions to minimize the energy-and-time cost of device-edge co-inference in presence of wireless channel fading. The problem is challenging because the model placement and model splitting decisions are strongly coupled, while involving two different time scales. We first tackle online model splitting by formulating an optimal stopping problem, where the finite horizon of the problem is determined by the model placement decision. In addition to deriving the optimal model splitting rule based on backward induction, we further investigate a simple one-stage look-ahead rule, for which we are able to obtain analytical expressions of the model splitting decision. The analysis is useful for us to efficiently optimize the model placement decision in a larger time scale. In particular, we obtain a closed-form model placement solution for the fully-connected multilayer perceptron with equal neurons. Simulation results validate the superior performance of the joint optimal model placement and splitting with various DNN structures.

A GPU-based accelerated ELM and deep-ELM training algorithms for traditional and deep neural networks classifiers

• In this paper, we have proposed novel parallel ELM and D-ELM algorithms on the GPU. The benefits of these parallel algorithms include reducing the training time of the algorithms and reducing the runtime significantly. Also, remarkably predictive 95 accuracy is presented. To the best of our knowledge, this is the first kind of parallelization of the D-ELM in which the high computing power of graphics processing equipped with CUDA is utilized to parallelize. Our main contribution is as follows. • All sub-operations of ELM and D-ELM which need to be parallelized 100 are parallelized on GPU, thus benefiting from the memory hierarchy. In the meantime, several optimizations are adopted to improve the training time and performance of our parallelization. • Maximizing parallel performance: It is very important organizing the algorithm in computational blocks that are able to run independently. 105 Also, the communication between the blocks is minimized to achieve better performance. • Improve the usage of memory hierarchy: It is especially important to execution maximize the number of operations on data stored in shared memory. This strategy is applied to ELM and D-ELM 110 algorithms. • Using of CUDA Optimal Libraries: In some operations of ELM and D-ELM algorithms, these useful tools have been used for improving the performance. • Maximizing the independent and concurrent execution of the times 115 auto-encoders in the D-ELM Algorithm: Given that many computational operations are located in this section, their concurrent and independent execution is extraordinary for improving the performance. • The implementation of the standard of ELM and D-ELM has a weak 120 learning speed by increasing the scale of data. We have optimally implemented serial versions of ELM and D-ELM algorithms as a basis and brought the cost of training time for different strategies using data with different scales. Serial part evaluations show that the cost of training time for ELM and D-ELM algorithms and the size of 125 the dataset have almost linearly related. • Our proposed parallel algorithms have well training time and prediction accuracy and offer high performance for iterative calculations. Since they have high computing power so another variant of machine learning algorithms that have high iterative 130 computations can benefit from our proposed methods. The extreme learning machine (ELM) has been effectively used for training single-layer neural networks. In recent years, great attention has been paid to deep extreme learning machine (D-ELM) structures. Deep neural network structures are trained via the ELM method. Some stacked auto-encoders followed by a simple ELM layer can be usually used for solving classification and regression tasks. Although ELM has been employed for speeding up the training process of the neural network, D-ELM based models suffer from some issues such as the time complexity and running time. In this paper, we explore how the evaluation of ELM and D-ELM can be accelerated. GPUs are used to speed up the training process of ELM and D-ELM models. In the proposed method, three separate phases are considered for the algorithms. In the first phase, loading and pre-processing the data are performed serially in the CPU. In the second and third phases, which respectively are the training and testing phases of the algorithm, all the matrix operations of the algorithms are implemented in parallel mode using the GPU memory hierarchy. Also, having access to highly efficient computational libraries, additional support is provided for GPU-based parallel computing. In the simulation setup, five sets of the database are applied to train the ELM and D-ELM on both CPU and GPU platforms. The results obtained show the proposed approach based on GPUs can remarkable save running time. Although both serial and parallel methods measure approximately the same accuracy, the parallel methods provided for the models reduce the run time significantly.

Progress of the Computer-Generated Holography Based on Deep Learning

With the explosive developments of deep learning, learning–based computer–generated holography (CGH) has become an effective way to achieve real–time and high–quality holographic displays. Plentiful learning–based methods with various deep neural networks (DNNs) have been proposed. In this paper, we focus on the rapid progress of learning–based CGH in recent years. The generation principles and algorithms of CGH are introduced. The DNN structures frequently used in CGH are compared, including U–Net, ResNet, and GAN. We review the developments and discuss the outlook of the learning–based CGH.

Large-scale photovoltaic system in green building: MPPT control based on deep neural network and dynamic time-window

Under the background of smart city, the concepts of “green building” and “net-zero energy building” become more and more popular for reducing the building power consumption. As a result, the technologies related to the design and intelligent control of building integrated green energy system develop rapidly in recent years. In this study, the topological structure of large-scale building integrated photovoltaic (BIPV) system is analyzed, and a novel data-driven maximum power point tracking (MPPT) methodology is developed. To be specific, several characteristic-variables for achieving efficient MPPT of large-scale BIPV system are proposed, and the data-driven MPPT model based on deep neural network (DNN) is developed. Then, the developed characteristic-variables and DNN model are verified by a comprehensive set of numerical experiments. The optimal DNN structure is also verified in detail in this study. In addition, in order to dynamically track the degradation of photovoltaic module and overcome its influence on DNN model, the time-window mechanism of BIPV knowledge-base is introduced, and the optimal length of time-window for different DNN structures is verified by numerical experiments. Experimental results show that the DNN model with developed characteristic-variables and time-window mechanism achieves accurate and robust forecasting performance on the MPPT of large-scale BIPV system.

Compression and Decompression of Internet Learning Images Based on GABTC

Internet Learning Image compression is a useful approach this is inevitable to save any snapshots in compressed sample. The compressed image is reconstructed the usage of image retrieval system for any applications. This method guarantees garage performance in various content management packages. Mainly, Internet Learning resource environment deals with greater area complexity. The e-learning storage area complexity may be reduced with the assist of image compression strategies. There are extraordinary forms of image compression techniques invented for constructing compressed assets. Block Truncation Coding (BTC) is a kind of lossy compression technique for reducing grey scale quantities the usage of blocking and quantizing stages. This is powerful technique for compressing the photos. In the scope, the want for extra active and Generative image compression methods is important to enhance BTC principles. This work proposes notably complex and novel Generative Adversarial BTC (GA-BTC) compression models with more than one variances. GABTC is developed with multi-layered Deep Neural Network (DNN) structures with GA neural models. The integration of each GA fashions and BTC standards improve the best of block constructions and reconstructions notably. The proposed work evaluates the version complexity and efficiency using diverse Internet Learning images with exclusive compression fine measurements.

Prediction of Muon Energy using Deep Neural Network with Multiple Coulomb Scattering Data

This study is based on the determination of muon beam energies using multiple Coulomb scattering data in artificial neural networks. Muon particles were scattered off a 50-layer lead object by using the G4beamline simulation program which is based on Geant4. Before working with deep neural networks, average scattering angle distributions in terms of the number of crossed layers were analyzed with the fitting method using the well-known formula for multiple Coulomb scattering to estimate muon beam energies. Subsequently, average scattering angles over the number of crossed layers from 1 to 10 were used in deep neural network structures to estimate the muon beam energy. It has been observed that deep neural networks significantly improve the resolutions compared to the ones obtained with the fitting method.

Brain tumor classification by using a novel convolutional neural network structure

AbstractBrain tumors located in the skull are among the health problems that cause serious consequences. Rapid and accurate detection of brain tumor types will ensure that the patient receives appropriate treatment in the early period, thus increasing the patient's chance of recovery and survival. In the literature, classification accuracies over 98% have been acquired automatically by using deep neural networks (DNN) for the brain tumor images such as glioma, meningioma, and pituitary. It is observed that researchers generally focused on achieving higher classification accuracy and therefore, they have used pre‐processing stages, augmentation processes, huge or hybrid DNN structures. These approaches have brought some disadvantages in terms of practical use of the developed methods: (i)The parameters of the pre‐processes should be carefully determined, otherwise the classification accuracy will decrease. (ii) In order to increase the classification performance, it is important to determine the coarse structure of the DNN correctly. If the DNN has many hyper‐parameters, the coarse structure will be determined in a long time. (iii) It is difficult to implement complex DNN structures or training algorithms in terms of practical use, because these methods need huge memory and high CPU computation. In this study, we have proposed a novel DNN model to increase the classification accuracy, and to decrease the number of weights in the structure, and to use less number of hyper‐parameters. We named this model, which uses a divergence‐based feature extractor, as DivFE‐v1 for short. 99.18% classification accuracy for the Figshare dataset is obtained by using the small‐sized DNN structure without any pre‐processing stage or augmentation process.

Machine Learning-Based Vision-Aided Beam Selection for mmWave Multiuser MISO System

In this letter, we propose a machine learning-based vision-aided beam selection (ML-VBS) for millimeter-wave indoor multi-user communications. The proposed scheme is aimed at addressing the beam selection overhead with narrow beams in a multi-user scenario. The proposed scheme relies on a base station (BS) equipped with a single camera to observe the scene and estimates the angles to the multiple users. Given the estimated angle information and the limited number of radio frequency chains at the BS, two serial deep neural network structures are employed for joint user and beam selection subject to a minimum rate constraint. The numerical evaluation shows that the proposed ML-VBS scheme achieves a good performance in terms of the multi-user angle estimation, achievable sum rate and low computational complexity compared to conventional beam selection techniques.

Learning deep neural networks' architectures using differential evolution. Case study: Medical imaging processing

The COVID-19 pandemic has changed the way we practice medicine. Cancer patient and obstetric care landscapes have been distorted. Delaying cancer diagnosis or maternal-fetal monitoring increased the number of preventable deaths or pregnancy complications. One solution is using Artificial Intelligence to help the medical personnel establish the diagnosis in a faster and more accurate manner. Deep learning is the state-of-the-art solution for image classification. Researchers manually design the structure of fix deep learning neural networks structures and afterwards verify their performance. The goal of this paper is to propose a potential method for learning deep network architectures automatically. As the number of networks architectures increases exponentially with the number of convolutional layers in the network, we propose a differential evolution algorithm to traverse the search space. At first, we propose a way to encode the network structure as a candidate solution of fixed-length integer array, followed by the initialization of differential evolution method. A set of random individuals is generated, followed by mutation, recombination, and selection. At each generation the individuals with the poorest loss values are eliminated and replaced with more competitive individuals. The model has been tested on three cancer datasets containing MRI scans and histopathological images and two maternal-fetal screening ultrasound images. The novel proposed method has been compared and statistically benchmarked to four state-of-the-art deep learning networks: VGG16, ResNet50, Inception V3, and DenseNet169. The experimental results showed that the model is competitive to other state-of-the-art models, obtaining accuracies between 78.73% and 99.50% depending on the dataset it had been applied on.

Land use classification of high resolution remote sensing images using an encoder based modified GAN architecture

The development of new deep learning algorithms has brought in a significant change in land use classification. Earlier models in remote sensing image classification were mainly based on machine learning architectures. Many recent models combined deep neural network structures for feature extraction and machine learning algorithms for classification. There are also some end to end deep learning architectures used for feature extraction and classification in the area of remote sensing image classification. Anyway such models are known to take more training data to yield good classification performance and they also necessitate more extensive technical support. The proposed model uses an encoder based modified Generative Adversarial Network (GAN) architecture for land use classification which can yield better results with less amount of training samples. GAN is shown to be a superior synthesiser and classifier than other deep learning models because it can work with a limited number of dataset samples. The validation of the proposed model is carried out on UC Merced and AID land use datasets. The classification results support the hypothesis that the proposed model can outperform conventional land use classification models.

RT-RCG: Neural Network and Accelerator Search Towards Effective and Real-time ECG Reconstruction from Intracardiac Electrograms.

There exists a gap in terms of the signals provided by pacemakers (i.e., intracardiac electrogram (EGM)) and the signals doctors use (i.e., 12-lead electrocardiogram (ECG)) to diagnose abnormal rhythms. Therefore, the former, even if remotely transmitted, are not sufficient for doctors to provide a precise diagnosis, let alone make a timely intervention. To close this gap and make a heuristic step towards real-time critical intervention in instant response to irregular and infrequent ventricular rhythms, we propose a new framework dubbed RT-RCG to automatically search for (1) efficient Deep Neural Network (DNN) structures and then (2) corresponding accelerators, to enable Real-Time and high-quality Reconstruction of ECG signals from EGM signals. Specifically, RT-RCG proposes a new DNN search space tailored for ECG reconstruction from EGM signals, and incorporates a differentiable acceleration search (DAS) engine to efficiently navigate over the large and discrete accelerator design space to generate optimized accelerators. Extensive experiments and ablation studies under various settings consistently validate the effectiveness of our RT-RCG. To the best of our knowledge, RT-RCG is the first to leverage neural architecture search (NAS) to simultaneously tackle both reconstruction efficacy and efficiency.

Algorithm Unrolling for Massive Access via Deep Neural Networks With Theoretical Guarantee

Massive access is a critical design challenge of Internet of Things (IoT) networks. In this paper, we consider the grant-free uplink transmission of an IoT network with a multiple-antenna base station (BS) and a large number of single-antenna IoT devices. Taking into account the sporadic nature of IoT devices, we formulate the joint activity detection and channel estimation (JADCE) problem as a group-sparse matrix estimation problem. This problem can be solved by applying the existing compressed sensing techniques, which however either suffer from high computational complexities or lack of algorithm robustness. To this end, we propose a novel algorithm unrolling framework based on the deep neural network to simultaneously achieve low computational complexity and high robustness for solving the JADCE problem. Specifically, we map the original iterative shrinkage thresholding algorithm (ISTA) into an unrolled recurrent neural network (RNN), thereby improving the convergence rate and computational efficiency through end-to-end training. Moreover, the proposed algorithm unrolling approach inherits the structure and domain knowledge of the ISTA, thereby maintaining the algorithm robustness, which can handle non-Gaussian preamble sequence matrix in massive access. With rigorous theoretical analysis, we further simplify the unrolled network structure by reducing the redundant training parameters. Furthermore, we prove that the simplified unrolled deep neural network structures enjoy a linear convergence rate. Extensive simulations based on various preamble signatures show that the proposed unrolled networks outperform the existing methods in terms of the convergence rate, robustness and estimation accuracy.

Deep encoder–decoder-NN: A deep learning-based autonomous vehicle trajectory prediction and correction model

An accurate vehicle trajectory prediction promotes understanding of the traffic environment and enables task criticality assessment in advanced driver assistance systems (ADASs) in autonomous vehicles and intelligent connected vehicles. Nevertheless, conventional prediction models are characterized by low prediction accuracy, the inability of long-term prediction, and a single-road section adaptation. To tackle these limitations, this study proposes a trajectory prediction model based on a deep encoder–decoder and a deep neural network (DNN). One modification included introducing an attention mechanism into the traditional encoder–decoder framework. Overall, 1794,1400,2100 trajectory samples from highways, intersections, and roundabouts are used to train the proposed framework and obtain optimal deep encoder–decoder architectures for different road section types. Since the experiments revealed no significant advantages of using the attention mechanism in deep encoder–decoder, the mechanism is not included in the optimal architecture. Next, to achieve higher prediction accuracy and better long-term prediction capability, different DNN structures are tested as trajectory correction networks, and the optimal DNN structure is selected. Finally, the experiments are conducted using the proposed deep encoder–decoder framework and the optimal DNN. The results show that the proposed model reaches 92.87%, 86.65%, and 89.15% average trajectory fit ratio (TFR) on a highway, intersection, and a roundabout, respectively. Therefore, the model enables accurate long-term predictions of vehicle trajectories in these road segments. The proposed model and presented results provide a basis for ADASs’ trajectory prediction algorithms.

A Comparative Analysis of Generative Neural Attention-based Service Chatbot

Companies constantly rely on customer support to deliver pre-and post-sale services to their clients through websites, mobile devices or social media platforms such as Twitter. In assisting customers, companies employ virtual service agents (chatbots) to provide support via communication devices. The primary focus is to automate the generation of conversational chat between a computer and a human by constructing virtual service agents that can predict appropriate and automatic responses to customers’ queries. This paper aims to present and implement a seq2seq-based learning task model based on encoder-decoder architectural solutions by training generative chatbots on customer support Twitter datasets. The model is based on deep Recurrent Neural Networks (RNNs) structures which are uni-directional and bi-directional encoder types of Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRU). The RNNs are augmented with an attention layer to focus on important information between input and output sequences. Word level embedding such as Word2Vec, GloVe, and FastText are employed as input to the model. Incorporating the base architecture, a comparative analysis is applied where baseline models are compared with and without the use of attention as well as different types of input embedding for each experiment. Bilingual Evaluation Understudy (BLEU) was employed to evaluate the model’s performance. Results revealed that while biLSTM performs better with Glove, biGRU operates better with FastText. Thus, the finding significantly indicated that the attention-based, bi-directional RNNs (LSTM or GRU) model significantly outperformed baseline approaches in their BLEU score as a promising use in future works.

NowCasting-Nets: Representation Learning to Mitigate Latency Gap of Satellite Precipitation Products Using Convolutional and Recurrent Neural Networks

Accurate and timely estimation of precipitation is critical for issuing hazard warnings (e.g., for flash floods or landslides). Current remotely sensed precipitation products have a few hours of latency, associated with the acquisition and processing of satellite data. By applying a robust nowcasting system to these products, it is (in principle) possible to mitigate this latency and improve their applicability, value, and impact. However, the development of such a system is complicated by the chaotic nature of the atmosphere, lack of sufficient knowledge about the evolution of precipitation systems based on previous observations, and the consequent rapid changes that can occur in the structures of precipitation systems. In this work, we develop two approaches (hereafter referred to as <i>NowCasting-nets</i>) that use recurrent and convolutional deep neural network (DNN) structures to address the challenge of precipitation nowcasting. A total of five models are trained using global precipitation measurement (GPM) Integrated MultisatellitE Retrievals for GPM (IMERG) precipitation data over the Eastern contiguous United States (CONUS) and then tested against independent data for the Eastern and Western CONUS. The models were designed to provide forecasts with a lead time of up to 1.5 h, and by using a feedback loop approach, the ability of the models to extend the forecast time to 4.5 h was also investigated. The performance of the models was compared against the random forest (RF) and linear regression (LR) machine learning (ML) methods, a persistence benchmark (BM) that uses the most recent observation as the forecast, and optical flow (OF). Independent IMERG observations were used as a reference, and experiments were conducted to examine both overall statistics and case studies involving specific precipitation events. Overall, the forecasts provided by the NowCasting-net models are superior, with the convolutional NowCasting-net (CNC) achieving 42&#x0025;, 24&#x0025;, 18&#x0025;, and 16&#x0025; improvement on the test set mean squared error (MSE) over the BM, LR, RF, and OF models, respectively, for the Eastern CONUS. Results of further testing over the Western CONUS (which was not part of the training data) are encouraging and indicate the ability of the proposed models to learn the dynamics of precipitation systems without having explicit access to motion vectors and other auxiliary features and then to generalize to different hydro-geo-climatic conditions.

Robust Perturbation for Visual Explanation: Cross-Checking Mask Optimization to Avoid Class Distortion.

Along with the outstanding performance of the deep neural networks (DNNs), considerable research efforts have been devoted to finding ways to understand the decision of DNNs structures. In the computer vision domain, visualizing the attribution map is one of the most intuitive and understandable ways to achieve human-level interpretation. Among them, perturbation-based visualization can explain the "black box" property of the given network by optimizing perturbation masks that alter the network prediction of the target class the most. However, existing perturbation methods could make unexpected changes to network predictions after applying a perturbation mask to the input image, resulting in a loss of robustness and fidelity of the perturbation mechanisms. In this paper, we define class distortion as the unexpected changes of the network prediction during the perturbation process. To handle that, we propose a novel visual interpretation framework, Robust Perturbation, which shows robustness against the unexpected class distortion during the mask optimization. With a new cross-checking mask optimization strategy, our proposed framework perturbs the target prediction of the network while upholding the non-target predictions, providing more reliable and accurate visual explanations. We evaluate our framework on three different public datasets through extensive experiments. Furthermore, we propose a new metric for class distortion evaluation. In both quantitative and qualitative experiments, tackling the class distortion problem turns out to enhance the quality and fidelity of the visual explanation in comparison with the existing perturbation-based methods.

Physics-based learning with channel attention for Fourier ptychographic microscopy.

Fourier ptychographic microscopy (FPM) is a computational imaging technology for large field-of-view, high resolution and quantitative phase imaging. In FPM, low-resolution intensity images captured with angle-varying illumination are synthesized in Fourier space with phase retrieval approaches. However, system errors such as pupil aberration and light-emitting diode (LED) intensity error seriously affect the reconstruction performance. In this article, we propose a physics-based neural network with channel attention for FPM reconstruction. With the channel attention module, which is introduced into physics-based neural networks for the first time, the spatial distribution of LED intensity can be adaptively corrected. Besides, the channel attention module is used to synthesize different Zernike modes and recover the pupil function. Detailed simulations and experiments are carried out to validate the effectiveness and robustness of the proposed method. Theresults demonstrate that our method achieves better performance in high-resolution complex field reconstruction, LED intensity correction and pupil function recovery compared with the state-of-art methods. The combination with deep neural network structures like channel attention modules significantly enhance the performance of physics-based neural networks and will promote the application of FPM in practical use.

Automated object detection on aerial images for limited capacity embedded device using a lightweight CNN model

With the growing demand for geospatial data, challenging aerial images with high spatial, spectral, and temporal resolution achieve excellent development. Currently, deep Convolutional Neural Network (CNN) structures are applied widely for object detection. Nevertheless, existing deep CNN-based models consist of complex network structures and require immense amounts of graphics processing unit (GPU) computation power with high energy consumption. Thus, achieving efficient real-time object detection for limited memory and processing capacity embedded device is a major challenge. This paper proposes a feasible and lightweight object detection model based on deep CNN where a mobile inverted bottleneck module is adopted in the backbone structure. Moreover, an enhanced spatial pyramid pooling is adopted to increase the receptive field in the network by concatenating the multi-scale local region features. The experimental results demonstrated that the proposed model achieved higher average precision and required the smallest memory storage compared to previous works. Moreover, the proposed model offers the best trade-offs in terms of detection accuracy, model size, and detection time which has excellent potential to be deployed on limited capacity embedded device.

Design of Fault Diagnosis Algorithm for Electromechanical System Based on Artificial Intelligence

A relatively perfect system for the fault diagnosis of mechanical and electrical products has been formed through decades of development. Nevertheless, the traditional fault diagnosis methods fail to cope with the gradual huge mechanical and electrical system. As a result, the advantages of fault diagnosis mode driven by data are increasingly prominent. Meanwhile, the effect of fault diagnosis has exceeded the traditional fault diagnosis methods in many fields. Through the use of the deep learning technology based on artificial intelligence, it carries out mapping and fitting. By fully taking advantages of neural network, it can effectively obtain the accurate classification of fault data. A fault diagnosis method based on the fault data of mechanical and electrical system is designed in this thesis. When it comes to the basic process, it is to take data sets for different mechanical and electrical products. Through the use of feature engineering method, it extracts the fault features of data. Through the use of deep learning technology, it carries out the intelligent diagnosis. According to the experimental results, it indicates that the fault diagnosis method based on deep learning technology can distinguish a variety of fault modes in mechanical and electrical system in an effective way. What’s more, good classification results in fault recognition have been achieved by a variety of deep convolutional neural network structures, so the feasibility of the method is further verified.