Deep Neural Network Structure Research Articles

3D plant root system reconstruction based on fusion of deep structure-from-motion and IMU

Roots play a critical role in the functioning of plants. However, it is still challenging to generate detailed 3D models of thin and complicated plant roots, due to the complexity of the structure and the limited textures. Limited by the difficulty of realization and inaccessibility of labeled data for training, few works have been put in exploring this problem using deep neural networks. To overcome this limitation, this paper presents a structure-from-motion based deep neural network structure for plant root reconstruction in a self-supervised manner, which can be applied by mobile phone platforms. In the training process of deep structure-from-motion, each depth is constrained from the depth map and predicted relative poses from their adjacent frames captured by the mobile phone cameras, and the LSTM-based network after CNN for pose estimation is learnt from the ego-motion constraints by further exploiting the temporal relationship between consecutive frames. IMU unit in the mobile phone is further utilized to improve the pose estimation network by continuously updating the correct scales from the gyroscope and accelerometer moment. Our proposed approach is able to solve the scale ambiguity in recovering the absolute scale of the real plant roots so that the approach can promote the performance of camera pose estimation and scene reconstruction jointly. The experimental results on both real plant root dataset and the rendered synthetic root dataset demonstrate the superior performance of our method compared with the classical and state-of-the-art learning-based structure-from-motion methods.

Flexible human motion transition via hybrid deep neural network and quadruple-like structure learning

Flexible human motion transition via hybrid deep neural network and quadruple-like structure learning

Emotional Deep Learning Programming Controller for Automatic Voltage Control of Power Systems

In recent years, the rapid development of artificial intelligence, especially deep learning technology, makes machine learning have application scenarios in the fields of power system stability analysis, coordination along with scheduling and load forecasting. This paper designs an emotional deep learning programming controller (EDLPC) for automatic voltage control of power systems. The designed EDLPC contains an emotional deep neural network (EDNN) structure and an artificial emotional Q-learning algorithm. Besides, a specially defined proportional-integral-derivative (PID) controller is added to the deep neural networks (DNNs) structure as the actuator of an EDNN to realize the automatic tuning of PID controller parameters. In terms of control, the controller combines the advantages of the EDNN and PID controller, meanwhile adopts a reinforcement learning algorithm to optimize the parameters. From the perspective of reinforcement learning, embedding prior knowledge into the output instructions of EDNN is helpful to weaken the fitting problem in the training process. Compared with the outputs of the DNN and Q-learning algorithm under the two cases, the EDLPC could gain the highest control performance with smaller voltage deviations. The simulation results verify the feasibility and effectiveness of the proposed method for automatic voltage control of power systems.

Genetic algorithm based deep learning neural network structure and hyperparameter optimization

Genetic algorithm based deep learning neural network structure and hyperparameter optimization

Flexible human motion transition via hybrid deep neural network and quadruple-like structure learning

Skeletal motion transition is of crucial importance to the animation creation. In this paper, we propose a hybrid deep learning framework that allows for efficient human motion transition. First, we integrate a convolutional restricted Boltzmann machine with deep belief network to extract the spatio-temporal features of each motion style, featuring on appropriate detection of transition points. Then, a quadruples-like data structure is exploited for motion graph building, motion splitting and indexing. Accordingly, the similar frames fulfilling the transition segments can be efficiently retrieved. Meanwhile, the transition length is reasonably computed according to the average speed of the motion joints. As a result, different kinds of diverse motions can be well transited with satisfactory performance. The experimental results show that the proposed transition approach brings substantial improvements over the state-of-the-art methods.

Dynamic one-shot target detection and classification using a pseudo-Siamese network and its application to Raman spectroscopy.

Target detection and classification by Raman spectroscopy are important techniques for biological and chemical defense in military operations. Conventionally, these techniques preprocess the observed spectra using smoothing or baseline correction and apply detection algorithms like the generalized likelihood ratio test, independent component analysis, nonnegative matrix factorization, etc. These conventional detection algorithms need preprocessing and multiple shots of Raman spectra to get a reasonable accuracy. Recently, techniques based on deep learning are being used for target detection and classification due to its great adaptability and high accuracy over other methods and due to no requirement for preprocessing. Deep learning may give a good performance, but need retraining when untrained class targets are introduced which is time-consuming and bothersome. We devise a novel algorithm using a variant of the pseudo-Siamese network, one of the deep learning algorithms, that does not need retraining to detect and classify untrained class targets. Our algorithm detects and classifies targets with only one shot. In addition, our algorithm does not need preprocessing. We verify our algorithm with Raman spectra measured using a Raman spectrometer.

Prediction of yarn unevenness based on BMNN

With the continuous development of deep learning, due to the complexity of the deep neural network structure and the limitation of training time, some scholars have proposed broad learning, the Broad Learning System (BLS). However, BLS currently only verifies that it has excellent effects on some of the network training data sets, and it does not necessarily have excellent effects on some actual data sets. In response to this, this paper uses the effect of BLS in predicting the unevenness of yarn quality in the yarn data set, and proposes a BLS-based multi-layer neural network (MNN) for the problems, which is called Broad Multilayer Neural Network (BMNN).

U-Net-Based Multispectral Image Generation From an RGB Image

Multispectral images have lower spatial resolution than RGB images. It is difficult to obtain multispectral images with both high spatial resolution and high spectral resolution because of expensive capture setup and sophisticated acquisition processes. In this paper, we propose a deep neural network structure based on U-Net to convert ordinary RGB images into multispectral images with high spectral resolution. Our variant U-Net neural network structure not only preserves detailed features of RGB images, but also promotes the fusion of different feature scales, enhancing the quality of multispectral image generation. Apart from the training stage, our proposed method does not require low-resolution multispectral images, as do some earlier learning-based methods; multispectral images can be obtained using only the corresponding RGB high-resolution images. We also employ the Inception block to achieve richer image features and the feature loss function to optimize the non-local features. Our proposed algorithm achieves state-of-the-art visual effects and quantitative measurements such as RMSE and rRMSE on several different public datasets.

Data-Driven Learning of Nonautonomous Systems

We present a numerical framework for recovering unknown non-autonomous dynamical systems with time-dependent inputs. To circumvent the difficulty presented by the non-autonomous nature of the system, our method transforms the solution state into piecewise integration of the system over a discrete set of time instances. The time-dependent inputs are then locally parameterized by using a proper model, for example, polynomial regression, in the pieces determined by the time instances. This transforms the original system into a piecewise parametric system that is locally time invariant. We then design a deep neural network structure to learn the local models. Once the network model is constructed, it can be iteratively used over time to conduct global system prediction. We provide theoretical analysis of our algorithm and present a number of numerical examples to demonstrate the effectiveness of the method.

Machine Learning-Based Beamforming in K-User MISO Interference Channels

In this paper, we consider a $K$ -user multiple-input and single-output interference channel and propose a machine learning-based beamforming design. To circumvent the difficulties of beamforming design in interference channels, we consider a beamforming scheme that combines two well-known beamforming schemes, which are maximum ratio transmission and zero-forcing, with one of finite combining factors. However, this problem is still NP-hard, which requires brute-force search for optimal beamforming, so we adopt the machine learning to find the optimal beamforming vectors. Our machine learning design exploits a deep neural network structure, whose input nodes take channel vectors with transmit power, while the output nodes return the combining factors for the transmitters’ beamforming. Our numerical results show that our proposed machine learning-based beamforming achieves a sum rate more than 99% of that with the best combining factors numerically found with brute-force search.

Examining the Causal Structures of Deep Neural Networks Using Information Theory.

Deep Neural Networks (DNNs) are often examined at the level of their response to input, such as analyzing the mutual information between nodes and data sets. Yet DNNs can also be examined at the level of causation, exploring “what does what” within the layers of the network itself. Historically, analyzing the causal structure of DNNs has received less attention than understanding their responses to input. Yet definitionally, generalizability must be a function of a DNN’s causal structure as it reflects how the DNN responds to unseen or even not-yet-defined future inputs. Here, we introduce a suite of metrics based on information theory to quantify and track changes in the causal structure of DNNs during training. Specifically, we introduce the effective information (EI) of a feedforward DNN, which is the mutual information between layer input and output following a maximum-entropy perturbation. The EI can be used to assess the degree of causal influence nodes and edges have over their downstream targets in each layer. We show that the EI can be further decomposed in order to examine the sensitivity of a layer (measured by how well edges transmit perturbations) and the degeneracy of a layer (measured by how edge overlap interferes with transmission), along with estimates of the amount of integrated information of a layer. Together, these properties define where each layer lies in the “causal plane”, which can be used to visualize how layer connectivity becomes more sensitive or degenerate over time, and how integration changes during training, revealing how the layer-by-layer causal structure differentiates. These results may help in understanding the generalization capabilities of DNNs and provide foundational tools for making DNNs both more generalizable and more explainable.

A Method to Select Optimal Deep Neural Network Model for Power Amplifiers

The power amplifier (PA) behavior models based on deep neural networks (DNNs) have been widely used. However, it is challenging to balance the network depth, the number of neurons, and the combination of input terms. Accordingly, how to obtain a suitable DNN model becomes a problem. This letter proposes a method to acquire the corresponding optimal DNN structure balancing multiple aspects for a PA by decoupling the relationship of the three factors mentioned above. The algorithm first chooses the optimal input combinations, and then utilizes the inputs to select the DNN structure with excellent capability to fit nonlinear functions and fewer coefficients. Experimental results verify that the DNN model acquired by the algorithm proposed can maintain superior performance with low complexity.

Ocean wave energy forecasting using optimised deep learning neural networks

Ocean renewable energy is a promising inexhaustible source of renewable energy, with an estimated harnessing potential of approximately 337 GW worldwide, which could re-shape the power generation mix. As with other sources of renewables, however, wave energy has an intermittent and irregular nature, which is a major concern for power system stability. Consequently, in order to integrate wave energy into power grids, it must be forecasted. This paper proposes using optimised deep learning neural networks to forecast the wave energy flux, and other wave parameters. In particular, we use moth-flame optimisation as the central decision-making unit to configure the deep neural network structure and the proper input data selection. Besides, the moth-flame optimisation algorithm was modified to improve its search space mechanisms. The forecasting skills are assessed using 13 datasets from locations across the Pacific and Atlantic coasts, and the Gulf of Mexico. The proposed optimised deep neural network performs well at all the sites, especially over short-term horizons, where it outperforms statistical and physics-based approaches.

Deep learning networks reflect cytoarchitectonic features used in brain mapping

The distribution of neurons in the cortex (cytoarchitecture) differs between cortical areas and constitutes the basis for structural maps of the human brain. Deep learning approaches provide a promising alternative to overcome throughput limitations of currently used cytoarchitectonic mapping methods, but typically lack insight as to what extent they follow cytoarchitectonic principles. We therefore investigated in how far the internal structure of deep convolutional neural networks trained for cytoarchitectonic brain mapping reflect traditional cytoarchitectonic features, and compared them to features of the current grey level index (GLI) profile approach. The networks consisted of a 10-block deep convolutional architecture trained to segment the primary and secondary visual cortex. Filter activations of the networks served to analyse resemblances to traditional cytoarchitectonic features and comparisons to the GLI profile approach. Our analysis revealed resemblances to cellular, laminar- as well as cortical area related cytoarchitectonic features. The networks learned filter activations that reflect the distinct cytoarchitecture of the segmented cortical areas with special regard to their laminar organization and compared well to statistical criteria of the GLI profile approach. These results confirm an incorporation of relevant cytoarchitectonic features in the deep convolutional neural networks and mark them as a valid support for high-throughput cytoarchitectonic mapping workflows.

Effective analysis of noise levels due to vehicular traffic in urban area using deep learning with OALO model

Nowadays, there has been an exponential grow in the number of vehicles moving on the roads, causing an unavoidable intensification of levels in the traffic noise. There is no second opinion on the fact the ever-zooming noise levels have adversely affected the health and welfare of a substantial section of society, especially those who are residing in the immediate vicinity of highways and urban roads. In this regard, a novel method intended for the improvement of the vehicular traffic noise prediction techniques namely the Deep Neural Network (DNN) is introduced. For optimizing the weight of DNN structure, we designed a meta-heuristic approach termed as the Oppositional based Antlion Optimization (OALO). Using the data of observed noise levels, traffic volume and average speed of vehicles, the noise parameters such as Equivalent continuous (A-weighted) sound level Leq and Percentile exceeded sound level, L10 are predicted. The predicted noise levels are compared with experimental and other existing prediction models. It is observed that the proposed DNN-OALO approach attains high accuracy and also accomplished a positive correlation between actual and predicted noise levels.

A Clothing Retrieval Model Based on DenseNet and Multi-Similarity Loss

With the rapid development of deep learning in recent years, Deep Neural Network structure has been applied to all aspects of image algorithm, including image classification and recognition, target detection, image retrieval. Image retrieval is suffering from some challenges, such as crease, cover, background. Based on these challenges, using the local information of the image to replace the image is proposed which learns embedding vectors by local information to reduce noise interference. Inspired by the above discussion, a new deep learning model named IAPM (Integrated and Partial Model) is proposed in this paper. According to global information and local information, Joint training is carried out. For feature extraction, DenseNet is used to extract the feature vector. And the idea of transfer learning is applied to transform classification problems to retrieval problems. Meanwhile, the multi-task model is adopted to construct a framework, and Multi-Similarity Loss is used to pull the position of the embedded vector in feature space. Experimental results show its effectiveness.

Compressing Deep Networks by Neuron Agglomerative Clustering.

In recent years, deep learning models have achieved remarkable successes in various applications, such as pattern recognition, computer vision, and signal processing. However, high-performance deep architectures are often accompanied by a large storage space and long computational time, which make it difficult to fully exploit many deep neural networks (DNNs), especially in scenarios in which computing resources are limited. In this paper, to tackle this problem, we introduce a method for compressing the structure and parameters of DNNs based on neuron agglomerative clustering (NAC). Specifically, we utilize the agglomerative clustering algorithm to find similar neurons, while these similar neurons and the connections linked to them are then agglomerated together. Using NAC, the number of parameters and the storage space of DNNs are greatly reduced, without the support of an extra library or hardware. Extensive experiments demonstrate that NAC is very effective for the neuron agglomeration of both the fully connected and convolutional layers, which are common building blocks of DNNs, delivering similar or even higher network accuracy. Specifically, on the benchmark CIFAR-10 and CIFAR-100 datasets, using NAC to compress the parameters of the original VGGNet by 92.96% and 81.10%, respectively, the compact network obtained still outperforms the original networks.

DEEP LEARNING OF PARAMETERIZED EQUATIONS WITH APPLICATIONS TO UNCERTAINTY QUANTIFICATION

We propose a learning algorithm for discovering unknown parameterized dynamical systems by using observational data of the state variables. Our method is built upon and extends the recent work of discovering unknown dynamical systems, in particular those using a deep neural network (DNN). We propose a DNN structure, largely based upon the residual network (ResNet), to not only learn the unknown form of the governing equation but also to take into account the random effect embedded in the system, which is generated by the random parameters. Once the DNN model is successfully constructed, it is able to produce system prediction over a longer term and for arbitrary parameter values. For uncertainty quantification, it allows us to conduct uncertainty analysis by evaluating solution statistics over the parameter space.

The Hardware and Algorithm Co-Design for Energy-Efficient DNN Processor on Edge/Mobile Devices

Deep neural network (DNN) has been widely studied due to its high performance and usability for various applications such as image classification, detection, segmentation, translation, and action recognition. Thanks to the universal applications and high performance of DNN algorithm, DNN is adopted for various AI platforms, including edge/mobile devices as well as cloud servers. However, high-performance DNN requires a large amount of computation and memory access, making it challenging to implement DNN operation on edge/mobile. There have been several ways to solve these problems, including algorithms as well as hardware for DNN. Algorithms that help accelerate DNN in hardware enable much more efficient operation of high-performance AI. This article aims to provide an overview of the recent hardware and algorithm co-design schemes enabling efficient processing of DNNs. Specifically, it will provide algorithm optimization methods for DNN structure, neurons, synapses, and data types. This paper also introduces optimization methods for hardware architectures, PE array, data-path control, and microarchitecture of PE. And we will also show examples of DNN algorithm and hardware co-designed ASICs.

A hybrid deep learning model of process-build interactions in additive manufacturing

Laser powder bed fusion (LPBF) is a technique of additive manufacturing (AM) that is often used to construct a metal object layer-by-layer. The quality of AM builds depends to a great extent on the minimization of different defects such as porosity and cracks that could occur by process deviation during machine operation. Therefore, there is a need to develop new analytical methods and tools to equip the LPBF process with the inspection frameworks that assess the process condition and monitor the porosity defect in real-time. Advanced sensing is recently integrated with the AM machines to cope with process complexity and improve information visibility. This opportunity lays the foundation for online monitoring and assessment of the in-process build layer. This study presents the hybrid deep neural network structure with two types of input data to monitor the process parameters that result in porosity defect in cylinders’ layers. Results demonstrate that statistical features extracted by wavelet transform and texture analysis along with original powder bed images, assist the model in reaching a robust performance. In order to illustrate the fidelity of the proposed model, the capability of the main pipeline is examined and compared with different machine learning models. Eventually, the proposed framework identified the process conditions with an F-score of 97.14%. This salient flaw detection ability is conducive to repair the defect in real-time and assure the quality of the final part before the completion of the process.