Deep Neural Network Design Research Articles

Material optimization of tri-directional functionally graded plates by using deep neural network and isogeometric multimesh design approach

The paper is aimed at enhancing computational performance for optimizing the material distribution of tri-directional functionally graded (FG) plates. We exploit advantages of using a non-uniform rational B-spline (NURBS) basis function for describing material distribution varying through all three directions of functionally graded (FG) plates. Two-dimensional free vibration and buckling behaviors of multi-directional (1D, 2D and 3D) FG plates analyzed by using a combination of generalized shear deformation theory (GSDT) and isogeometric analysis (IGA) is first proposed. This approach can help to save a significant amount of computational cost while still ensure the accuracy of the solutions. The effectiveness and reliability of the present method are demonstrated by comparing it to other methods in the literature. The obtained results are in excellent agreement with the reference ones. More importantly, data sets consisting of input-output pairs are randomly generated from the analysis process through iterations for the training process in deep neural networks (DNN). DNN is utilized as an analysis tool to supplant finite element analysis to reduce computational cost. By using DNN, behaviors of the multi-directional FG plates are directly predicted from those material distributions. Optimal material distributions of tri-directional FG plates under free vibration or compression in various volume fraction constraints are found by using modified symbiotic organisms search (mSOS) algorithm for the first time. Moreover, an isogeometric multimesh design technique is also used to diminish a large number of design variables in optimization. Optimal results obtained by DNN are compared with those of IGA to verify the effectiveness of the proposed method.

A NoC-based simulator for design and evaluation of deep neural networks

The astonishing development in the field of artificial neural networks (ANN) has brought significant advancement in many application domains, such as pattern recognition, image classification, and computer vision. ANN imitates neuron behaviors and makes a decision or prediction by learning patterns and features from the given data set. To reach higher accuracies, neural networks are getting deeper, and consequently, the computation and storage demands on hardware platforms are steadily increasing. In addition, the massive data communication among neurons makes the interconnection more complex and challenging. To overcome these challenges, ASIC-based DNN accelerators are being designed which usually incorporate customized processing elements, fixed interconnection, and large off-chip memory storage. As a result, DNN computation involves large memory accesses due to frequent load/off-loading data, which significantly increases the energy consumption and latency. Also, the rigid architecture and interconnection among processing elements limit the efficiency of the platform to specific applications. In recent years, Network-on-Chip-based (NoC-based) DNN becomes an emerging design paradigm because the NoC interconnection can help to reduce the off-chip memory accesses while offers better scalability and flexibility. To evaluate the NoC-based DNN in the early design stage, we introduce a cycle-accurate NoC-based DNN simulator, called DNNoC-sim. To support various operations such as convolution and pooling in the modern DNN models, we first propose a DNN flattening technique to convert diverse DNN operation into MAC-like operations. In addition, we propose a DNN slicing method to evaluate the large-scale DNN models on a resource-constraint NoC platform. The evaluation results show a significant reduction in the off-chip memory accesses compared to the state-of-the-art DNN model. We also analyze the performance and discuss the trade-off between different design parameters.

Deep Neural Network Architecture: Application for Facial Expression Recognition

There are great challenges to build a model or architecture in Deep Learning and integrate it into a real-time application. One of these challenges is the construction or acquisition of large quality datasets (thousands or millions of objects). Another one is to have great computing potential for the architecture learning process. Finally, efficient architectures are needed for the design of deep neural networks, which requires expertise, human experience and practical work. This work presents a deep neural network architecture to classify two feelings of facial expression (happy and sad). A set of data is also created that present great changes in: image environments, facial expression, pose, age, ethnicity and others. The evidence presented shows a competitive architecture and indicates an accuracy greater than 90% with noisy data. Finally, the implementation of a real-time application for facial expression recognition is shown.

SAC-Net: Spatial Attenuation Context for Salient Object Detection

This paper presents a new deep neural network design for salient object detection by maximizing the integration of local and global image context within, around, and beyond the salient objects. Our key idea is to adaptively propagate and aggregate the image context features with variable attenuation over the entire feature maps. To achieve this, we design the spatial attenuation context (SAC) module to recurrently translate and aggregate the context features independently with different attenuation factors and then to attentively learn the weights to adaptively integrate the aggregated context features. By further embedding the module to process individual layers in a deep network, namely SAC-Net, we can train the network end-to-end and optimize the context features for detecting salient objects. Compared with 29 state-of-the-art methods, experimental results show that our method performs favorably over all the others on six common benchmark data, both quantitatively and visually.

AdaNS: Adaptive Non-Uniform Sampling for Automated Design of Compact DNNs

This paper introduces an adaptive sampling methodology for automated compression of Deep Neural Networks (DNNs) for accelerated inference on resource-constrained platforms. Modern DNN compression techniques comprise various hyperparameters that require per-layer customization. Our objective is to locate an optimal hyperparameter configuration that leads to lowest model complexity while adhering to a desired inference accuracy. We design a score function that evaluates the aforementioned optimality. The optimization problem is then formulated as searching for the maximizers of this score function. To this end, we devise a non-uniform adaptive sampler that aims at reconstructing the band-limited score function. We reduce the total number of required objective function evaluations by realizing a targeted sampler. We propose three adaptive sampling methodologies, i.e., AdaNS-Zoom, AdaNS-Genetic, and AdaNS-Gaussian, where new batches of samples are chosen based on the history of previous evaluations. Our algorithms start sampling from a uniform distribution over the entire search-space and iteratively adapt the sampling distribution to achieve highest density around the function maxima. This, in turn, allows for a low-error reconstruction of the objective function around its maximizers. Our extensive evaluations corroborate AdaNS effectiveness by outperforming existing rule-based and Reinforcement Learning methods in terms of DNN compression rate and/or inference accuracy.

Application and Need-Based Architecture Design of Deep Neural Networks

This paper applies a hybrid evolutionary approach to a convolutional neural network (CNN) and determines the number of layers and filters based on the application and user need. It integrates compact genetic algorithm with stochastic gradient descent (SGD) for simultaneously evolving structure and parameters of the CNN. It defines an effectual string representation for combining structure and parameters of the CNN. The compact genetic algorithm helps in the evolution of network structure by optimizing the number of convolutional layers and number of filters in each convolutional layer. At the same time, an optimal set of weight parameters of the network is obtained using the SGD law. This approach amalgamates exploration in network space by compact genetic algorithm and exploitation in weight space with SGD in an effective manner. The proposed approach also incorporates user-defined parameters in the cost function in an elegant manner which controls the network structure and hence the performance of the network based on the users need. The effectiveness of the proposed approach has been demonstrated on four benchmark datasets, namely MNIST, COIL-100, CIFAR-10 and CIFAR-100. The obtained results clearly demonstrate the potential of the proposed approach by evolving architectures based on the nature of the application and the need of the user.

Research on Classification and Recognition of Object Image Based on Convolutional Neural Network

Image recognition and classification using a convolution neural network is an important application of image processing. How too reasonably to design the convolutional layer of the convolutional neural network, the number of hidden layers and optimize the parameters of the convolutional neural network to ensure high accuracy and efficiency in image recognition and classification are an extremely important part. The core of this thesis is the model design and parameter optimization of deep convolutional neural networks. This paper mainly designs, implements, optimizes and adjusts the model structure of the convolutional network of Tensor Flow framework platform. We redesigned the convolutional neural network model to a depth of 19 layers and used two data sets for training, testing and parameter optimization. Experimental results show that the convolution neural network models presented in this paper is superior to other neural network models in accuracy and efficiency of image recognition and classification, and has a good guiding role in solving practical engineering problems.

Deep neural networks for choice analysis: Architecture design with alternative-specific utility functions

Whereas deep neural network (DNN) is increasingly applied to choice analysis, it is challenging to reconcile domain-specific behavioral knowledge with generic-purpose DNN, to improve DNN’s interpretability and predictive power, and to identify effective regularization methods for specific tasks. To address these challenges, this study demonstrates the use of behavioral knowledge for designing a particular DNN architecture with alternative-specific utility functions (ASU-DNN) and thereby improving both the predictive power and interpretability. Unlike a fully connected DNN (F-DNN), which computes the utility value of an alternative k by using the attributes of all the alternatives, ASU-DNN computes it by using only k’s own attributes. Theoretically, ASU-DNN can substantially reduce the estimation error of F-DNN because of its lighter architecture and sparser connectivity, although the constraint of alternative-specific utility can cause ASU-DNN to exhibit a larger approximation error. Empirically, ASU-DNN has 2–3% higher prediction accuracy than F-DNN over the whole hyperparameter space in a private dataset collected in Singapore and a public dataset available in the R mlogit package. The alternative-specific connectivity is associated with the independence of irrelevant alternative (IIA) constraint, which as a domain-knowledge-based regularization method is more effective than the most popular generic-purpose explicit and implicit regularization methods and architectural hyperparameters. ASU-DNN provides a more regular substitution pattern of travel mode choices than F-DNN does, rendering ASU-DNN more interpretable. The comparison between ASU-DNN and F-DNN also aids in testing behavioral knowledge. Our results reveal that individuals are more likely to compute utility by using an alternative’s own attributes, supporting the long-standing practice in choice modeling. Overall, this study demonstrates that behavioral knowledge can guide the architecture design of DNN, function as an effective domain-knowledge-based regularization method, and improve both the interpretability and predictive power of DNN in choice analysis. Future studies can explore the generalizability of ASU-DNN and other possibilities of using utility theory to design DNN architectures.

A Novel Action Recognition Framework Based on Deep-Learning and Genetic Algorithms

Recognition of human actions in partially cluttered environments is an important research field of computer vision and human-computer interaction. This field has recently garnered attention from a large number of academic researchers in various fields of application. This study proposes a novel deep-learning-based architecture for the recognition and prediction of human actions based on a hybrid model. The main contribution of this study is to propose a new hybrid architecture, integrating four wide-ranging pre-trained network models in an optimized manner, using a metaheuristic algorithm. This architecture consists of four main stages: namely, the creation of the data set, the design of deep neural network (DNN) architecture, training and optimization of the proposed DNN architecture, and evaluation of the trained DNN. By adapting the aforementioned architecture, reliable features are obtained for the training procedure. In order to validate the superiority of the proposed architecture over other state-of-the-art studies, a performance evaluation between these architectures is presented using benchmark datasets. The results reveal that the proposed architecture outperforms previously developed architectures in terms of predicting human actions.

APNAS: Accuracy-and-Performance-Aware Neural Architecture Search for Neural Hardware Accelerators

Designing resource-efficient deep neural networks (DNNs) is a challenging task due to the enormous diversity of applications as well as their time-consuming design, training, optimization, and evaluation cycles, especially the resource-constrained embedded systems. To address these challenges, we propose a novel DNN design framework called accuracy-and-performance-aware neural architecture search (APNAS), which can generate DNNs efficiently, as it does not require hardware devices or simulators while searching for optimized DNN model configurations that offer both inference accuracy and high execution performance. In addition, to accelerate the process of DNN generation, APNAS is built on a weight sharing and reinforcement learning-based exploration methodology, which is composed of a recurrent neural network controller as its core to generate sample DNN configurations. The reward in reinforcement learning is formulated as a configurable function to consider the sample DNNs' accuracy and cycle count required to run on a target hardware architecture. To further expedite the DNN generation process, we devise analytical models for cycle count estimation instead of running millions of DNN configurations on real hardware. We demonstrate that these analytical models are highly accurate and provide cycle count estimates identical to those of a cycle-accurate hardware simulator. Experiments that involve quantitatively varying hardware constraints demonstrate that APNAS requires only 0.55 graphics processing unit (GPU) days on a single Nvidia GTX 1080Ti GPU to generate DNNs that offer an average of 53% fewer cycles with negligible accuracy degradation (on average 3%) for image classification compared to state-of-the-art techniques.

Pyramid Spatial Context Features for Salient Object Detection

Salient object detection aims to obtain the most attractive objects from the input images, which severs as a pre-processing step for many image processing tasks. This paper presents a novel deep neural network design for salient object detection by formulating a pyramid spatial context module, PSC module for short, to capture the spatial context information at multiple scales. To achieve this, we first adopt convolutional operations with different dilated rates to generate the feature maps with different respective fields, and then use the two-round recurrent translations to explore multiple types of spatial context features on these feature maps. By further inserting this module in a deep network, namely PSCNet, we are able to optimize the network in an end-to-end manner for salient object detection. We evaluate the proposed method on six public benchmark datasets by comparing it with 25 salient object detection methods. The experimental results demonstrate that our PSCNet performs favorably against all the other methods.

Deep Neural Network Design and Implementation for Predicting Malignancy of Cancer Tumors

Breast cancer is the most frequent cancer among women after lung cancer. It is second popular cause of death in the world. Breast cancer cells usually form a tumor (abnormal tissue).There are two types of breast cancer tumors: those that are non-cancerous, or ‘benign’, and those that are cancerous, which are ‘malignant’. Early detection of cancer tumors can prevent its mortality and can take respective precautions according to the type of cancer tumor (either it is benign or malignant). The main objective of this paper is to early diagnosis of the malignancy of cancer tumors which helps to decrease the death rate and helps to give more lives. The selection of suitable deep learning technique is a challenge for the diagnosis of breast cancer. So we created model for prediction of malignancy of cancer tumors using neural network to analyze risk levels that helps in prognosis. It is useful for a doctor to predict the stage of cancer and take respective precautions

Deep neural networks design and analysis for automatic phase pickers from three-component microseismic recordings

SUMMARYIt is essential to pick P-wave and S-wave arrival times rapidly and accurately for the microseismic monitoring systems. Meanwhile, it is not easy to identify the arrivals at a true phase automatically using traditional picking method. This is one of the reasons that many researchers are trying to introduce deep neural networks to solve these problems. Convolutional neural networks (CNNs) are very attractive for designing automatic phase pickers especially after introducing the fundamental network structure from semantic segmentation field, which can give the probability outputs for every labelled phase at every sample in the recordings. The typical segmentation architecture consists of two main parts: (1) an encoder part trained to extracting coarse semantic features; (2) a decoder part responsible not only for recovering the input resolution at the output but also for obtaining sparse representation of the objects. The fundamental segmentation structure performs well; however, the influence of the parameters in the structure on the pickers has not been investigated. It means that the structure design just depends on experience and tests. In this paper, we solve two main questions to give some guidance on network design. First, we show what sparse features will learn from the three-component microseismic recordings using CNNs. Second, the influence of two key parameters in the network on pickers, namely, the depth of decoder and activation functions, is analysed. Increasing the number of levels for a certain layer in the decoder will increase the burden of demand on trainable parameters, but it is beneficial to the accuracy of the model. Reasonable depth of the decoder can balance prediction accuracy and the demand of labelled data, which is important for microseismic systems because manual labelling process will decrease the real-time performance in monitoring tasks. Standard rectified linear unit (ReLU) and leaky rectified linear unit (Leaky ReLU) with different negative slopes are compared for the analysis. Leaky ReLU with a small negative slope can improve the performance of a given model than ReLU activation function by keeping some information about the negative parts.

GenSyth: a new way to understand deep learning

GenSyth: a new way to understand deep learning

Automatic Design of Deep Networks with Neural Blocks

In recent years, deep neural networks (DNNs) have achieved great successes in many areas, such as cognitive computation, pattern recognition, and computer vision. Although many hand-crafted deep networks have been proposed in the literature, designing a well-behaved neural network for a specific application requires high-level expertise yet. Hence, the automatic architecture design of DNNs has become a challenging and important problem. In this paper, we propose a new reinforcement learning method, whose action policy is to select neural blocks and construct deep networks. We define the action search space with three types of neural blocks, i.e., dense block, residual block, and inception-like block. Additionally, we have also designed several variants for the residual and inception-like blocks. The optimal network is automatically learned by a Q-learning agent, which is iteratively trained to generate well-performed deep networks. To evaluate the proposed method, we have conducted experiments on three datasets, MNIST, SVHN, and CIFAR-10, for image classification applications. Compared with existing hand-crafted and auto-generated neural networks, our auto-designed neural network delivers promising results. Moreover, the proposed reinforcement learning algorithm for deep networks design only runs on one GPU, demonstrating much higher efficiency than most of the previous deep network search approaches.

A deep dense inception network for protein beta-turn prediction.

Beta-turn prediction is useful in protein function studies and experimental design. Although recent approaches using machine-learning techniques such as support vector machine (SVM), neural networks, and K nearest neighbor have achieved good results for beta-turn prediction, there is still significant room for improvement. As previous predictors utilized features in a sliding window of 4-20 residues to capture interactions among sequentially neighboring residues, such feature engineering may result in incomplete or biased features and neglect interactions among long-range residues. Deep neural networks provide a new opportunity to address these issues. Here, we proposed a deep dense inception network (DeepDIN) for beta-turn prediction, which takes advantage of the state-of-the-art deep neural network design of dense networks and inception networks. A test on a recent BT6376 benchmark data set shows that DeepDIN outperformed the previous best tool BetaTPred3 significantly in both the overall prediction accuracy and the nine-type beta-turn classification accuracy. A tool, called MUFold-BetaTurn, was developed, which is the first beta-turn prediction tool utilizing deep neural networks. The tool can be downloaded at http://dslsrv8.cs.missouri.edu/~cf797/MUFoldBetaTurn/download.html.

Using the structure of genome data in the design of deep neural networks for predicting amyotrophic lateral sclerosis from genotype.

MotivationAmyotrophic lateral sclerosis (ALS) is a neurodegenerative disease caused by aberrations in the genome. While several disease-causing variants have been identified, a major part of heritability remains unexplained. ALS is believed to have a complex genetic basis where non-additive combinations of variants constitute disease, which cannot be picked up using the linear models employed in classical genotype–phenotype association studies. Deep learning on the other hand is highly promising for identifying such complex relations. We therefore developed a deep-learning based approach for the classification of ALS patients versus healthy individuals from the Dutch cohort of the Project MinE dataset. Based on recent insight that regulatory regions harbor the majority of disease-associated variants, we employ a two-step approach: first promoter regions that are likely associated to ALS are identified, and second individuals are classified based on their genotype in the selected genomic regions. Both steps employ a deep convolutional neural network. The network architecture accounts for the structure of genome data by applying convolution only to parts of the data where this makes sense from a genomics perspective.ResultsOur approach identifies potentially ALS-associated promoter regions, and generally outperforms other classification methods. Test results support the hypothesis that non-additive combinations of variants contribute to ALS. Architectures and protocols developed are tailored toward processing population-scale, whole-genome data. We consider this a relevant first step toward deep learning assisted genotype–phenotype association in whole genome-sized data.Availability and implementationOur code will be available on Github, together with a synthetic dataset (https://github.com/byin-cwi/ALS-Deeplearning). The data used in this study is available to bona-fide researchers upon request.Supplementary information Supplementary data are available at Bioinformatics online.

POLYBiNN: Binary Inference Engine for Neural Networks using Decision Trees

Convolutional Neural Networks (CNNs) and Deep Neural Networks (DNNs) have gained significant popularity in several classification and regression applications. The massive computation and memory requirements of DNN and CNN architectures pose particular challenges for their FPGA implementation. Moreover, programming FPGAs requires hardware-specific knowledge that many machine-learning researchers do not possess. To make the power and versatility of FPGAs available to a wider deep learning user community and to improve DNN design efficiency, we introduce POLYBiNN, an efficient FPGA-based inference engine for DNNs and CNNs. POLYBiNN is composed of a stack of decision trees, which are binary classifiers in nature, and it utilizes AND-OR gates instead of multipliers and accumulators. POLYBiNN is a memory-free inference engine that drastically cuts hardware costs. We also propose a tool for the automatic generation of a low-level hardware description of the trained POLYBiNN for a given application. We evaluate POLYBiNN and the tool for several datasets that are normally solved using fully connected layers. On the MNIST dataset, when implemented in a ZYNQ-7000 ZC706 FPGA, the system achieves a throughput of up to 100 million image classifications per second with 90 ns latency and 97.26% accuracy. Moreover, POLYBiNN consumes 8× less power than the best previously published implementations, and it does not require any memory access. We also show how POLYBiNN can be used instead of the fully connected layers of a CNN and apply this approach to the CIFAR-10 dataset.

Direction-Aware Spatial Context Features for Shadow Detection and Removal

Shadow detection and shadow removal are fundamental and challenging tasks, requiring an understanding of the global image semantics. This paper presents a novel deep neural network design for shadow detection and removal by analyzing the spatial image context in a direction-aware manner. To achieve this, we first formulate the direction-aware attention mechanism in a spatial recurrent neural network (RNN) by introducing attention weights when aggregating spatial context features in the RNN. By learning these weights through training, we can recover direction-aware spatial context (DSC) for detecting and removing shadows. This design is developed into the DSC module and embedded in a convolutional neural network (CNN) to learn the DSC features at different levels. Moreover, we design a weighted cross entropy loss to make effective the training for shadow detection and further adopt the network for shadow removal by using a euclidean loss function and formulating a color transfer function to address the color and luminosity inconsistencies in the training pairs. We employed two shadow detection benchmark datasets and two shadow removal benchmark datasets, and performed various experiments to evaluate our method. Experimental results show that our method performs favorably against the state-of-the-art methods for both shadow detection and shadow removal.

Reduced-Complexity Deep Neural Networks Design Using Multi-Level Compression

Deep Neural Network has achieved great success in many fields. However, many DNN models are both deep and large thereby causing high storage and energy consumption during the training and inference phases. This paper proposes multi-level compression framework. By utilizing cross-layer parameter-reducing techniques ranging from structure compression to weight compression to representation compression, the proposed compression strategy can enable order-of-magnitude reduction in network size for both training and inference with negligible accuracy loss, thereby leading to very high-efficiency and high-accuracy DNN models. Experiments show that the proposed strategy can achieve around 1.8K compression ratio in terms of dense matrices and around 30x for the overall model.