Traditional Deep Neural Networks Research Articles

GANTL: Towards Practical and Real-Time Topology Optimization with Conditional GANs and Transfer Learning

Abstract Many machine learning methods have been recently developed to circumvent the high computational cost of the gradient-based topology optimization. These methods typically require extensive and costly datasets for training, have a difficult time generalizing to unseen boundary and loading conditions and to new domains, and do not take into consideration topological constraints of the predictions, which produces predictions with inconsistent topologies. We present a deep learning method based on generative adversarial networks for generative design exploration. The proposed method combines the generative power of conditional GANs with the knowledge transfer capabilities of transfer learning methods to predict optimal topologies for unseen boundary conditions. We also show that the knowledge transfer capabilities embedded in the design of the proposed algorithm significantly reduces the size of the training dataset compared to the traditional deep learning neural or adversarial networks. Moreover, we formulate a topological loss function based on the bottleneck distance obtained from the persistent diagram of the structures and demonstrate a significant improvement in the topological connectivity of the predicted structures. We use numerous examples to explore the efficiency and accuracy of the proposed approach for both seen and unseen boundary conditions in 2D.

Vertical federated DNN training

In the training process of distributed machine learning, the data possessed at distinct companies usually contain different features. Labels may even be lacked at certain companies. Therefore, one option is that multiple companies perform joint training in the form of federated learning (FL). Existing studies on FL confines to traditional machine learning (ML), including linear regression, logistic regression, and decision tree. In this work, vertical federated learning is extended from traditional ML to deep neural networks (DNN). A privacy-preserving DNN model training scheme based on homomorphic encryption is proposed for vertically segmented datasets. It is shown that the proposed federated DNN scheme can achieve similar precision as that of the traditional centralized DNN scheme.

Explainable scale distillation for hyperspectral image classification

The land-covers within an observed remote sensing scene are usually of different scales; therefore, the ensemble of multi-scale information is a commonly used strategy to achieve more accurate scene interpretation; however, this process suffers from being time-consuming. In terms of this issue, this paper proposes a scale distillation network to explore the possibility that single-scale classification network can achieve the same (or even better) classification performance compared with multi-scale one. The proposed scale distillation network consists of a cumbersome multi-scale teacher network and a lightweight single-scale student network. The former is trained for multi-scale information learning, and the latter improves the classification accuracy by accepting the knowledge from the multi-scale teacher network and its true label. The experimental results show the advantages of scale distillation on hyperspectral image classification. The single-scale student network can even achieve higher evaluation accuracy than the multi-scale teacher network. In addition, a faithful explainable scale network is designed to visually explain the trained scale distillation network. The traditional deep neural network is a black-box and lacks interpretability. The explanation of the trained network can explore more hidden information from the predictions. We visually explain the prediction results of scale distillation network, and the results show that the explainable scale network can more precisely analyze the relationship between the learned scale features and the land-cover categories. Moreover, the possible application of the explainable scale network on classification is further discussed in this study.

The Research Progress of the Deep Hybrid Model in the Field of Text Classification

The traditional deep convolution neural network model cannot extract the context information effectively in dealing with complex long text data sets, and it is difficult to obtain the deep semantic information of the text. The deep hybrid neural network can optimize and improve the local feature extraction ability of CNN model while preserving the ability of local feature extraction. It has achieved good performance on complex data sets, so it has been paid more and more attention by researchers. Firstly, this article sorts out the current mainstream text classification data sets. Secondly, the model of the hybrid neural network based on the convolutional neural network construction is as follows: the improvement of the CNN model; technology fusion based on CNN model and the CNN-based model mixing three categories carry out analysis and sorting. Finally summarize the current problems in the text classification field, and look forward to future development and research.

Image Colorization Progress: A Review of Deep Learning Techniques for Automation of Colorization

Image colorization is the process of taking an input gray- scale (black and white) image and then producing an output colorized image that represents the semantic color tones of the input. Since the past few years, the process of automatic image colorization has been of significant interest and a lot of progress has been made in the field by various researchers. Image colorization finds its application in many domains including medical imaging, restoration of historical documents, etc. There have been different approaches to solve this problem using Convolutional Neural Networks as well as Generative Adversarial Networks. These colorization networks are not only based on different architectures but also are tested on varied data sets. This paper aims to cover some of these proposed approaches through different techniques. The results between the generative models and traditional deep neural networks are compared along with presenting the current limitations in those. The paper proposes a summarized view of past and current advances in the field of image colorization contributed by different authors and researchers.

ForceNet: An offline cutting force prediction model based on neuro-physical learning approach

Accurate cutting force prediction serves as an important reference to the optimization of numerically controlled machining process. Traditional cutting force modeling via theoretical cutting mechanism hampers accurate prediction for actual machining process due to its highly suppressed modeling flexibility. On the other hand, machine learning based modeling approaches demand large amount of diversified labeled samples to achieve comparable prediction results, while collecting these samples can be tedious and costly because the cutter workpiece engagement (CWE) keeps changing during actual process. This paper presents a cutting force prediction model, named ForceNet, which incorporates elementary physical priori into structured neural networks to predict cutting force for end-milling process of complex CWE. The main idea is to use grayscale images to represent CWE geometry, providing a universal input to the ForceNet. Unlike traditional deep neural networks served as an unexplainable black box, the core of the ForceNet is constructed by the vector summation of directional primitive cutting force elements, which are approximated using elementary neural networks. Preliminary results indicate that ForceNet outperformed existing methods not only with greater prediction accuracy in unseen cutting situations, but also with less training data needed thanks to its inherent neuro-physical structure.

HADLN: Hybrid Attention-Based Deep Learning Network for Automated Arrhythmia Classification.

In recent years, with the development of artificial intelligence, deep learning model has achieved initial success in ECG data analysis, especially the detection of atrial fibrillation. In order to solve the problems of ignoring the correlation between contexts and gradient dispersion in traditional deep convolution neural network model, the hybrid attention-based deep learning network (HADLN) method is proposed to implement arrhythmia classification. The HADLN can make full use of the advantages of residual network (ResNet) and bidirectional long–short-term memory (Bi-LSTM) architecture to obtain fusion features containing local and global information and improve the interpretability of the model through the attention mechanism. The method is trained and verified by using the PhysioNet 2017 challenge dataset. Without loss of generality, the ECG signal is classified into four categories, including atrial fibrillation, noise, other, and normal signals. By combining the fusion features and the attention mechanism, the learned model has a great improvement in classification performance and certain interpretability. The experimental results show that the proposed HADLN method can achieve precision of 0.866, recall of 0.859, accuracy of 0.867, and F1-score of 0.880 on 10-fold cross-validation.

Handwritten Bangla Numerical Digit Recognition Using Fine Regulated Deep Neural Network

The recognition of handwritten Bangla digit is providing significant progress on optical character recognition (OCR). It is a very critical task due to the similar pattern and alignment of handwriting digits. With the progress of modern research on optical character recognition, it is reducing the complexity of the classification task by several methods, a few problems encounter during recognition and wait to be solved with simpler methods. The modern emerging field of artificial intelligence is the Deep Neural Network, which promises a solid solution to these few handwritten recognition problems. This paper proposed a fine regulated deep neural network (FRDNN) for the handwritten numeric character recognition problem that uses convolutional neural network (CNN) models with regularization parameters which makes the model generalized by preventing the overfitting. This paper applied Traditional Deep Neural Network (TDNN) and Fine regulated deep neural network (FRDNN) models with a similar layer experienced on BanglaLekha-Isolated databases and the classification accuracies for the two models were 96.25% and 96.99%, respectively over 100 epochs. The network performance of the FRDNN model on the BanglaLekha-Isolated digit dataset was more robust and accurate than the TDNN model and depend on experimentation. Our proposed method is obtained a good recognition accuracy compared with other existing available methods.

Perceptual similarity measurement based on generative adversarial neural networks in graphics design

Measuring the similarity between images is of paramount importance in computer vision. However, the commonly used pixelwise similarity metrics do not match well with perceptual similarity. The purpose of this paper is to propose a visual similarity measurement method, which can be effectively used for plagiarism detection in graphic design. Plagiarism detection of designs refers to the identification and determination of major similarities. It is difficult to carry out the similarity learning process in traditional deep neural network due to the insufficient of training samples. To overcome this problem, a novel scheme is proposed for measuring perceptual similarity of graphics by using a constraint Generative Adversarial Network (GAN) model. The generator of GAN is used to create similar graphics following the common plagiarism features of logo design. Unlike the traditional discriminator which judges the authenticity of the generated image and the original image, the modified discriminator is used to calculate the perceptual similarity of the graphics pair. In graphics design, plagiarism mainly focuses on the changes of shape, color and style, which has certain cognitive subjectivity. Therefore, design experts were invited to participate in a group of cognitive analysis experiments. A perceptual constraint model is established to limit the generation of plagiarized graphics according to “design and visual rationality”. Promising results demonstrate that the proposed method can be used for plagiarism detection of logo design. Given its effectiveness and conceptual simplicity, I hope it can serve as a baseline and contribute to the future research on plagiarism detection of artworks.

RNN-LSTM Based Deep Learning Model for Tor Traffic Classification

ABSTRACT Tor is an anonymous browser software running on an overlay network. Due to the nature of the end-to-end encryption channel, it is hard to analyse the network traffic. Thus, intruders prefer the Tor browser to hide their identity and access the offensive content. Tor relays are secure from network monitoring, tracking and surveillance. There are so many research contributions for tracking the network traffic and classifying it based on various features and attributes. In this paper, we explained RNN-LSTM-based deep learning model to classify the network traffic based on their nature Tor/non-Tor. We have tested the model with open data sets ISCXTor2016 data sets and samples retrieved in our environment using CIC-flowmeter-4.0. The binary classification model using RNN-LSTM classifies the network traffic with better accuracy and precision. The same experiment conducted in the traditional deep neural network model provides large false positives and false negatives. Here we also present a detailed study and analysis of the model compare with ANN classifiers and genetic-based feature selection method.

Memory-efficient deep learning inference with incremental weight loading and data layout reorganization on edge systems

Pattern recognition applications such as face recognition and agricultural product detection have drawn a rapid interest on Cyber–Physical–Social-Systems (CPSS). These CPSS applications rely on the deep neural networks (DNN) to conduct the image classification. However, traditional DNN inference models in the cloud could suffer from network delay fluctuations and privacy leakage problems. In this regard, current real-time CPSS applications are preferred to be deployed on edge-end embedded devices. Constrained by the computing power and memory limitations of edge devices, improving the memory management efficacy is the key to improving the quality of service for model inference. First, this study explored the incremental loading strategy of model weights for the model inference. Second, the memory space at runtime is optimized through data layout reorganization from the spatial dimension. In particular, the proposed schemes are orthogonal to existing models. Experimental results demonstrate that the proposed approach reduced the memory consumption by 61.05% without additional inference time overhead.

Multidimensional Uncertainty-Aware Evidential Neural Networks

Traditional deep neural networks (NNs) have significantly contributed to the state-of-the-art performance in the task of classification under various application domains. However, NNs have not considered inherent uncertainty in data associated with the class probabilities where misclassification under uncertainty may easily introduce high risk in decision making in real-world contexts (e.g., misclassification of objects in roads leads to serious accidents). Unlike Bayesian NN that indirectly infer uncertainty through weight uncertainties, evidential NNs (ENNs) have been recently proposed to explicitly model the uncertainty of class probabilities and use them for classification tasks. An ENN offers the formulation of the predictions of NNs as subjective opinions and learns the function by collecting an amount of evidence that can form the subjective opinions by a deterministic NN from data. However, the ENN is trained as a black box without explicitly considering inherent uncertainty in data with their different root causes, such as vacuity (i.e., uncertainty due to a lack of evidence) or dissonance (i.e., uncertainty due to conflicting evidence). By considering the multidimensional uncertainty, we proposed a novel uncertainty-aware evidential NN called WGAN-ENN (WENN) for solving an out-of-distribution (OOD) detection problem. We took a hybrid approach that combines Wasserstein Generative Adversarial Network (WGAN) with ENNs to jointly train a model with prior knowledge of a certain class, which has high vacuity for OOD samples. Via extensive empirical experiments based on both synthetic and real-world datasets, we demonstrated that the estimation of uncertainty by WENN can significantly help distinguish OOD samples from boundary samples. WENN outperformed in OOD detection when compared with other competitive counterparts.

Deep Neural Network Behavioral Modeling Based on Transfer Learning for Broadband Wireless Power Amplifier

The behavior model based on the artificial neural network has been widely used in the broadband power amplifier (PA). Although the deep neural network (DNN) performs well in the PA modeling with high-dimensional inputs, the training time of the DNN model is still long. This letter proposes a PA modeling method based on transfer learning to reduce training time without sacrificing modeling performance. In the proposed method, the model can be divided into two parts. The first part is defined as a predesigned filter that can extract the features of PA, and the second part is defined as adaptation layers that can be used to fit the real PA output. Experimental results show that the proposed method can effectively reduce the training time and ensure good modeling performance compared with the traditional DNN model.

Construction of a deep sparse filtering network for rotating machinery fault diagnosis

Deep neural networks (DNNs) have shown potential in intelligent fault diagnosis of rotating machinery. However, traditional DNNs such as the back-propagation neural network are highly sensitive to the initial weights and easily fall into the local optimum, which restricts the feature learning capability and diagnostic performance. To overcome the above problems, a deep sparse filtering network (DSFN) constructed by stacked sparse filtering is developed in this paper and applied to fault diagnosis. The developed DSFN is pre-trained by sparse filtering in an unsupervised way. The back-propagation algorithm is employed to optimize the DSFN after pre-training. Then, the DSFN-based intelligent fault diagnosis method is validated using two experiments. The results show that pre-training with sparse filtering and fine-tuning can help the DSFN search for the optimal network parameters, and the DSFN can learn discriminative features adaptively from rotating machinery datasets. Compared with classical methods, the developed diagnostic method can diagnose rotating machinery faults with higher accuracy using fewer training samples.

Distance-based hyperspectral open-set classification of deep neural networks

ABSTRACT Using deep learning, the supervised hyperspectral image (HSI) classification methods are based on the ideal assumption of closed sets, in which all testing classes are defined a priori. However, it is impossible to collect all classes while training in the open set setting where unknown classes can be submitted while testing. Traditional deep neural networks lack the ability to perceive the unknown classes without external supervision; therefore, they will misclassify them as known classes. This problem is called the open set classification (OSC). This paper proposes a distance-based OSC method for deep neural networks to solve the problem. The method introduces the class anchor clustering (CAC) classifier and loss to train deep neural networks, which can make known classes cluster around fixed class centres in the logit space. Boxplot and extreme value theory (EVT) are then used to determine the distance between input instances and their nearest class centres for unknown rejection without sacrificing the classification accuracy of known classes dramatically. Many experiments on HSI datasets indicated the validity of the HSI–OSC method.

Multi-stream hybrid architecture based on cross-level fusion strategy for fine-grained crop species recognition in precision agriculture

Precision farming aims to optimizing the crop production process and managing sustainable supply chain practices as more efficient and reasonable as possible. Recently, various advanced technologies, such as deep-learning and internet of things (IoT), have achieved remarkable intelligence progress in realistic agricultural conditions. However, crops species recognition can be considered as fine-grained visual classification (FGVC) problem, suffering the low inter-class discrepancy and high intra-class variances from the subordinate categories, which is more challenging than common basic-level category classification depended on traditional deep neural networks (DNNs). This paper presents a fine-grained visual recognition model named as MCF-Net to classifying different crop species in practical farmland scenes. Proposed MCF-Net is consisted of cross stage partial network (CSPNet) as backbone module, three parallel sub-networks, and cross-level fusion module. With multi-stream hybrid architecture utilizing massive fine-granulometric information, MCF-Net obtains preferable representation ability for distinguishing interclass discrepancy and tolerating intra-class variances. In addition, the end-to-end implementation of MCF-Net is optimized by cross-level fusion strategy to accurately identify different crop categories. Several experiments on in-field CropDeepv2 datasets demonstrate that our method favorably against the state-of-the-art methods. The recognition accuracy and F1-score of MCF-Net achieves very competitive results up to 90.6% and 0.962 separately, both of which outperforming contrasted models indicate better recognition accuracy and model stability of our method. Moreover, the overall parameters of MCF-Net are only 807 MByte with achieving a good balance between model’s performance and complexity. It is acceptable and suitable to the implementation of IoT platforms in precision agricultural practices.

MDLdroidLite: A Release-and-Inhibit Control Approach to Resource-Efficient Deep Neural Networks on Mobile Devices

Mobile deep learning (MDL) has emerged as a privacy-preserving learning paradigm for mobile devices. This paradigm offers unique features such as privacy preservation, continual learning and low-latency inference to the building of personal mobile sensing applications. However, squeezing Deep Learning to mobile devices is extremely challenging due to resource constraint. Traditional Deep Neural Networks (DNNs) are usually over-parametered, hence incurring huge resource overhead for on-device learning. In this paper, we present a novel on-device deep learning framework named MDLdroidLite that transforms traditional DNNs into resource-efficient model structures for on-device learning. To minimize resource overhead, we propose a novel release-and-inhibit control (RIC) approach based on Model Predictive Control theory to efficiently grow DNNs from tiny to backbone. We also design a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">gate-based</i> fast adaptation mechanism for channel-level knowledge transformation to quickly adapt new-born neurons with existing neurons, enabling safe parameter adaptation and fast convergence for on-device training. Our evaluations show that MDLdroidLite boosts on-device training on various PMS datasets with 28× to 50× less model parameters, 4× to 10× less floating number operations than the state-of-the-art model structures while keeping the same accuracy level.

An Optimized Neural Network Model for Facial Expression Recognition over Traditional Deep Neural Networks

Emotions have a key role in Feedback analysis to provide a good customer service, the main seven emotions are Anger, Disgust, Fear, Happy, Neutral, Sad and Surprise. There are several advantages, an efficient Facial Emotion Recognition model can help us in self-discipline and control over the drivers, while they are driving the vehicle. Low resolution and Low-reliable images are main problems in this field. We proposed a new model which can efficiently perform on Low resolution and Low-reliable images. We created a low resolution facial expression dataset (LRFE) by collecting various images from different resources, which contains low resolution images. We also proposed a new hybrid filtering method, which is a combination of Gaussian, Bilateral, Non local means filtering techniques. Densenet-121 achieves 0.60 0.68 accuracy on fer2013 and LRFE respectively. When hybrid filtering method is combined with Densenet-121, it achieved 0.95 accuracy. Similarly Resnet-50, MobileNet, Xception models performed effectively when combined with the hybrid filtering method. The proposed convolutional neural network(CNN) model achieved 0.65 accuracy on fer2013 dataset, while the existing models like Resnet-50, MobileNet, Densenet-121 and Xception obtained 0.60 0.57 0.60 0.52 accuracies on fer2013 respectively. The proposed model when combined with hybrid filtering method achieved 0.85 accuracy. Clearly the proposed model outperforms the traditional methods. When the hybrid filtering method is combined with the CNN models, there is significant increase in the accuracy.

A Lightweight Convolutional Neural Network for Real-Time Facial Expression Detection

In this paper our group proposes and designs a lightweight convolutional neural network (CNN) for detecting facial emotions in real-time and in bulk to achieve a better classification effect. We verify whether our model is effective by creating a real-time vision system. This system employs multi-task cascaded convolutional networks (MTCNN) to complete face detection and transmit the obtained face coordinates to the facial emotions classification model we designed firstly. Then it accomplishes the task of emotion classification. Multi-task cascaded convolutional networks have a cascade detection feature, one of which can be used alone, thereby reducing the occupation of memory resources. Our expression classification model employs Global Average Pooling to replace the fully connected layer in the traditional deep convolution neural network model. Each channel of the feature map is associated with the corresponding category, eliminating the black box characteristics of the fully connected layer to a certain extent. At the same time, our model marries the residual modules and depth-wise separable convolutions, reducing large quantities of parameters and making the model more portable. Finally, our model is tested on the FER-2013 dataset. It only takes 3.1% of the 16GB memory, that is, only 0.496GB memory is needed to complete the task of classifying facial expressions. Not only can our model be stored in an 872.9 kilobytes file, but also its accuracy has reached 67% on the FER-2013 dataset. And it has good detection and recognition effects on those figures which are out of the dataset.

Stock Index Prices Prediction via Temporal Pattern Attention and Long-Short-Term Memory

This study attempts to predict stock index prices using multivariate time series analysis. The study’s motivation is based on the notion that datasets of stock index prices involve weak periodic patterns, long-term and short-term information, for which traditional approaches and current neural networks such as Autoregressive models and Support Vector Machine (SVM) may fail. This study applied Temporal Pattern Attention and Long-Short-Term Memory (TPA-LSTM) for prediction to overcome the issue. The results show that stock index prices prediction through the TPA-LSTM algorithm could achieve better prediction performance over traditional deep neural networks, such as recurrent neural network (RNN), convolutional neural network (CNN), and long and short-term time series network (LSTNet).