Deep Learning Problems Research Articles

Tender Leaf Identification for Early-Spring Green Tea Based on Semi-Supervised Learning and Image Processing

Tea is one of the most common beverages in the world. Automated machinery that is suitable for plucking high-quality green tea is necessary for tea plantations and the identification of tender leaves is one of the key techniques. In this paper, we proposed a method that combines semi-supervised learning and image processing to identify tender leaves. Both in two-dimensional and three-dimensional space, the three R, G, and B components of tender leaves and their backgrounds were trained and tested. The gradient-descent method and the Adam algorithm were used to optimize the objective function, respectively. The results show that the average accuracy of tender leaf identification is 92.62% and the average misjudgment rate is 18.86%. Our experiments have shown that green tea tender leaves in early spring can be identified effectively using the model based on semi-supervised learning, which has strong versatility and perfect adaptability, so as to improve the problem of deep learning requiring a large number of labeled samples.

Modeling and Analysis of Multifocus Picture Division Algorithm Based on Deep Learning

As a complex machine learning algorithm, deep learning can extract object shape information and more complex and advanced information in images by using a deep learning model. In order to solve some problems of deep learning in image feature extraction and classification, this paper designs a modeling method of multifocus image segmentation algorithm based on deep learning. The acceleration effect of FPGA (field programmable gate array) on deep learning and weight sharing is analyzed. By introducing deep learning, the trouble of determining the weight coefficient is eliminated, and the energy function is simplified. Therefore, the relevant parameters of multifocus image segmentation can be easily set, and better results can be obtained. The multifocus image segmentation algorithm based on deep learning can not only obtain closed and smooth segmentation curves but also adaptively deal with topology changes due to high segmentation accuracy and stable algorithm. The results show that the model effectively combines the local and global information of the image, so that the model has good robustness. The depth learning algorithm is used to calculate the average value of local inner and outer pixels of an image. Even for complex images, relatively simple contour curves can be obtained.

Combining Deep Learning and the Heat Flux Method for In-Situ Thermal-Transmittance Measurement Improvement

Transmission losses through the building envelope account for a large proportion of building energy balance. One of the most important parameters for determining transmission losses is thermal transmittance. Although thermal transmittance does not take into account dynamic parameters, it is traditionally the most commonly used estimation of transmission losses due to its simplicity and efficiency. It is challenging to estimate the thermal transmittance of an existing building element because thermal properties are commonly unknown or not all the layers that make up the element can be found due to technical-drawing information loss. In such cases, experimental methods are essential, the most common of which is the heat-flux method (HFM). One of the main drawbacks of the HFM is the long measurement duration. This research presents the application of deep learning on HFM results by applying long-short term memory units on temperature difference and measured heat flux. This deep-learning regression problem predicts heat flux after the applied model is properly trained on temperature-difference input, which is backpropagated by measured heat flux. The paper shows the performance of the developed procedure on real-size walls under the simulated environmental conditions, while the possibility of practical application is shown in pilot in-situ measurements.

Forest Fire Segmentation from Aerial Imagery Data Using an Improved Instance Segmentation Model

In recent years, forest-fire monitoring methods represented by deep learning have been developed rapidly. The use of drone technology and optimization of existing models to improve forest-fire recognition accuracy and segmentation quality are of great significance for understanding the spatial distribution of forest fires and protecting forest resources. Due to the spreading and irregular nature of fire, it is extremely tough to detect fire accurately in a complex environment. Based on the aerial imagery dataset FLAME, this paper focuses on the analysis of methods to two deep-learning problems: (1) the video frames are classified as two classes (fire, no-fire) according to the presence or absence of fire. A novel image classification method based on channel domain attention mechanism was developed, which achieved a classification accuracy of 93.65%. (2) We propose a novel instance segmentation method (MaskSU R-CNN) for incipient forest-fire detection and segmentation based on MS R-CNN model. For the optimized model, the MaskIoU branch is reconstructed by a U-shaped network in order to reduce the segmentation error. Experimental results show that the precision of our MaskSU R-CNN reached 91.85%, recall 88.81%, F1-score 90.30%, and mean intersection over union (mIoU) 82.31%. Compared with many state-of-the-art segmentation models, our method achieves satisfactory results on forest-fire dataset.

HoD-Net: High-Order Differentiable Deep Neural Networks and Applications

We introduce a deep architecture named HoD-Net to enable high-order differentiability for deep learning. HoD-Net is based on and generalizes the complex-step finite difference (CSFD) method. While similar to classic finite difference, CSFD approaches the derivative of a function from a higher-dimension complex domain, leading to highly accurate and robust differentiation computation without numerical stability issues. This method can be coupled with backpropagation and adjoint perturbation methods for an efficient calculation of high-order derivatives. We show how this numerical scheme can be leveraged in challenging deep learning problems, such as high-order network training, deep learning-based physics simulation, and neural differential equations.

An fMRI Sequence Representation Learning Framework for Attention Deficit Hyperactivity Disorder Classification

For attention deficit hyperactivity disorder (ADHD), a common neurological disease, accurate identification is the basis for treatment. In this paper, a novel end-to-end representation learning framework for ADHD classification of functional magnetic resonance imaging (fMRI) sequences is proposed. With such a framework, the complexity of the sequence representation learning neural network decreases, the overfitting problem of deep learning for small samples cases is solved effectively, and superior classification performance is achieved. Specifically, a data conversion module was designed to convert a two-dimensional sequence into a three-dimensional image, which expands the modeling area and greatly reduces the computational complexity. The transfer learning method was utilized to freeze or fine-tune the parameters of the pre-trained neural network to reduce the risk of overfitting in the cases with small samples. Hierarchical feature extraction can be performed automatically by combining the sequence representation learning modules with a weighted cross-entropy loss. Experiments were conducted both with individual imaging sites and combining them, and the results showed that the classification average accuracies with the proposed framework were 73.73% and 72.02%, respectively, which are much higher than those of the existing methods.

HFM: A Hybrid Feature Model Based on Conditional Auto Encoders for Zero-Shot Learning.

Zero-Shot Learning (ZSL) is related to training machine learning models capable of classifying or predicting classes (labels) that are not involved in the training set (unseen classes). A well-known problem in Deep Learning (DL) is the requirement for large amount of training data. Zero-Shot learning is a straightforward approach that can be applied to overcome this problem. We propose a Hybrid Feature Model (HFM) based on conditional autoencoders for training a classical machine learning model on pseudo training data generated by two conditional autoencoders (given the semantic space as a condition): (a) the first autoencoder is trained with the visual space concatenated with the semantic space and (b) the second autoencoder is trained with the visual space as an input. Then, the decoders of both autoencoders are fed by the test data of the unseen classes to generate pseudo training data. To classify the unseen classes, the pseudo training data are combined to train a support vector machine. Tests on four different benchmark datasets show that the proposed method shows promising results compared to the current state-of-the-art when it comes to settings for both standard Zero-Shot Learning (ZSL) and Generalized Zero-Shot Learning (GZSL).

Facial landmark disentangled network with variational autoencoder

Learning disentangled representation of data is a key problem in deep learning. Specifically, disentangling 2D facial landmarks into different factors (e.g., identity and expression) is widely used in the applications of face reconstruction, face reenactment and talking head et al.. However, due to the sparsity of landmarks and the lack of accurate labels for the factors, it is hard to learn the disentangled representation of landmarks. To address these problem, we propose a simple and effective model named FLD-VAE to disentangle arbitrary facial landmarks into identity and expression latent representations, which is based on a Variational Autoencoder framework. Besides, we propose three invariant loss functions in both latent and data levels to constrain the invariance of representations during training stage. Moreover, we implement an identity preservation loss to further enhance the representation ability of identity factor. To the best of our knowledge, this is the first work to end-to-end disentangle identity and expression factors simultaneously from one single facial landmark.

Low-Bitwidth Convolutional Neural Networks for Wireless Interference Identification

Wireless interference identification (WII) is critical for non-cooperative communication systems in both civilian and military scenarios. Recently, deep learning (DL) based WII methods have been proposed with impressive performances. However, these methods did not consider the quantization problem for DL based methods of WII, which is an indispensable process when deploying deep neural networks into hardware units. This paper addresses the problem of training quantized convolutional neural networks (CNNs), with low-precision weights as well as activations for WII. Optimizing a low-bit width network is very challenging due to the non-differentiable quantization function. To mitigate the difficulty of training, we propose three effective approaches. Firstly, we propose to train the quantized network with the guidance of the full-precision counterpart. The quantized network can learn from the full-precision counterpart. Unfortunately, training an extra full-precision network to assist a quantized model is cumbersome and computationally expensive. To this end, we further propose a training mechanism which makes use of the manually designed probability distributions to provide a virtual full-precision counterpart for guiding the training of the quantization network without extra computational cost. Thirdly, to make the gradient back-propagates more easily, we propose novel auxiliary output modules, which can be seamlessly incorporated into the proposed quantization networks. Experimental results validate the effectiveness of the proposed methods. Furthermore, it is shown that training 3-bit and 4-bit precision networks with the proposed methods leads to performance improvement as compared to their full precision counterparts with standard network architectures.

Correlated-informed neural networks: A new machine learning framework to predict pressure drop in micro-channels

Accurate pressure drop estimation in forced boiling phenomena is important during the thermal analysis and the geometric design of cryogenic heat exchangers. However, current methods to predict the pressure drop have one of two problems: lack of accuracy or generalization to different situations. In this work, we present the correlated-informed neural networks (CoINN), a new paradigm in applying the artificial neural network (ANN) technique combined with a successful pressure drop correlation as a mapping tool to predict the pressure drop of zeotropic mixtures in micro-channels. The proposed approach is inspired by Transfer Learning, which is highly used in deep learning problems with reduced datasets. Our method improves the ANN performance by transferring the knowledge of the Sun & Mishima correlation for the pressure drop to the ANN. The correlation having physical and phenomenological implications for the pressure drop in micro-channels considerably improves the performance and generalization capabilities of the ANN. The final architecture consists of three inputs: the mixture vapor quality, the micro-channel inner diameter, and the available pressure drop correlation. The results show the benefits gained using the correlated-informed approach predicting experimental data used for training and a posterior test with a mean relative error (mre) of 6%, lower than the Sun & Mishima correlation of 13%. Additionally, this approach can be extended to other mixtures and experimental settings, a missing feature in other approaches for mapping correlations using ANNs for heat transfer applications.

Dataset mismatched steganalysis using subdomain adaptation with guiding feature

The generalization problem in deep learning has always been an important problem to be solved. In the field of steganalysis, generalization is also an important factor that makes steganalysis models difficult to deploy in real-world scenarios. For a group of suspicious images that never appeared in the training set, the pre-trained deep learning-based steganalysis models tend to suffer from distinct performance degradation. To address this limitation, in this paper, a feature-guided subdomain adaptation steganalysis framework is proposed to improve the performance of the pre-trained models when detecting new data. Initially, the source domain and target domain will be divided into subdomains according to class, and the distributions of the relevant subdomains are aligned by subdomain adaptation. Afterward, the guiding feature is generated to make the division of subdomains more stable and precise. When it is used to detect three spatial steganographic algorithms with a wide variety of datasets and payloads, the experimental results show that the proposed steganalysis framework can significantly improve the average accuracy of SRNet model by 5.4% at 0.4bpp, 8.5% at 0.2bpp, and 8.0% at 0.1bpp in the case of dataset mismatch.

EP4DDL: addressing straggler problem in heterogeneous distributed deep learning

EP4DDL: addressing straggler problem in heterogeneous distributed deep learning

Investigating Transfer Learning in Graph Neural Networks

Graph neural networks (GNNs) build on the success of deep learning models by extending them for use in graph spaces. Transfer learning has proven extremely successful for traditional deep learning problems, resulting in faster training and improved performance. Despite the increasing interest in GNNs and their use cases, there is little research on their transferability. This research demonstrates that transfer learning is effective with GNNs, and describes how source tasks and the choice of GNN impact the ability to learn generalisable knowledge. We perform experiments using real-world and synthetic data within the contexts of node classification and graph classification. To this end, we also provide a general methodology for transfer learning experimentation and present a novel algorithm for generating synthetic graph classification tasks. We compare the performance of GCN, GraphSAGE and GIN across both synthetic and real-world datasets. Our results demonstrate empirically that GNNs with inductive operations yield statistically significantly improved transfer. Further, we show that similarity in community structure between source and target tasks support statistically significant improvements in transfer over and above the use of only the node attributes.

Imbalanced Sample Selection With Deep Reinforcement Learning for Fault Diagnosis

An imbalanced number of faulty and normal samples causes serious damage to the performance of the conventional diagnosis methods. To settle the data-imbalance fault diagnosis problem, this article presents a novel general imbalanced sample selection strategy (DiagSelect) based on deep reinforcement learning. In DiagSelect, the problem of imbalanced sample selection from the training set is formulated as a multiarmed bandit problem of deep reinforcement learning. The nondifferentiable optimization problem of imbalanced sample selection can be solved by the Markov decision process. The parameters of DiagSelect can be optimized by REINFORCE with the feedback of the validation set. DiagSelect performs intelligent imbalanced sample selection to obtain better diagnosis performance autonomously. As a data-level technique, DiagSelect can be easily used in conjunction with the conventional diagnosis models. DiagSelect is validated in a synthetic dataset and an industrial process dataset. The results have shown the effectiveness, stability, and transferability of DiagSelect.

Challenge of the performance management of trust control systems with deep learning

Introduction. The significance of machine learning under the conditions of digital transformation of industry, and methods of implementing deep learning to provide the performance of trust management systems are considered. The necessity of using convolutional artificial neural networks for deep machine learning is determined. Various technologies and architectures for the implementation of artificial neural networks are briefly considered; a comparative analysis of their performance is carried out. The work objective is to study the need to develop new approaches to the architecture of computing machines for solving problems of deep machine learning in the trust management system implementation.Materials and Methods. In the context of digital transformation, the use of artificial intelligence reaches a new level. The technical implementation of artificial neural systems with deep machine learning is based on the use of one of three basic technologies: high performance computing (HPC) with parallel data processing, neuromorphic computing (NC), and quantum computing (QC).Results. Implementation models for deep machine learning, basic technologies and architecture of computing machines, as well as requirements for trust assurance in control systems using deep machine learning are analyzed. The problem of shortage of computation power for solving such problems is identified. None of the currently existing technologies can solve the full range of learning and impedance problems. The current level of technology does not provide information security and reliability of neural networks. The practical implementation of trust management systems with deep machine learning based on existing technologies for a significant part of the tasks does not provide a sufficient level of performance.Discussion and Conclusions. The study made it possible to identify the challenge of the computation power shortage for solving problems of deep machine learning. Through the analysis of the requirements for trust management systems, the external challenges of their implementation on the basis of existing technologies, and the need to develop new approaches to the computer architecture are determined.

Study on Resistant Hierarchical Fuzzy Neural Networks

Novel resistant hierarchical fuzzy neural networks are proposed in this study and their deep learning problems are investigated. These fuzzy neural networks can be used to model complex controlled plants and can also be used as fuzzy controllers. In general, real-world data are usually contaminated by outliers. These outliers may have undesirable or unpredictable influences on the final learning machines. The correlations between the target and each of the predictors are utilized to partition input variables into groups so that each group becomes the input variables of a fuzzy system in each level of the hierarchical fuzzy neural network. In order to enhance the resistance of the learning machines, we use the least trimmed squared error as the cost function. To test the resistance of learning machines to adverse effects of outliers, we add at the output node some noise from three different types of distributions, namely, normal, Laplace, and uniform distributions. Real-world datasets are used to compare the performances of the proposed resistant hierarchical fuzzy neural networks, resistant densely connected artificial neural networks, and densely connected artificial neural networks without noise.

Following the Correct Direction: Renovating Sparsified SGD Towards Global Optimization in Distributed Edge Learning

Distributed edge learning collaborates powerful edge devices to train a shared global model. Since the frequent communication between the server and workers is very expensive, it is desired to accelerate the learning process. The gradient sparsification is an efficient method that only uploads a small subset of gradient elements. However, most existing works neglect the distributed nature of local datasets, and consequently the local gradients uploaded by edge devices cannot follow the global correct optimization direction well, which results in the loss of accuracy. In this paper, we propose a new gradient sparsification with a renovating mechanism, called Global Renovating Stochastic Gradient Descent (GRSGD). GRSGD utilizes the previous-round global gradient to estimate the current global one and renovates the current zero-sparsified local gradients. It mitigates the communication overhead while making the convergence direction of training closer to the global optimization, accelerating the distributed edge learning process. We provide a theoretical convergence guarantee for our algorithm based on the non-convex assumption, which better fits most deep learning problems. With extensive experiments in PyTorch, we show that GRSGD effectively accelerates the learning process with a smaller communication cost and a faster convergence rate on most training tasks. For example, on ImageNet MnasNet, GRSGD cuts down the gradient size from 8.47MB to 2.13MB while achieving 9.6%+ higher accuracy.

Primary-User-Friendly Dynamic Spectrum Anti-Jamming Access: A GAN-Enhanced Deep Reinforcement Learning Approach

This letter studies the problem of deep reinforcement learning (DRL)-based dynamic spectrum anti-jamming access in overlay cognitive radio networks. To prevent secondary user (SU) from interfering with primary user (PU) and being jammed by jammer, we propose a PU-friendly dynamic spectrum anti-jamming access scheme. First, a generative adversarial network (GAN)-based virtual environment is proposed to simulate spectrum environment. Then, a DRL-based channel decision network (CDN) is trained to learn the optimal spectrum access policy in the virtual environment. Finally, SU accesses spectrum environment under the guidance of the trained CDN. Simulation results show that the proposed scheme is able to elude both PU signals and jamming completely and converges much faster than the scheme that trains the CDN in spectrum environment from scratch.

A Systematic Review on Data Scarcity Problem in Deep Learning: Solution and Applications

Recent advancements in deep learning architecture have increased its utility in real-life applications. Deep learning models require a large amount of data to train the model. In many application domains, there is a limited set of data available for training neural networks as collecting new data is either not feasible or requires more resources such as in marketing, computer vision, and medical science. These models require a large amount of data to avoid the problem of overfitting. One of the data space solutions to the problem of limited data is data augmentation. The purpose of this study focuses on various data augmentation techniques that can be used to further improve the accuracy of a neural network. This saves the cost and time consumption required to collect new data for the training of deep neural networks by augmenting available data. This also regularizes the model and improves its capability of generalization. The need for large datasets in different fields such as computer vision, natural language processing, security, and healthcare is also covered in this survey paper. The goal of this paper is to provide a comprehensive survey of recent advancements in data augmentation techniques and their application in various domains.

GSCCTL: a general semi-supervised scene classification method for remote sensing images based on clustering and transfer learning

ABSTRACT Recently, much research has shown that deep learning methods are superior in scene classification for remote sensing images (HSIs). However, the lack of labelled samples and computational resources present serious obstacles for HSI scene classification. To address these problems, we consider the pragmatic deep learning problem with only a few or no labelled samples. A general semi-supervised scene classification method based on clustering and transfer learning named GSCCTL is proposed to transfer outstanding supervised learning models to semi-supervised scene classification. Specifically, the GSCCTL model iterates between (1) scene clustering and (2) transfer learning to improve scene classification performance for the HSI data set. Since the clustering results may contain noise, we propose a new method to calculate the distance between the samples and the clustering center and add a reliable sample selection operation to the data after clustering. Initially, the model’s scene classification performance is weak, and transfer learning is carried out on a few samples close to the clustering center. In subsequent iterations, the model is constantly optimized, more and more samples are selected as reliable samples for training, and the model’s scene classification performance is strengthened. Through iterative clustering and transfer learning, the model is progressively optimized and gradually converged until the end condition is reached. Extensive experiments on the UCMerced, AID, and NWPU-RESISC45 remote sensing data sets demonstrate that the GSCCTL model based on clustering and transfer learning improved scene classification accuracy with only a few labelled samples and is suitable for semi-supervised scene classification.