Deep Learning Problems Research Articles

A review of multimodal deep learning methods for genomic-enabled prediction in plant breeding.

Deep learning methods have been applied when working to enhance the prediction accuracy of traditional statistical methods in the field of plant breeding. Although deep learning seems to be a promising approach for genomic prediction, it has proven to have some limitations, since its conventional methods fail to leverage all available information. Multimodal deep learning methods aim to improve the predictive power of their unimodal counterparts by introducing several modalities (sources) of input information. In this review, we introduce some theoretical basic concepts of multimodal deep learning and provide a list of the most widely used neural network architectures in deep learning, as well as the available strategies to fuse data from different modalities. We mention some of the available computational resources for the practical implementation of multimodal deep learning problems. We finally performed a review of applications of multimodal deep learning to genomic selection in plant breeding and other related fields. We present a meta-picture of the practical performance of multimodal deep learning methods to highlight how these tools can help address complex problems in the field of plant breeding. We discussed some relevant considerations that researchers should keep in mind when applying multimodal deep learning methods. Multimodal deep learning holds significant potential for various fields, including genomic selection. While multimodal deep learning displays enhanced prediction capabilities over unimodal deep learning and other machine learning methods, it demands more computational resources. Multimodal deep learning effectively captures intermodal interactions, especially when integrating data from different sources. To apply multimodal deep learning in genomic selection, suitable architectures and fusion strategies must be chosen. It is relevant to keep in mind that multimodal deep learning, like unimodal deep learning, is a powerful tool but should be carefully applied. Given its predictive edge over traditional methods, multimodal deep learning is valuable in addressing challenges in plant breeding and food security amid a growing global population.

Physics-informed deep learning for structural dynamics under moving load

Physics-informed deep learning has emerged as a promising approach that incorporates physical constraints into the model, reduces the amount of data required, and demonstrates robustness and potential in dealing with limited datasets for a variety of studies. However, several key challenges still exist, with one being the spectral bias problem of deep learning in the simulation of functions with multi-frequency features. To overcome the challenge, this study proposes a novel physics-informed deep learning method, which integrates physics-informed neural network with Fourier transform so as to solve partial differential equations in the frequency domain, thus alleviating the problem of spectral bias of neural networks in the simulation of multi-frequency functions. In addition, the proposed method is used to focus on the forward simulation and parameter inverse identification issues in structural dynamics under moving loads. To illustrate the superiority of the method, the issues of dynamic response of simply supported beams under moving loads are presented as case studies, and the performance of the method in multiple cases is analysed and discussed. The research results demonstrate the feasibility and effectiveness of the method for structural dynamics simulation and parameter inverse identifications using limited datasets.

Edge-Intelligence-Powered Joint Computation Offloading and Unmanned Aerial Vehicle Trajectory Optimization Strategy

UAV-based air-ground integrated networks offer a significant benefit in terms of providing ubiquitous communications and computing services for Internet of Things (IoT) devices. With the empowerment of edge intelligence (EI) technology, they can efficiently deploy various intelligent IoT applications. However, the trajectory of UAVs can significantly affect the quality of service (QoS) and resource optimization decisions. Joint computation offloading and UAV trajectory optimization bring many challenges, including coupled decision variables, information uncertainty, and long-term queue delay constraints. Therefore, this paper introduces an air-ground integrated architecture with EI and proposes a TD3-based joint computation offloading and UAV trajectory optimization (TCOTO) algorithm. Specifically, we use the principle of the TD3 algorithm to transform the original problem into a cumulative reward maximization problem in deep reinforcement learning (DRL) to obtain the UAV trajectory and offloading strategy. Additionally, the Lyapunov framework is used to convert the original long-term optimization problem into a deterministic short-term time-slot problem to ensure the long-term stability of the UAV queue. Based on the simulation results, it can be concluded that our novel TD3-based algorithm effectively solves the joint computation offloading and UAV trajectory optimization problems. The proposed algorithm improves the performance of the system energy efficiency by 3.77%, 22.90%, and 67.62%, respectively, compared to the other three benchmark schemes.

Harnessing Dynamic Heterogeneous Redundancy to Empower Deep Learning Safety and Security

The rapid development of deep learning (DL) models has been accompanied by various safety and security challenges, such as adversarial attacks and backdoor attacks. By analyzing the current literature on attacks and defenses in DL, we find that the ongoing adaptation between attack and defense makes it impossible to completely resolve these issues. In this paper, we propose that this situation is caused by the inherent flaws of DL models, namely non-interpretability, non-recognizability, and non-identifiability. We refer to these issues as the Endogenous Safety and Security (ESS) problems. To mitigate the ESS problems in DL, we propose using the Dynamic Heterogeneous Redundant (DHR) architecture. We believe that introducing diversity is crucial for resolving the ESS problems. To validate the effectiveness of this approach, we conduct various case studies across multiple application domains of DL. Our experimental results confirm that constructing DL systems based on the DHR architecture is more effective than existing DL defense strategies.

A data-driven approach for volume feature recognition based on cell graph

ABSTRACT Machining feature recognition is one of the key technologies for realizing automated process planning. The machining feature represented by volume contains more complete information due to its property of forming a closed region which facilitates subsequent process planning. The existing volume feature recognition approaches mainly rely on expert-defined rules to combine volumes into the desired feature. When dealing with some complex parts, volume feature recognition faces combinatorial explosion, and it is difficult to formulate complete combination rules manually. This research proposes a novel data-driven approach for volume machining feature recognition. The proposed approach does not require manually defined rules and can learn various complex volume combination rules from the data, which can be implicitly represented by the structure and parameters of the data-driven model. The volume machining feature recognition is first converted to machining feature subgraph (MFS) detection by creating the zone graph. MFS detection can then be achieved by graph edge cut. Finally, the edge cut is modeled as a binary classification deep learning problem, which is solved by graph neural network. Two datasets are created, and case studies are carried out on them. The results show that the proposed approach is effective for volume machining feature recognition.

A brief review of hypernetworks in deep learning

Hypernetworks, or hypernets for short, are neural networks that generate weights for another neural network, known as the target network. They have emerged as a powerful deep learning technique that allows for greater flexibility, adaptability, dynamism, faster training, information sharing, and model compression. Hypernets have shown promising results in a variety of deep learning problems, including continual learning, causal inference, transfer learning, weight pruning, uncertainty quantification, zero-shot learning, natural language processing, and reinforcement learning. Despite their success across different problem settings, there is currently no comprehensive review available to inform researchers about the latest developments and to assist in utilizing hypernets. To fill this gap, we review the progress in hypernets. We present an illustrative example of training deep neural networks using hypernets and propose categorizing hypernets based on five design criteria: inputs, outputs, variability of inputs and outputs, and the architecture of hypernets. We also review applications of hypernets across different deep learning problem settings, followed by a discussion of general scenarios where hypernets can be effectively employed. Finally, we discuss the challenges and future directions that remain underexplored in the field of hypernets. We believe that hypernetworks have the potential to revolutionize the field of deep learning. They offer a new way to design and train neural networks, and they have the potential to improve the performance of deep learning models on a variety of tasks. Through this review, we aim to inspire further advancements in deep learning through hypernetworks.

Application of deep learning models in the identification and screening of fake news

As data sets and data streams continue to expand, traditional machine learning is becoming less effective in predicting fake news. This paper is a review of deep learning in fake news detection and prevention. Author takes the model based on convolutional neural network as an example to illustrate the principle and application of deep learning in fake news detection, including OPCNN-FAKE, Dual-channel Convolutional Neural Networks with Attention-pooling (DC-CNN) model which is completely based on Convolutional Neural Network (CNN), and Convolutional Neural Network-Long Short Term Memory (CNN-LSTM) model which combines convolutional neural network with long-short time model. These models have obvious advantages in accuracy over traditional machine learning models. This paper then points out the problems of deep learning in the field of fake news identification: it does not have good scalability and slow training speed. The author proposes possible solutions, and widely uses transfer learning and uses distributed computing platforms, such as spark, to train models. Hope this review can help the research on fake news prediction using deep learning.

Temporal self-supervised domain adaptation network for machinery fault diagnosis under multiple non-ideal conditions

Although deep learning techniques have been widely used in machinery fault diagnosis, the key issue of feature learning under multiple non-ideal conditions, i.e., strong noise interference, inconsistent data distributions and class imbalance, is not tackled effectively. Thus, a novel deep neural network (DNN) named temporal self-supervised domain adaptation network (TSSDAN) is developed for machinery fault diagnosis in this article. Firstly, a temporal self-supervised learning (TSSL) method is implemented to tackle the issue of strong noise interference and class imbalance. Secondly, a specific feature extractor, perception auto-encoder (PAE) is implemented for signal feature learning by mimicking the working mode of the biological visual system and spinal reflex system. Thirdly, a cascaded domain adaptation (CDA) method is constructed to tackle the domain shift issue by considering the feature distribution discrepancy of various domains for all perception levels. Two cases (i.e., rotor and gearbox) are used in this study to evaluate the performance of TSSDAN. The experiment results illustrate the outperformance of TSSDAN in solving the problem of deep feature learning and machinery fault diagnosis under multiple non-ideal conditions.

Enhancing spatial modeling and risk mapping of six air pollutants using synthetic data integration with convolutional neural networks

Air pollution poses significant risks to human health and the environment, necessitating effective air quality management strategies. This study presents a novel approach to air quality management by integrating an autoencoder (AE) with a convolutional neural network (CNN) algorithm in Tehran city of Iran. One of the primary and vital problems in deep learning is model complexity, and the complexity of a model is affected by data distribution, data complexity, and information volume. AE provide a helpful way to denoise input data and make building deep learning models much more efficient. The proposed methodology enables spatial modeling and risk mapping of six air pollutants, namely, particulate matter 2.5 (PM2.5), particulate matter 10 (PM10), sulfur dioxide (SO2), nitrogen dioxide (NO2), ozone (O3), and carbon monoxide (CO). For air pollution modelling, data from a spatial database containing the annual average of six pollutants from 2012 to 2022 was utilized. The model considered various parameters influencing air pollution: altitude, humidity, distance to industrial areas, NDVI (normalized difference vegetation index), population density, rainfall, distance to the street, temperature, traffic volume, wind direction, and wind speed. The risk map accuracy was assessed using the area under the receiver operating characteristic (ROC) curve for six pollutants. Among them, NO2, PM10, CO, PM2.5, O3, and SO2 exhibited the highest accuracy with values of 0.964, 0.95, 0.896, 0.878, 0.877, and 0.811, respectively, in the risk map generated by the CNN-AE model. The findings demonstrated the CNN-AE model’s impressive precision when generating the pollution risk map.

On the Number of Linear Regions of Convolutional Neural Networks With Piecewise Linear Activations.

One fundamental problem in deep learning is understanding the excellent performance of deep Neural Networks (NNs) in practice. An explanation for the superiority of NNs is that they can realize a large family of complicated functions, i.e., they have powerful expressivity. The expressivity of a Neural Network with Piecewise Linear activations (PLNN) can be quantified by the maximal number of linear regions it can separate its input space into. In this paper, we provide several mathematical results needed for studying the linear regions of Convolutional Neural Networks with Piecewise Linear activations (PLCNNs), and use them to derive the maximal and average numbers of linear regions for one-layer PLCNNs. Furthermore, we obtain upper and lower bounds for the number of linear regions of multi-layer PLCNNs. Our results suggest that deeper PLCNNs have more powerful expressivity than shallow PLCNNs, while PLCNNs have more expressivity than fully-connected PLNNs per parameter, in terms of the number of linear regions.

Enhancing active noise control through stacked autoencoders: Training strategies, comparative analysis, and evaluation with practical setup

This work is motivated by the imperative to overcome the limitations of conventional active noise control (ANC) techniques. These limitations are rooted in linear systems like the least mean square algorithm when faced with nonlinear distortions in the acoustic environment and system electronics, such as loudspeakers. This work addresses the nonlinear ANC problem by conceptualizing ANC as a deep learning problem. A novel stacked autoencoder (SAE) model is proposed, which is trained to estimate the noise that should be played at a loudspeaker so that the noise received at an error microphone may be attenuated. ANC experiments are initially set up in an anechoic room to study the fundamental performance characteristics of the architecture without extraneous considerations of room impulse response. This is followed by performance analysis in a normal reflective room. Actual noise, comprising broadband and tonal noise, is recorded in both an anechoic room and a normal reflective room. The importance of these experiments in the design process lies in their ability to account for nonlinear effects in actual setups, surpassing the limitations of relying solely on models. Experiment outcomes demonstrate that the proposed SAE model yields a noise reduction (NR) of up to 26.44 dB for wideband noise and up to 35.74 dB for tonal noise types. Additionally, there is an improvement in the extent of the quiet zone provided by the proposed SAE for tonal noise cases as compared to the traditional filtered-x least mean square (FxLMS) technique.

A Review on Rainfall Nowcasting Methodology using different ML and DL Models

Weather is influenced by several factors such as temperature, pressure, air movement, moisture/water vapor, and the Earth's rotation. Forecasting weather with high geographic precision is both challenging and computationally demanding. This study explores a nowcasting method for meteorological radar images. Building on the concept of an unsupervised representation problem in deep learning, next-frame prediction is an emerging area in computer vision where future images are predicted based on previous ones. This has numerous applications, including in robotics and autonomous driving. The paper reviews the latest next-frame prediction networks, categorizing them into two main approaches: machine learning and deep learning. It compares these strategies by discussing their respective advantages and disadvantages. Lastly, the paper outlines the most promising directions for future research.

Irregular Event Classification in Surveillance Videos Using an Auto-Encoder Deep Learning Framework

Anomaly detection is a major deep-learning problem that has a vast area to work upon. We have used a Convolutional Long Short-Term Memory (LSTM) Auto-encoder for the detection of irregular events based on regular events. Anomaly detection can be approached as a supervised problem, but the problem arises when we dig into varieties of anomalies which has many options and can be difficult to supervise, so the use of an unsupervised method is taken, and the results are calculated based on the regularity score of the sequences. The higher the score of regularity, the higher the chances of regularity in the sequence of video frames. The anomaly is detected on the frame level i.e. which frame has the irregularity, and which has regularity in it. The proposed model outperforms in terms of precision (0.97), recall (0.91), and f-measure (0.94) than the existing techniques.

Deep deductive reasoning is a hard deep learning problem

Deep Deductive Reasoning refers to the training and then executing of deep learning systems to perform deductive reasoning in the sense of formal, mathematical logic. We discuss why this is an interesting and relevant problem to study, and explore how hard it is as a deep learning problem. In particular, we present some of the progress made on this topic in recent years, understand some of the theoretical limitations that can be assessed from existing literature, and discuss negative results we have obtained regarding improving on the state of the art.

Ship-Radiated Noise Separation in Underwater Acoustic Environments Using a Deep Time-Domain Network

Ship-radiated noise separation is critical in both military and economic domains. However, due to the complex underwater environments with multiple noise sources and reverberation, separating ship-radiated noise poses a significant challenge. Traditionally, underwater acoustic signal separation has employed blind source separation methods based on independent component analysis. Recently, the separation of underwater acoustic signals has been approached as a deep learning problem. This involves learning the features of ship-radiated noise from training data. This paper introduces a deep time-domain network for ship-radiated noise separation by leveraging the power of parallel dilated convolution and group convolution. The separation layer employs parallel dilated convolution operations with varying expansion factors to better extract low-frequency features from the signal envelope while preserving detailed information. Additionally, we use group convolution to reduce the expansion of network size caused by parallel convolution operations, enabling the network to maintain a smaller size and computational complexity while achieving good separation performance. The proposed approach is demonstrated to be superior to the other common networks in the DeepShip dataset through comprehensive comparisons.

EResNet-SVM: an overfitting-relieved deep learning model for recognition of plant diseases and pests.

The accurate recognition and early warning for plant diseases and pests are a prerequisite of intelligent prevention and control for plant diseases and pests. As a result of the phenotype similarity of the hazarded plant after plant diseases and pests occur, as well as the interference of the external environment, traditional deep learning models often face the overfitting problem in phenotype recognition of plant diseases and pests, which leads to not only the slow convergence speed of the network, but also low recognition accuracy. Motivated by the above problems, the present study proposes a deep learning model EResNet-support vector machine (SVM) to alleviate the overfitting for the recognition and classification of plant diseases and pests. First, the feature extraction capability of the model is improved by increasing feature extraction layers in the convolutional neural network. Second, the order-reduced modules are embedded and a sparsely activated function is introduced to reduce model complexity and alleviate overfitting. Finally, a classifier fused by SVM and fully connected layers are introduced to transforms the original non-linear classification problem into a linear classification problem in high-dimensional space to further alleviate the overfitting and improve the recognition accuracy of plant diseases and pests. The ablation experiments further demonstrate that the fused structure can effectively alleviate the overfitting and improve the recognition accuracy. The experimental recognition results for typical plant diseases and pests show that the proposed EResNet-SVM model has 99.30% test accuracy for eight conditions (seven plant diseases and one normal), which is 5.90% higher than the original ResNet18. Compared with the classic AlexNet, GoogLeNet, Xception, SqueezeNet and DenseNet201 models, the accuracy of the EResNet-SVM model has improved by 5.10%, 7%, 8.10%, 6.20% and 1.90%, respectively. The testing accuracy of the EResNet-SVM model for 6 insect pests is 100%, which is 3.90% higher than that of the original ResNet18 model. This research provides not only useful references for alleviating the overfitting problem in deep learning, but also a theoretical and technical support for the intelligent detection and control of plant diseases and pests. © 2024 Society of Chemical Industry.

Deep Reinforcement Learning for Mobile Robot Path Planning

Path planning is an important problem with the the applications in many aspects, such as video games, robotics etc. This paper proposes a novel method to address the problem of Deep Reinforcement Learning (DRL) based path planning for a mobile robot. We design DRL-based algorithms, including reward functions, and parameter optimization, to avoid time-consuming work in a 2D environment. We also designed an Two-way search hybrid A* algorithm to improve the quality of local path planning. We transferred the designed algorithm to a simple embedded environment to test the computational load of the algorithm when running on a mobile robot. Experiments show that when deployed on a robot platform, the DRL-based algorithm in this article can achieve better planning results and consume less computing resources.

Comparative study on the performance of ConvLSTM and ConvGRU in classification problems—taking early warning of short-duration heavy rainfall as an example

Convolutional long short-term memory (ConvLSTM) and convolutional gated recurrent unit (ConvGRU) are two widely adopted deep learning models that combine recurrent mechanisms with convolutional operations for spatiotemporal sequences forecasting. To clarify the convergence speed and classification ability of the above two models, using the same model architecture to predict the same classification problem is necessary. This research treats the district-level warning of short-duration heavy rainfall in Beijing as a binary classification problem in deep learning, and composite radar reflectivity data of the Beijing–Tianjin–Hebei radar network and rainfall data from automatic weather stations in Beijing are used for training and performance evaluation. The results show that the convergence speed of ConvGRU is approximately 25% faster than that of ConvLSTM. The early-warning performances of ConvLSTM and ConvGRU have similar trends with region, time, and rain intensity, but most of the scores of ConvLSTM are higher, and in a few cases, ConvGRU has higher scores.摘要卷积长短期记忆单元ConvLSTM和卷积门控循环单元ConvGRU是两种广泛应用的深度学习单元, 通过将循环机制与卷积运算相结合, 常常用于时空序列的预测. 为了明确上述两种模型的收敛速度和分类能力, 需要使用相同的模型架构对相同的分类问题进行预测. 本研究将北京短时强降水区级预警问题看作深度学习中的二分类问题, 使用京津冀雷达网的组合反射率数据和北京区域内的自动气象站降雨数据进行深度学习模型的训练和评估. 结果表明, ConvGRU的收敛速度比 ConvLSTM快约25%. ConvLSTM和ConvGRU的预警性能随地区, 时间, 降雨强度的变化趋势相似, 但大部分ConvLSTM的得分较高, 少数情况下ConvGRU的得分较高.

Improving the performance of supervised deep learning for regulatory genomics using phylogenetic augmentation.

Supervised deep learning is used to model the complex relationship between genomic sequence and regulatory function. Understanding how these models make predictions can provide biological insight into regulatory functions. Given the complexity of the sequence to regulatory function mapping (the cis-regulatory code), it has been suggested that the genome contains insufficient sequence variation to train models with suitable complexity. Data augmentation is a widely used approach to increase the data variation available for model training, however current data augmentation methods for genomic sequence data are limited. Inspired by the success of comparative genomics, we show that augmenting genomic sequences with evolutionarily related sequences from other species, which we term phylogenetic augmentation, improves the performance of deep learning models trained on regulatory genomic sequences to predict high-throughput functional assay measurements. Additionally, we show that phylogenetic augmentation can rescue model performance when the training set is down-sampled and permits deep learning on a real-world small dataset, demonstrating that this approach improves data efficiency. Overall, this data augmentation method represents a solution for improving model performance that is applicable to many supervised deep-learning problems in genomics. The open-source GitHub repository agduncan94/phylogenetic_augmentation_paper includes the code for rerunning the analyses here and recreating the figures.

Improving Classification Performance in Dendritic Neuron Models through Practical Initialization Strategies.

A dendritic neuron model (DNM) is a deep neural network model with a unique dendritic tree structure and activation function. Effective initialization of its model parameters is crucial for its learning performance. This work proposes a novel initialization method specifically designed to improve the performance of DNM in classifying high-dimensional data, notable for its simplicity, speed, and straightforward implementation. Extensive experiments on benchmark datasets show that the proposed method outperforms traditional and recent initialization methods, particularly in datasets consisting of high-dimensional data. In addition, valuable insights into the behavior of DNM during training and the impact of initialization on its learning performance are provided. This research contributes to the understanding of the initialization problem in deep learning and provides insights into the development of more effective initialization methods for other types of neural network models. The proposed initialization method can serve as a reference for future research on initialization techniques in deep learning.