Multiple Deep Neural Networks Research Articles

Graph deep learning recognition of port ship behavior patterns from a network approach

The detection and recognition of ship mobile behavior patterns based on automatic identification system (AIS) data are crucial for ship navigation and maritime supervision. However, present methods, such as rule-based constraints and density clustering extraction, still suffer from restrictions imposed by artificial settings of parameters and thresholds. The lack of neighborhood context information in unstructured and serialized ship trajectory vector data makes it challenging to incorporate advanced methods such as deep learning for pattern learning. This study builds multiple semi-supervised deep learning graph neural network (GNN) models based on the message passing paradigm for recognizing ship mobile behavior patterns by constructing an AIS trajectory network and computing multi-order node features through Delaunay triangulation. It can get rid of the dependence on parameters of traditional rule-based and unsupervised learning methods. It also reduces the dependence on auxiliary information in trajectory qualitative by introducing multi-order node features. We conducted the experiments on cargo ship data from New York and Singapore to test our method. The model's classification accuracy is more than 99% for the New York dataset and more than 95% for the Singapore dataset. This method can be extended to other types of ships and traffic trajectory vector data with the excellent application potential of GNN in pattern recognition of vector mobile data. The code and experimental data are available at: https://github.com/destiny1103/DT-GNN.

Supplementation of deep neural networks with simplified physics-based features to increase accuracy of plate fundamental frequency predictions

To improve predictive machine learning-based models limited by sparse data, supplemental physics-related features are introduced into a deep neural network (DNN). While some approaches inject physics through differential equations or numerical simulation, improvements are possible using simplified relationships from engineering references. To evaluate this hypothesis, thin rectangular plates were simulated to generate training datasets. With plate dimensions and material properties as input features and fundamental natural frequency as the output, predictive performance of a data-driven DNN-based model is compared with models using supplemental inputs, such as modulus of rigidity. To evaluate model accuracy improvements, these additional features are injected into various DNN layers, and the network is trained with four different dataset sizes. When evaluated against independent data of similar features to the training sets, supplementation provides no statistically-significant prediction error reduction. However, notable accuracy gains occur when independent test data is of material and dimensions different from the original training set. Furthermore, when physics-enhanced data is injected into multiple DNN layers, reductions in mean error from 33.2% to 19.6%, 34.9% to 19.9%, 35.8% to 22.4%, and 43.0% to 28.4% are achieved for dataset sizes of 261, 117, 60, and 30, respectively, demonstrating potential for generalizability using a data supplementation approach. Additionally, when compared with other methods—such as linear regression and support vector machine (SVM) approaches—the physics-enhanced DNN demonstrates an order of magnitude reduction in percentage error for dataset sizes of 261, 117, and 60 and a 30% reduction for a size of 30 when compared with a cubic SVM model independently tested with data divergent from the training and validation set.

Data Representation for Deep Learning - Based Arabic Text Summarization Performance Using Python Results

A sequence-to-sequence model is used as the foundation for a suggested abstractive Arabic text summarizing system. Our goal is to create a sequence-to-sequence model by utilizing multiple deep artificial neural networks and determining which one performs the best. The encoder and decoder have been developed using several layers of recurrent neural networks, gated recurrent units, recursive neural networks, convolutional neural networks, long short-term memory, and bidirectional long short-term memory. We are re-implementing the fundamental summarization model in this study, which uses the sequence-to-sequence framework. Using a Google Colab Jupiter notebook that runs smoothly, we have constructed these models using the Keras library. The results further demonstrate that one of the key techniques that has led to breakthrough performance with deep neural networks is the use of Gensim for word embeddings over other text representations by abstractive summarization models, along with FastText, a library for efficient learning of word representations and sentence classification.

Ensemble detection of hand joint ankylosis and subluxation in radiographic images using deep neural networks

The modified total Sharp score (mTSS) is often used as an evaluation index for joint destruction caused by rheumatoid arthritis. In this study, special findings (ankylosis, subluxation, and dislocation) are detected to estimate the efficacy of mTSS by using deep neural networks (DNNs). The proposed method detects and classifies finger joint regions using an ensemble mechanism. This integrates multiple DNN detection models, specifically single shot multibox detectors, using different training data for each special finding. For the learning phase, we prepared a total of 260 hand X-ray images, in which proximal interphalangeal (PIP) and metacarpophalangeal (MP) joints were annotated with mTSS by skilled rheumatologists and radiologists. We evaluated our model using five-fold cross-validation. The proposed model produced a higher detection accuracy, recall, precision, specificity, F-value, and intersection over union than individual detection models for both ankylosis and subluxation detection, with a detection rate above 99.8% for the MP and PIP joint regions. Our future research will aim at the development of an automatic diagnosis system that uses the proposed mTSS model to estimate the erosion and joint space narrowing score.

Reliable ECG analysis using recognition scores from multiple deep neural networks

Reliable ECG analysis using recognition scores from multiple deep neural networks

Optimizing Job Offloading Schedule for Collaborative DNN Inference

Deep Neural Networks (DNNs) have been widely deployed in mobile applications. DNN inference latency is a critical metric to measure the service quality of those applications. Collaborative inference is a promising approach for latency optimization, where partial inference workloads are offloaded from mobile devices to cloud servers. Model partition problems for collaborative inference have been well studied. However, little attention has been paid to optimizing offloading pipeline for multiple DNN inference jobs. In practice, mobile devices usually need to process multiple DNN inference jobs simultaneously. We propose to jointly optimize the DNN partitioning and pipeline scheduling for multiple inference jobs. We theoretically analyze the optimal scheduling conditions for homogeneous chain-structure DNNs. Based on the analysis, we proposed near-optimal partitioning and scheduling methods for chain-structure DNNs. We also extend those methods for general-structure DNNs. In addition, we extend our problem scenario to handle heterogeneous DNN inference jobs. A layer-level scheduling algorithm is proposed. Theoretical analyses show that our proposed method is optimal when computation graphs are tree-structure. Our joint optimization methods are evaluated in a real-world testbed. Experiment results show that our methods can significantly reduce the overall inference latency of multiple inference jobs compared to partition-only or schedule-only approaches.

Progressive Poisoned Data Isolation for Training-Time Backdoor Defense

Deep Neural Networks (DNN) are susceptible to backdoor attacks where malicious attackers manipulate the model's predictions via data poisoning. It is hence imperative to develop a strategy for training a clean model using a potentially poisoned dataset. Previous training-time defense mechanisms typically employ an one-time isolation process, often leading to suboptimal isolation outcomes. In this study, we present a novel and efficacious defense method, termed Progressive Isolation of Poisoned Data (PIPD), that progressively isolates poisoned data to enhance the isolation accuracy and mitigate the risk of benign samples being misclassified as poisoned ones. Once the poisoned portion of the dataset has been identified, we introduce a selective training process to train a clean model. Through the implementation of these techniques, we ensure that the trained model manifests a significantly diminished attack success rate against the poisoned data. Extensive experiments on multiple benchmark datasets and DNN models, assessed against nine state-of-the-art backdoor attacks, demonstrate the superior performance of our PIPD method for backdoor defense. For instance, our PIPD achieves an average True Positive Rate (TPR) of 99.95% and an average False Positive Rate (FPR) of 0.06% for diverse attacks over CIFAR-10 dataset, markedly surpassing the performance of state-of-the-art methods. The code is available at https://github.com/RorschachChen/PIPD.git.

Critical Path-Based Backdoor Detection for Deep Neural Networks.

Backdoor attack to deep neural networks (DNNs) is among the predominant approaches to bring great threats into artificial intelligence. The existing methods to detect backdoor attacks focus on the perspective of distributions in DNNs, however, limited by its ability of generalization across DNN models. In this article, a critical-path-based backdoor detector (CPBD) is proposed, which approaches to detect backdoor attacks via DNN's interpretability. CPBD is designed to efficiently discover the characteristics of backdoors, which distinguish the critical paths in the attacked DNNs. To deal with the intractably large number of neurons, we propose to simplify the neurons, and the preserved key nodes are integrated into a set of critical paths. Thus, a DNN model can be formulated as a combination of several critical paths. Afterward, the detection of backdoors is performed based on the analysis of critical paths corresponding to different classes. Then, combining all the above steps, the CPBD algorithm is integrated to present the results in a standard and systematic manner. In addition, CPBD is able to locate neurons associated with malicious triggers, the combination of which is named as trigger propagation path. Extensive experiments are conducted, which testify the efficiency of the proposed method on multiple DNNs and different trigger sizes.

Time‐based DDoS attack detection through hybrid LSTM‐CNN model architectures: An investigation of many‐to‐one and many‐to‐many approaches

SummaryInternet data thefts, intrusions and DDoS attacks are some of the big concerns for the network security today. Detection of these anomalies, is gaining tremendous impetus with the development of machine learning and artificial intelligence. Even now researchers are shifting the base from machine learning to the deep neural architectures with auto‐feature selection capabilities. We in this paper propose multiple deep neural network architectures which can select, co‐learn and teach the gradients of the neural network by itself with no human intervention. This is what we call as meta‐learning. The models are configured in both many to one and many to many design architectures. We combine long short‐term memory (LSTM), bi‐directional long short‐term memory (BiLSTM), convolutional neural network (CNN) layers along with attention mechanism to achieve the higher accuracy values among all the available deep learning model architectures. LSTMs overcomes the vanishing and exploding gradient problem of RNN and attention mechanism mimics the human cognitive attention that screens the network flow to obtain the key features for network traffic classification. In addition, we also add multiple convolutional layers to get the key features for network traffic classification. We get the time series analysis of the traffic done for the possibility of a DDoS attack without using any feature selection techniques and without balancing the dataset. The performance analysis is done based on confusion matrix scores, that is, accuracy, false alarm rate (FAR), sensitivity, specificity, false‐positive rate (FPR), F1 score, area under curve (AUC) analysis and loss functions on well‐known public benchmark KDD Cup'99 data set. The results of our experiments reveal that our models outperform existing techniques, showing their superiority in performance.

Multiple Branch Deep Neural Network and Feature Fusion Model for Underwater Image Enhancement

Underwater imaging faces formidable challenges due to light absorption, scattering, and restricted visibility, demanding sophisticated enhancement methods. This research presents a novel Multiple Branch Deep Neural Network and Feature Fusion (MB-DNN-FF) Model tailored specifically for Underwater Images Enhancement (UWIE), with a focus on its application to the UFO120 dataset. Driven by the limitations of existing techniques, our model harnesses the potential of Deep Learning (DL) and Feature Fusion (FF) to effectively address the intricate complexities present in underwater environments. The innovative architecture incorporates multiple branches, each strategically designed to tackle distinct challenges such as contrast degradation, limited visibility, and color distortion. Central to our model is feature fusion, a critical aspect that harmoniously integrates information from diverse branches, thereby enhancing overall image quality. The training and optimization processes are thoroughly detailed, encompassing unique strategies and loss functions fine-tuned for the nuances of UWIE, with emphasis on the characteristics of the UFO120 dataset. Experimental evaluations leverage the comprehensive UFO120 dataset, employing established performance metrics to quantitatively assess the efficacy of our model. Results exhibit substantial improvements over baseline models and state-of-the-art methods, showcasing the effectiveness and versatility of the proposed multiple branch model, particularly on the UFO120 dataset. The discussion interprets these findings in the context of underwater imaging challenges, highlighting the model's effectiveness and outlining potential applications. This research contributes a valuable asset to the area of UWIE, offering a nuanced and potent solution through the integration of multiple branches and feature fusion, validated on the UFO120 dataset. The conclusions underscore the significance of the MB-DNN-FF in advancing the state-of-the-art in UWIE, opening avenues for future research and practical applications in marine sciences, surveillance, and exploration.

Uncertainty Quantification of Vibroacoustics with Deep Neural Networks and Catmull–Clark Subdivision Surfaces

This study proposes an uncertainty quantification method based on deep neural networks and Catmull–Clark subdivision surfaces for vibroacoustic problems. The deep neural networks are utilized as a surrogate model to efficiently generate samples for stochastic analysis. The training data are obtained from numerical simulation by coupling the isogeometric finite element method and the isogeometric boundary element method. In the simulation, the geometric models are constructed with Catmull–Clark subdivision surfaces, and meantime, the physical fields are discretized with the same spline functions as used in geometric modelling. Multiple deep neural networks are trained to predict the sound pressure response for various parameters with different numbers and dimensions in vibroacoustic problems. Numerical examples are provided to demonstrate the effectiveness of the proposed method.

Utilization of Machine Learning Techniques in Hot‐Film Based Airflow Rate Sensors for Improving Flow Measurement

This article presents a novel airflow rate sensing method based on the principle of a hot‐film flow sensing device with data‐driven machine learning (ML) models. In addition, to combine the two signals of the sensor (the resistance changes of the heaters) and predict the output (airflow rate), three different ML multivariate regression models, i.e., multiple linear regression (MLR), k‐nearest neighbor (KNN), and deep neural network (DNN) models, are trained and compared using 8400 experimentally obtained data. Using sensor fusion techniques, the average mean absolute error (MAE) and mean squared error (MSE) of the KNN model are determined to be 0.01522 and 0.00132, respectively, in the range of 0–5.07 standard liters per minute. Compared with the average results obtained using only a single input, those obtained using a dual input indicate a significant decrease in the MAE and MSE by 85.69% and 96.68%, respectively. Furthermore, a transient analysis of the ML‐based flow sensor is conducted to investigate the response time and transient characteristics of the MLR, KNN, and DNN models. The results of this study contribute to the advancement of airflow management systems for various industrial applications, such as building ventilation, gas leakage detection, and energy systems.

Deep Neural Network Confidence Calibration from Stochastic Weight Averaging

Overconfidence in deep neural networks (DNN) reduces the model’s generalization performance and increases its risk. The deep ensemble method improves model robustness and generalization of the model by combining prediction results from multiple DNNs. However, training multiple DNNs for model averaging is a time-consuming and resource-intensive process. Moreover, combining multiple base learners (also called inducers) is hard to master, and any wrong choice may result in lower prediction accuracy than from a single inducer. We propose an approximation method for deep ensembles that can obtain ensembles of multiple DNNs without any additional costs. Specifically, multiple local optimal parameters generated during the training phase are sampled and saved by using an intelligent strategy. We use cycle learning rates starting at 75% of the training process and save the weights associated with the minimum learning rate in every iteration. Saved sets of the multiple model parameters are used as weights for a new model to perform forward propagation during the testing phase. Experiments on benchmarks of two different modalities, static images and dynamic videos, show that our method not only reduces the calibration error of the model but also improves the accuracy of the model.

Hierarchical Pruning of Deep Ensembles with Focal Diversity

Deep neural network ensembles combine the wisdom of multiple deep neural networks to improve the generalizability and robustness over individual networks. It has gained increasing popularity to study and apply deep ensemble techniques in the deep learning community. Some mission-critical applications utilize a large number of deep neural networks to form deep ensembles to achieve desired accuracy and resilience, which introduces high time and space costs for ensemble execution. However, it still remains a critical challenge whether a small subset of the entire deep ensemble can achieve the same or better generalizability and how to effectively identify these small deep ensembles for improving the space and time efficiency of ensemble execution. This article presents a novel deep ensemble pruning approach, which can efficiently identify smaller deep ensembles and provide higher ensemble accuracy than the entire deep ensemble of a large number of member networks. Our hierarchical ensemble pruning approach (HQ) leverages three novel ensemble pruning techniques. First, we show that the focal ensemble diversity metrics can accurately capture the complementary capacity of the member networks of an ensemble team, which can guide ensemble pruning. Second, we design a focal ensemble diversity based hierarchical pruning approach, which will iteratively find high quality deep ensembles with low cost and high accuracy. Third, we develop a focal diversity consensus method to integrate multiple focal diversity metrics to refine ensemble pruning results, where smaller deep ensembles can be effectively identified to offer high accuracy, high robustness and high ensemble execution efficiency. Evaluated using popular benchmark datasets, we demonstrate that the proposed hierarchical ensemble pruning approach can effectively identify high quality deep ensembles with better classification generalizability while being more time and space efficient in ensemble decision making. We have released the source codes on GitHub at https://github.com/git-disl/HQ-Ensemble .

ECG-based emotion recognition using random convolutional kernel method

The recognition of human emotions based on physiological signals has garnered significant attention from various fields such as human–computer interaction, cognitive-behavioral science, and the treatment of emotion-related diseases. Using the easily accessible ECG signal for emotion recognition presents a viable approach. However, method relying on hand-crafted features necessitate task-specific domain knowledge and consume substantial time to design suitable features, and studies employing deep learning technologies incur greater computational complexity and extended training durations. To address these problems, we proposed an emotion recognition method that employs random convolutional kernels on ECG signals. Following ECG signal preprocessing, a large number of random convolutional kernels were used to transform the signals, and the resulting features are utilized to classify the various emotion states. In addition to accurately and automatically extracting the multi-scale features from ECG signals, combining the random convolutional kernels with ECG signal alone also reduces the computational complexity and training time compared to methods using multiple physiological signals or deep neural networks. Validation on three publicly available datasets revealed that the proposed method achieved recognition accuracies of 91.5%, 96% and 92.5% for valence, arousal and dominance in AMIGOS dataset, and 95.1%, 98.8% and 90.2% for same labels in DREAMER dataset. The accuracy for valence, arousal and affective states in WESAD dataset are 94.5%, 91.8% and 91.8%. The results show that the proposed method outperforms previous researches in multi-class emotion recognition, thus offering a new and promising approach for swift and precise emotion recognition.

Deep Learning and Likelihood Approaches for Viral Phylogeography Converge on the Same Answers Whether the Inference Model Is Right or Wrong.

Analysis of phylogenetic trees has become an essential tool in epidemiology. Likelihood-based methods fit models to phylogenies to draw inferences about the phylodynamics and history of viral transmission. However, these methods are often computationally expensive, which limits the complexity and realism of phylodynamic models and makes them ill-suited for informing policy decisions in real-time during rapidly developing outbreaks. Likelihood-free methods using deep learning are pushing the boundaries of inference beyond these constraints. In this paper, we extend, compare, and contrast a recently developed deep learning method for likelihood-free inference from trees. We trained multiple deep neural networks using phylogenies from simulated outbreaks that spread among 5 locations and found they achieve close to the same levels of accuracy as Bayesian inference under the true simulation model. We compared robustness to model misspecification of a trained neural network to that of a Bayesian method. We found that both models had comparable performance, converging on similar biases. We also implemented a method of uncertainty quantification called conformalized quantile regression that we demonstrate has similar patterns of sensitivity to model misspecification as Bayesian highest posterior density (HPD) and greatly overlap with HPDs, but have lower precision (more conservative). Finally, we trained and tested a neural network against phylogeographic data from a recent study of the SARS-Cov-2 pandemic in Europe and obtained similar estimates of region-specific epidemiological parameters and the location of the common ancestor in Europe. Along with being as accurate and robust as likelihood-based methods, our trained neural networks are on average over 3 orders of magnitude faster after training. Our results support the notion that neural networks can be trained with simulated data to accurately mimic the good and bad statistical properties of the likelihood functions of generative phylogenetic models.

Deep Reinforcement Learning-Based Scheduling for NR-U/WiGig Coexistence in Unlicensed mmWave Bands

This paper investigates the coexistence of the New Radio-based access to Unlicensed spectrum (NR-U) network and the Wireless Gigabit (WiGig) network in unlicensed millimeter-wave (mmWave) bands. To enable the NR-U network to achieve equitable and harmonious spectrum sharing with WiGig systems, we develop two new classes of user equipment (UE) scheduling schemes by exploiting the deep reinforcement learning (DRL) technique. Specifically, we first propose the distributed deep reinforcement learning scheduling (DeepDS) scheme, wherein multiple deep neural networks (DNNs) are used to make decisions for different panels in an independent fashion. Thereafter, to reduce the computational cost of adopting multiple DNNs, we design the centralized deep reinforcement learning scheduling (DeepCS) scheme that introduces the shared DNN framework to perform decisions for all panels in parallel at one time. The objective of both DeepDS and DeepCS is to maximize the total data rate of the NR-U network with as little interference to WiGig systems as possible, while satisfying the quality of service (QoS) requirement for each UE. We first formulate this problem into the constrained Markov decision process framework. To address the multi-constraint issue, we put forth a new DRL algorithm that incorporates the Lagrangian primal-dual optimization into the deep Q-network framework, referred to as adaptive multi-constraint deep Q-network (AMC-DQN). With AMC-DQN, both DeepDS and DeepCS can achieve their goals even without acquiring prior operations about the WiGig network. Simulation results show that compared with the state-of-the-art omniLBT and dirLBT, both DeepDS and DeepCS yield significant performance benefits in terms of the total network data rate. We also demonstrate the ability of DeepDS and DeepCS to satisfy the QoS requirements of different UEs and their robustness against various simulation setups. Furthermore, compared with DeepDS, DeepCS can save a large amount of computational cost although at the expense of a slightly lower data rate.

Construction Activity Analysis of Workers Based on Human Posture Estimation Information

Identifying workers’ construction activities or behaviors can enable managers to better monitor labor efficiency and construction progress. However, current activity analysis methods for construction workers rely solely on manual observations and recordings, which consumes considerable time and has high labor costs. Researchers have focused on monitoring on-site construction activities of workers. However, when multiple workers are working together, current research cannot accurately and automatically identify the construction activity. This research proposes a deep learning framework for the automated analysis of the construction activities of multiple workers. In this framework, multiple deep neural network models are designed and used to complete worker key point extraction, worker tracking, and worker construction activity analysis. The designed framework was tested at an actual construction site, and activity recognition for multiple workers was performed, indicating the feasibility of the framework for the automated monitoring of work efficiency.

LaF: Labeling-free Model Selection for Automated Deep Neural Network Reusing

Applying deep learning (DL) to science is a new trend in recent years, which leads DL engineering to become an important problem. Although training data preparation, model architecture design, and model training are the normal processes to build DL models, all of them are complex and costly. Therefore, reusing the open-sourced pre-trained model is a practical way to bypass this hurdle for developers. Given a specific task, developers can collect massive pre-trained deep neural networks from public sources for reusing. However, testing the performance (e.g., accuracy and robustness) of multiple deep neural networks (DNNs) and recommending which model should be used is challenging regarding the scarcity of labeled data and the demand for domain expertise. In this article, we propose a labeling-free (LaF) model selection approach to overcome the limitations of labeling efforts for automated model reusing. The main idea is to statistically learn a Bayesian model to infer the models’ specialty only based on predicted labels. We evaluate LaF using nine benchmark datasets, including image, text, and source code, and 165 DNNs, considering both the accuracy and robustness of models. The experimental results demonstrate that LaF outperforms the baseline methods by up to 0.74 and 0.53 on Spearman’s correlation and Kendall’s τ, respectively.

Development of prediction model for body weight and energy balance indicators from milk traits in lactating dairy cows based on deep neural networks

To develop a body weight (BW) prediction model using milk production traits and present a useful indicator for energy balance (EB) evaluation in dairy cows. Data were collected from 30 Holstein cows using an automatic milking system. BW prediction models were developed using multiple linear regression (MLR), local regression (LOESS), and deep neural networks (DNN). Milk production traits readily available on commercial dairy farms, such as energy-corrected milk (ECM), fat-to-protein ratio, days in milk (DIM), and parity, were used as input variables for BW prediction. The EB was evaluated as the difference between energy intake and energy demand. The DNN model showed the greatest predictive accuracy for BW compared with the LOESS and MLR models. The BW predicted using the DNN model was used to calculate the energy demand. Our results revealed that the day on which the EB status transitioned from negative to positive differed among cows. The cows were assigned to one of the three EB index groups. EB index 1 indicated that the day of EB transition was within DIM ≤ 70. The EB indexes 2 and 3 were 70 < DIM ≤ 140 and 140 < DIM ≤ 305, respectively. EB index 3 had the lowest EB, which is the slowest to transition from a negative to a positive energy balance compared with EB indexes 1 and 2. The highest ECM and feed efficiency were observed for EB index 3. The calving interval was the shortest for EB index 1. EB of individual cows during lactation can be estimated and monitored with moderately high accuracy using EB indexes.