Multiple Neural Network Models Research Articles

A Probabilistic Descent Ensemble for Malware Prediction Using Deep Learning

INTRODUCTION: Introducing a Probabilistic Descent Ensemble (PDE) approach for enhancing malware prediction through deep learning leverages the power of multiple neural network models with distinct architectures and training strategies to achieve superior accuracy while minimizing false positives. OBJECTIVES: Combining Stochastic Gradient Descent (SGD) with early stopping is a potent approach to optimising deep learning model training. Early stopping, a vital component, monitors a validation metric and halts training if it stops improving or degrades, guarding against overfitting. METHODS: This synergy between SGD and early stopping creates a dynamic framework for achieving optimal model performance adaptable to diverse tasks and datasets, with potential benefits including reduced training time and enhanced generalization capabilities. RESULTS: The proposed work involves training a Gaussian NB classifier with SGD as the optimization algorithm. Gaussian NB is a probabilistic classifier that assumes the features follow a Gaussian (normal) distribution. SGD is an optimization algorithm that iteratively updates model parameters to minimize a loss function. CONCLUSION: The proposed work gives an accuracy of 99% in malware prediction and is free from overfitting and local minima.

Multi-Source Information Graph Embedding with Ensemble Learning for Link Prediction

Link prediction is a key technique for connecting entities and relationships in a graph reasoning field. It leverages known information about the graph structure data to predict missing factual information. Previous studies have either focused on the semantic representation of a single triplet or on the graph structure data built on triples. The former ignores the association between different triples, and the latter ignores the true meaning of the node itself. Furthermore, common graph-structured datasets inherently face challenges, such as missing information and incompleteness. In light of this challenge, we present a novel model called Multi-source Information Graph Embedding with Ensemble Learning for Link Prediction (EMGE), which can effectively improve the reasoning of link prediction. Ensemble learning is systematically applied throughout the model training process. At the data level, this approach enhances entity embeddings by integrating structured graph information and unstructured textual data as multi-source information inputs. The fusion of these inputs is effectively addressed by introducing an attention mechanism. During the training phase, the principle of ensemble learning is employed to extract semantic features from multiple neural network models, facilitating the interaction of enriched information. To ensure effective model learning, a novel loss function based on contrastive learning is devised, effectively minimizing the discrepancy between predicted values and the ground truth. Moreover, to enhance the semantic representation of graph nodes in link prediction, two rules are introduced during the aggregation of graph structure information. These rules incorporate the concept of spreading activation, enabling a more comprehensive understanding of the relationships between nodes and edges in the graph. During the testing phase, the EMGE model is validated on three datasets, including WN18RR, FB15k-237, and a private Chinese financial dataset. The experimental results demonstrate a reduction in the mean rank (MR) by 0.2 times, an improvement in the mean reciprocal rank (MRR) by 5.9%, and an increase in the Hit@1 by 12.9% compared to the baseline model.

Stacked neural network for predicting polygenic risk score

In recent years, the utility of polygenic risk scores (PRS) in forecasting disease susceptibility from genome-wide association studies (GWAS) results has been widely recognised. Yet, these models face limitations due to overfitting and the potential overestimation of effect sizes in correlated variants. To surmount these obstacles, we devised the Stacked Neural Network Polygenic Risk Score (SNPRS). This novel approach synthesises outputs from multiple neural network models, each calibrated using genetic variants chosen based on diverse p-value thresholds. By doing so, SNPRS captures a broader array of genetic variants, enabling a more nuanced interpretation of the combined effects of these variants. We assessed the efficacy of SNPRS using the UK Biobank data, focusing on the genetic risks associated with breast and prostate cancers, as well as quantitative traits like height and BMI. We also extended our analysis to the Korea Genome and Epidemiology Study (KoGES) dataset. Impressively, our results indicate that SNPRS surpasses traditional PRS models and an isolated deep neural network in terms of accuracy, highlighting its promise in refining the efficacy and relevance of PRS in genetic studies.

Deep-Learning-Based Real-Time Visual Pollution Detection in Urban and Textile Environments

The environmental physiognomy of an area can significantly diminish its aesthetic appeal, rendering it susceptible to visual pollution, the unbeaten scourge of modern urbanization. In this study, we propose using a deep learning network and a robotic vision system integrated with Google Street View to identify streets and textile-based visual pollution in Dhaka, the megacity of Bangladesh. The issue of visual pollution extends to the global apparel and textile industry, as well as to various common urban elements such as billboards, bricks, construction materials, street litter, communication towers, and entangled electric wires. Our data collection encompasses a wide array of visual pollution elements, including images of towers, cables, construction materials, street litter, cloth dumps, dyeing materials, and bricks. We employ two open-source tools to prepare and label our dataset: LabelImg and Roboflow. We develop multiple neural network models to swiftly and accurately identify and classify visual pollutants in this work, including Faster SegFormer, YOLOv5, YOLOv7, and EfficientDet. The tuna swarm optimization technique has been used to select the applied models’ final layers and corresponding hyperparameters. In terms of hardware, our proposed system comprises a Xiaomi-CMSXJ22A web camera, a 3.5-inch touchscreen display, and a Raspberry Pi 4B microcontroller. Subsequently, we program the microcontroller with the YOLOv5 model. Rigorous testing and trials are conducted on these deep learning models to evaluate their performance against various metrics, including accuracy, recall, regularization and classification losses, mAP, precision, and more. The proposed system for detecting and categorizing visual pollution within the textile industry and urban environments has achieved notable results. Notably, the YOLOv5 and YOLOv7 models achieved 98% and 92% detection accuracies, respectively. Finally, the YOLOv5 technique has been deployed into the Raspberry Pi edge device for instantaneous visual pollution detection. The proposed visual pollutants detection device can be easily mounted on various platforms (like vehicles or drones) and deployed in different urban environments for on-site, real-time monitoring. This mobility is crucial for comprehensive street-level data collection, potentially engaging local communities, schools, and universities in understanding and participating in environmental monitoring efforts. The comprehensive dataset on visual pollution will be published in the journal following the acceptance of our manuscript.

VeriTrain: Validating MLaaS Training Efforts via Anomaly Detection

Machine learning as a service (MLaaS) offers users the benefit of training state-of-the-art neural network models on fast hardware with low costs. However, it also brings security concerns since the user does not fully trust the cloud. To prove to the user that the ML training results are legitimate, existing approaches mainly adopt cryptographic techniques such as secure multi-party computation, which incur large overheads. In this paper, we model the problem of verifying ML training efforts as an anomaly detection problem. We design a verification system, dubbed <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">VeriTrain</small> , which combines unsupervised anomaly detection approaches and hypothesis testing techniques to verify the legitimacy of training efforts on the MLaaS cloud. <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">VeriTrain</small> is run inside trusted execution environments (TEEs) on the same cloud machine to ensure the integrity of its execution. We consider a threat model where the cloud model trainer is a lazy attacker and tries to fool <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">VeriTrain</small> with minimum training effort. We perform extensive evaluations on multiple neural network models and datasets, which shows that <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">VeriTrain</small> performs well in detecting parameter updates crafted by the attacker. We also implement <sc xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">VeriTrain</small> with Intel SGX and show that it only incurs moderate overheads.

Performance Comparison of Prediction of Hydraulic Jump Length Under Multiple Neural Network Models

Performance Comparison of Prediction of Hydraulic Jump Length Under Multiple Neural Network Models

A novel interval-valued carbon price analysis and forecasting system based on multi-objective ensemble strategy for carbon trading market

Accurate forecasting of carbon price is crucial for the efficient management and stable operation of carbon markets. Earlier studies are limited to point and interval forecasts based on single-valued carbon price and lack analysis and forecasting based on interval-valued carbon price. Therefore, this study proposes a novel analysis and forecasting system from a new perspective of interval-valued carbon price. Specifically, a carbon price analysis sub-system is developed to investigate the directional causal relationship between the upper and lower bounds of the interval-valued carbon price series. The carbon price forecasting sub-system is developed by designing a data preprocessing module, sub-model forecasting module, and multi-objective ensemble module. The data preprocessing module adopts the decomposition algorithm to preprocess the interval-valued carbon price. Then the sub-model forecasting module utilizes multiple neural network models to predict the highest and lowest prices. Finally, the multi-objective ensemble module adopts a non-linear and multi-objective ensemble strategy to ensemble the forecasting results of the sub-models. It can be found that the consideration of both upper and lower bounds of interval-valued carbon price within the range leads to higher prediction accuracy for the highest or lowest price predictions. Additionally, the ensemble model can effectively leverage the strengths of individual sub-models, resulting in more precise and stable predictions. The average absolute percentage errors for the highest and lowest price predictions in the Hubei and Guangzhou carbon trading markets are 0.8574%, 1.2738%, 0.9774%, and 1.8217% respectively, vividly demonstrating the effectiveness of the proposed system in carbon price prediction.

Ensemble Neural Networks for the Development of Storm Surge Flood Modeling: A Comprehensive Review

This review paper focuses on the use of ensemble neural networks (ENN) in the development of storm surge flood models. Storm surges are a major concern in coastal regions, and accurate flood modeling is essential for effective disaster management. Neural network (NN) ensembles have shown great potential in improving the accuracy and reliability of such models. This paper presents an overview of the latest research on the application of NNs in storm surge flood modeling and covers the principles and concepts of ENNs, various ensemble architectures, the main challenges associated with NN ensemble algorithms, and their potential benefits in improving flood forecasting accuracy. The main part of this paper pertains to the techniques used to combine a mixed set of predictions from multiple NN models. The combination of these models can lead to improved accuracy, robustness, and generalization performance compared to using a single model. However, generating neural network ensembles also requires careful consideration of the trade-offs between model diversity, model complexity, and computational resources. The ensemble must balance these factors to achieve the best performance. The insights presented in this review paper are particularly relevant for researchers and practitioners working in coastal regions where accurate storm surge flood modeling is critical.

A Comparative Study of Deep Learning Approaches for Amyotrophic Lateral Sclerosis Super-Resolution Image Classification

Amyotrophic lateral sclerosis is a fatal degenerative neurological disease known as motor neuron disease. According to research, amyotrophic lateral sclerosis affects nerve cells related to movement in the brain and spinal cord, leading to the death of motor neurons, making the brain unable to control muscle movement, and resulting in a significant loss of motor nerve cells, leading to muscle atrophy. In the early stages of ALS, symptoms are mild, often go unnoticed, and can easily be confused with other diseases. Patients may only feel symptoms such as weakness, convulsions, and fatigue, gradually develop muscle atrophy all over the body, difficulty swallowing, and finally die of respiratory failure. TDP-43 is a DNA/RNA binding protein typically located in the nucleus regulating various steps of RNA metabolism. Meanwhile, TDP-43 is the most prominent pathological protein in the characteristics of ALS patients.[1] TDP-43 is depleted in the nucleus in nearly all ALS cases but accumulates in the cytoplasm.Deep learning is learning the internal laws and representation levels of sample data. The information obtained from these learning processes can significantly help interpret data such as text, images, and sounds. Its goal is to enable machines to analyze and learn like humans and recognize data. Additionally, super-resolution microscopy can be used to observe the conformation of proteins in biological samples, thereby gaining insight into the structure and assembly mechanism of TDP-43 aggregates.This project aims to use super-resolution images for machine learning to build a model for classifying ALS patients from non-ALS patients. At the same time, multiple neural network models are used for training and then compared to select the model with the highest accuracy.

Knowledge Base Question Answering via Semantic Analysis

Knowledge Question Answering is one of the important research directions in the field of robot intelligence. It is mainly based on background knowledge to analyze users’ questions and generate answers. It is one of the important application methods of knowledge graph technology. Compared with the traditional expert system of question and answer, it has the advantage of a large-scale background knowledge base and the traceability and interpretability of the question-answering process. Compared with the current ChatGPT (Chat Generative Pre-trained Transformer) technology, it has advantages in the proprietary segmentation field. Aiming at the problem of the accuracy of existing knowledge question-answering methods being low, this paper studies the method of semantic analysis for knowledge question-answering under the support of a knowledge database, proposes a knowledge question-answering method based on the superposition of multiple neural network models, and conducts experimental verification on the publicly available NLPCC2016KBQA(Knowledge Q&amp;A Tasks in the 2016 Natural Language Processing and Chinese Computing Conference) data set. The experimental results show that the F1 value of this method is higher than that of the baseline model.

TestSGD : Interpretable Testing of Neural Networks against Subtle Group Discrimination

Discrimination has been shown in many machine learning applications, which calls for sufficient fairness testing before their deployment in ethic-relevant domains. One widely concerning type of discrimination, testing against group discrimination, mostly hidden , is much less studied, compared with identifying individual discrimination . In this work, we propose TestSGD , an interpretable testing approach that systematically identifies and measures hidden (which we call “subtle”) group discrimination of a neural network characterized by conditions over combinations of the sensitive attributes . Specifically, given a neural network, TestSGD first automatically generates an interpretable rule set that categorizes the input space into two groups. Alongside, TestSGD also provides an estimated group discrimination score based on sampling the input space to measure the degree of the identified subtle group discrimination, which is guaranteed to be accurate up to an error bound. We evaluate TestSGD on multiple neural network models trained on popular datasets including both structured data and text data. The experiment results show that TestSGD is effective and efficient in identifying and measuring such subtle group discrimination that has never been revealed before. Furthermore, we show that the testing results of TestSGD can be used to mitigate such discrimination through retraining with negligible accuracy drop.

Decoding urban green spaces: Deep learning and google street view measure greening structures

Street greenery is an essential component in assessing the ecological quality of urban areas. Panoramic images of streets provide a more comprehensive representation of the streetscape, enabling researchers to explore the diverse composition of street greenery and its correlation with the physical and mental health of urban residents. In this study, we propose a method based on semantic segmentation of panoramic street images to evaluate the structure of urban street greenery and calculate the proportion of its different broad structural categories of plants. To accomplish this, we constructed a new street view dataset called Street Greening Space Structure Dataset (S-G-S-S), which is specifically designed for street greening structure evaluation. Using the DeepLabV3 + neural network model, we trained the system for semantic segmentation of panoramic streetscape images to improve the accuracy of Panorama View Green View index (PVGVI) while accurately identifying the street greenery structure. To verify the accuracy and stability of the method, we conducted an empirical study in the Livingston area of western New York City, USA, using multiple neural network models and comparing the results. The findings indicate that the proposed method can better visualize the greening structure at the urban street level, offering a valuable tool for urban planners and policymakers to assess the ecological quality of urban areas.

Dynamical analysis and vibration estimation of a flexible plate with enhanced active constrained layer damping treatment by combinatorial neural networks of surrogates

The effective estimation of the vibration of spacecrafts are frontier issues in the aerospace industry. But the vibration of aerospace components such as space solar panels and flexible manipulators under large overall rotating motion is usually very complex due to strong nonlinearity and often requires a lot of repetitive and burdensome calculations. Artificial neural network (ANN) is composed of interconnected neurons, and a surrogate model can be established to effectively predict the mechanical characteristics. In addition, the computational efficiency will be greatly improved if the object is changed from the model that is established by commercial finite element method (FEM) software or numerical calculations to the surrogates. However, the stochasticity of the estimation by single neural network is inevitable, and some neural networks much rely on the initial weight and threshold. This paper will implement a new combinatorial strategy that relies on multiple neural network models, which have different activation functions and regression feedback processes based on reasonable selection of sample points. To solve the elongated training time led by comparing these neural networks, genetic algorithm (GA) is used to optimize the iteration speed of one neural network when finding the optimal output value. Thus, better estimation accuracy will be achieved by designing an algorithm to combine these combinatorial neural networks of surrogates (CNNS). Based on the proposed method, the vibration estimation of a rotating flexible plate with enhanced active constrained layer damping (EACLD) treatment, and the vibration suppression capacity of the composite structure under various operating conditions will be investigated.

Novel CNN Accelerator Design With Dual Benes Network Architecture

We presented a novel hardware architecture that uses dual Benes networks to accelerate Convolutional Neural Network (CNN) algorithms. This architecture can reduce the need for high-speed buses and maintain a high-speed connection between execution units and memories. Also, this design can work with multiple neural network models by changing the Benes network configuration only due to the reorderability of the non-blocking networks. The proposed architecture can save the time of re-implementing the hardware design. Thus, this architecture can act as a general CNN accelerator. We successfully implemented our CNN accelerator on the Xilinx Virtex UltraScale+ VU19P and evaluated it by running the CNN model LeNet-5.

An Augmented Model of Rutting Data Based on Radial Basis Neural Network

The rutting depth is an important index to evaluate the damage degree of the pavement. Therefore, establishing an accurate rutting depth prediction model can guide pavement design and provide the necessary basis for pavement maintenance. However, the sample size of pavement rutting depth data is small, and the sampling is not standardized, which makes it hard to establish a prediction model with high accuracy. Based on the data of RIOHTrack’s asphalt pavement structure, this study builds a reliable data-augmented model. In this paper, different asphalt rutting data augmented models based on Gaussian radial basis neural networks are constructed with the temperature and loading of asphalt pavements as the main features. Experimental results show that the method outperforms classical machine learning methods in data augmentation, with an average root mean square error of 3.95 and an average R-square of 0.957. Finally, the augmented data of rutting depth is constructed for training, and multiple neural network models are used for prediction. Compared with unaugmented data, the prediction accuracy is increased by 50%.

A multiple neural network and reinforcement learning-based strategy for process control

In this study, we propose a novel process control strategy by assimilating multiple neural network (MNN) and reinforcement learning (RL) to generate an MNNRL controller. The artificial neural networks (ANNs) constituting the MNN are trained using repositories of optimal control and state data to predict control actions to attain or stay at a set-point. This ability is significantly improved by developing an RL component for the MNNRL controller to provide enhanced set-point tracking, robustness against disturbances, and real-time learning to successfully adapt in the presence of persistent parameter upsets. The controller performance is examined in three different case studies utilizing the simulation of continuous processes with varying degrees of nonlinearity. Relative to both MNN and nonlinear model predictive control (NMPC), the MNNRL controller is found to provide better performance with about 32% less lower integral absolute error (IAE), and 50% reduced settling time with suppressed over-/under-shoots in controlled variables. Further, it is observed that the controller is fairly robust against disturbances, and can adapt via real-time learning to altered process dynamics caused by any persistent process upsets. The controller computation time is observed to be about an order of magnitude less in comparison to NMPC.

Privacy-preserving deep learning for electricity consumer characteristics identification

Deep learning models trained from smart meter data have proven to be effective in predicting socio-demographic characteristics of electricity consumers, which can help retailers provide personalized service to electricity customers. Traditionally, deep learning models are trained in a centralized manner to gather large amounts of data to ensure effectiveness and efficiency. However, gathering smart meter data in plaintext may raise privacy concerns since the data is privately owned by different retailers. This indicates an imminent need for privacy-preserving deep learning. This paper proposes several secure multi-party computation (MPC) protocols that enable deep learning training and inference for electricity consumer characteristics identification while keeping the retailer’s raw data confidential. In our protocols, the retailers secret-share their raw data to three computational servers, which implement deep learning training and inference through lightweight replicated secret sharing techniques. We implement and benchmark multiple neural network models and optimization strategies. Comprehensive experiments are conducted on the Irish Commission for Energy Regulation (CER) dataset to verify that our MPC-based protocols have comparable performance.

Research on the Risk Classification of Cruise Ship Fires Based on an Attention-Bp Neural Network

Abstract Due to the relatively closed environment, complex internal structure, and difficult evacuation of personnel, it is more difficult to prevent ship fires than land fires. In this paper, taking the large cruise ship as the research object, the physical model of a cruise cabin fire is established through PyroSim software, and the safety indexes such as smoke temperature, CO concentration, and visibility are numerically simulated. An Attention-BP neural network model is designed for realizing the intelligent identification of a cabin fire and dividing the risk level, which integrates the diagnosis results of multiple neural network models through the self-Attention mechanism and adaptively distributes the weight of each BP neural network model. The proposed model can provide decision-making reference for subsequent fire-fighting measures and personnel evacuation. Experimental results show that the proposed Attention-BP neural network model can effectively realize the early warning of the fire risk level. Compared with other machine learning algorithms, it has the highest stability and accuracy and reduces the uncertainty of early cabin fire warning.

A Small Sample Recognition Model for Poisonous and Edible Mushrooms based on Graph Convolutional Neural Network.

The automatic identification of disease types of edible mushroom crops and poisonous crops is of great significance for improving crop yield and quality. Based on the graph convolutional neural network theory, this paper constructs a graph convolutional network model for the identification of poisonous crops and edible fungi. By constructing 6 graph convolutional networks with different depths, the model uses the training mechanism of graph convolutional networks to analyze the results of disease identification and completes the automatic extraction of the disease characteristics of the poisonous crops by overfitting problem. During the simulation, firstly, the relevant PlantVillage dataset is used to obtain the pretrained model, and the parameters are adjusted to fit the dataset. The network framework is trained and parameterized with prior knowledge learned from large datasets and finally synthesized by training multiple neural network models and using direct averaging and weighting to synthesize their predictions. The experimental results show that the graph convolutional neural network model that integrates multi-scale category relationships and dense links can use dense connection technology to improve the representation ability and generalization ability of the model, and the accuracy rate generally increases by 1%–10%. The average recognition rate is about 91%, which greatly promotes the ability to identify the diseases of poisonous crops.

Hybrid Model Compression for Multi-Task Network

In recent years, deep neural networks have made significant achievements in various fields in science and engineering. To achieve outstanding performance, neural network models with very deep structures and an extensive number of parameters are usually used, leading to huge memory storage and computational resources requirements of the hardware systems, and thus preventing them from being deployed in devices with hardware limits. In particular, such problem will be further amplified when multiple neural network models for several tasks are to be integrated into one single platform at the same time. In this paper, a hybrid joint-network optimization model is proposed to solve this multi-task multi-model compression problem. The idea of hard parameter sharing in multi-task learning is adopted. Specifically, we turn the sub-network of the same structure of multiple models into a hybrid network model, in which the same parameters are shared for all the tasks. In addition, a unique sparse matrix for each task is added to each task independently. Experimental results show that the proposed compression methods can not only achieve a high compression ratio but also get performance improvement for some tasks.