Multiple Neural Networks Research Articles

Prediction Capability of Geomagnetic Events from Solar Wind Data Using Neural Networks

Multiple neural network architectures, with different structural composition and complexity, are implemented in this study with the aim of providing multi-hour-ahead warnings of severe geomagnetic disturbances, based on in situ measurements of the solar wind plasma and magnetic field acquired at the Lagrangian point L1. First, a statistical analysis of the interplanetary data was performed to point out which are the most relevant parameters to be provided as input to the neural networks, and a preprocessing of the data set was implemented to face its heavy imbalance (the Earth’s magnetosphere is in fact mostly at rest). Then, neural networks were tested to evaluate their performance. It turned out that, in a binary classification problem, recurrent approaches are best at predicting critical events both 1 and 8 hr in advance, achieving a balanced accuracy of 94% and 70%, respectively. Finally, in an attempt at multistep prediction of the criticality of future geomagnetic events from 1–8 hr ahead, more complex neural networks, built by merging the different types of basic convolutional and recurrent architectures, have been shown to outperform single-step and state-of-the-art approaches with a balanced accuracy of at least 70%. Interestingly, the accuracy peaks at 4 hr, corresponding to the waiting time between the detection of solar drivers at L1 and the onset of the geomagnetic storm (as previously obtained by statistical investigations), suggesting that on average this is the time the Earth’s magnetosphere takes to react to the solar event.

Improved Performance for PMSM Sensorless Control Based on Robust-Type Controller, ESO-Type Observer, Multiple Neural Networks, and RL-TD3 Agent.

This paper summarizes a robust controller based on the fact that, in the operation of a permanent magnet synchronous motor (PMSM), a number of disturbance factors naturally occur, among which both changes in internal parameters (e.g., stator resistance Rs and combined inertia of rotor and load J) and changes in load torque TL can be mentioned. In this way, the performance of the control system can be maintained over a relatively wide range of variation in the types of parameters mentioned above. It also presents the synthesis of robust control, the implementation in MATLAB/Simulink, and an improved version using a reinforcement learning twin-delayed deep deterministic policy gradient (RL-TD3) agent, working in tandem with the robust controller to achieve superior performance of the PMSM sensored control system. The comparison of the proposed control systems, in the case of sensored control versus the classical field oriented control (FOC) structure, based on classical PI-type controllers, is made both in terms of the usual response time and error speed ripple, but also in terms of the fractal dimension (DF) of the rotor speed signal, by verifying the hypothesis that the use of a more efficient control system results in a higher DF of the controlled variable. Starting from a basic structure of an ESO-type observer which, by its structure, allows the estimation of both the PMSM rotor speed and a term incorporating the disturbances on the system (from which, in this case, an estimate of the PMSM load torque can be extracted), four variants of observers are proposed, obtained by combining the use of a multiple neural network (NN) load torque observer and an RL-TD3 agent. The numerical simulations performed in MATLAB/Simulink validate the superior performance obtained by using properly trained RL-TD3 agents, both in the case of sensored and sensorless control.

A two-stage interval-valued carbon price forecasting model based on bivariate empirical mode decomposition and error correction.

Economic development has brought about global greenhouse gas emissions and, thus, global climate change, a common challenge worldwide and urgently needs to be addressed. Accurate carbon price forecasting plays a pivotal role in providing a reasonable basis for carbon pricing and ensuring the healthy development of carbon markets. Therefore, this paper proposes a two-stage interval-valued carbon price combination forecasting model based on bivariate empirical mode decomposition (BEMD) and error correction. In Stage I, the raw carbon price and multiple influencing factors are decomposed into several interval sub-modes by BEMD. Then, we select artificial intelligence-based multiple neural network methods such as IMLP, LSTM, GRU, and CNN to conduct combination forecasting for interval sub-modes. In Stage II, the error generated in Stage I is calculated, and LSTM is used to predict the error; then, the error forecasting result is added to the first stage result to obtain the error-corrected forecasting result. Taking the carbon trading prices of Hubei, Guangdong, and the national carbon market, China, as the research object, the empirical analysis proves that the combination forecasting of interval sub-modes of Stage I outperforms the single forecasting method. In addition, the error correction technique in Stage II can further improve the forecasting accuracy and stability, which is an effective model for interval-valued carbon price forecasting. This study can help policymakers formulate regulatory policies to reduce carbon emissions and help investors avoid risks.

Deep self-organizing cube: A novel multi-dimensional classifier for multiple output learning

Multi-dimensional classification (MDC) task can be considered the most inclusive description of all classifications tasks, as it joins multiple class spaces and their multiple class members into a single compound classification problem. The challenges in MDC arise from the possible class dependencies across different class spaces, as well as the imbalance of labels in training datasets due to lack of all possible combinations. In this paper, we propose a straightforward, yet efficient, MDC deep learning classifier, named Deep Self-Organizing Cube (DSOC) that can model dependencies among classes in multiple class spaces, while consolidating its ability to classify rare combinations of labels. DSOC is formed of two n-dimensional components, namely the Hypercube Classifier and the multiple DSOC Neural Networks connected to the hypercube. The multiple neural networks component is responsible for feature selection and segregation of classes, while the Hypercube classifier is responsible for creating the semantics among multiple class spaces and accommodate the model for rare sample classification. DSOC is a multiple-output learning algorithm that successfully classify samples across all class spaces simultaneously. To challenge the proposed DSOC model, we conducted an assessment on seventeen benchmark datasets in the four types of classification tasks, binary, multi-class, multi-label and multi-dimensional. The obtained results were compared to four standard classifiers and eight competitive state-of-the-art approaches reported in literature. The DSOC has achieved superior performance over standard classifiers as well as the state-of-the-art approaches in all the four classification tasks. Moreover, in terms of Exact Match accuracy metrics, DSOC has outperformed all state-of-the-art approaches in 77.8% of the cases, which reflects the superior ability of DSOC to model dependencies and successfully classify rare samples across all dimensions simultaneously.

Ensemble Learning Methods for Dirty Data: A Keynote at CIKM 2022

Neural network ensemble is a collaborative learning paradigm that utilizes multiple neural networks to solve a complex learning problem. Constructing predictive models with high generalization performance is an important and yet most challenging goal for robust AI systems in the presence of dirty data. Given a target learning task, popular approaches have been dedicated to designing and finding the top performing model. However, it is difficult in general to estimate the best model when available data is finite, possibly dirty, or insufficient for the problem. The problem of dirty data in machine learning (ML) can be characterized by the out of distribution data and the digital or physical deception of data. Such dirty data may cause unintended or harmful behavior for well trained ML models. In this paper, a curated version of my keynote at ACM CIKM 2022, I will first give a brief overview of ensemble learning methodology. Then I will review different types of dirty data that could deceive well-trained ML models. Finally, I will describe a focal diversity optimized ensemble learning framework, developed at Georgia Tech, for measuring, enforcing, and combining multiple neural networks, delivering high generalization performance of ensemble learner, while maximizing ensemble utility and resilience to dirty data. Date: 17 October 2022.

ChampKit: A framework for rapid evaluation of deep neural networks for patch-based histopathology classification

Background and ObjectiveHistopathology is the gold standard for diagnosis of many cancers. Recent advances in computer vision, specifically deep learning, have facilitated the analysis of histopathology images for many tasks, including the detection of immune cells and microsatellite instability. However, it remains difficult to identify optimal models and training configurations for different histopathology classification tasks due to the abundance of available architectures and the lack of systematic evaluations. Our objective in this work is to present a software tool that addresses this need and enables robust, systematic evaluation of neural network models for patch classification in histology in a light-weight, easy-to-use package for both algorithm developers and biomedical researchers. MethodsHere we present ChampKit (Comprehensive Histopathology Assessment of Model Predictions toolKit): an extensible, fully reproducible evaluation toolkit that is a one-stop-shop to train and evaluate deep neural networks for patch classification. ChampKit curates a broad range of public datasets. It enables training and evaluation of models supported by timm directly from the command line, without the need for users to write any code. External models are enabled through a straightforward API and minimal coding. As a result, Champkit facilitates the evaluation of existing and new models and deep learning architectures on pathology datasets, making it more accessible to the broader scientific community. To demonstrate the utility of ChampKit, we establish baseline performance for a subset of possible models that could be employed with ChampKit, focusing on several popular deep learning models, namely ResNet18, ResNet50, and R26-ViT, a hybrid vision transformer. In addition, we compare each model trained either from random weight initialization or with transfer learning from ImageNet pretrained models. For ResNet18, we also consider transfer learning from a self-supervised pretrained model. ResultsThe main result of this paper is the ChampKit software. Using ChampKit, we were able to systemically evaluate multiple neural networks across six datasets. We observed mixed results when evaluating the benefits of pretraining versus random intialization, with no clear benefit except in the low data regime, where transfer learning was found to be beneficial. Surprisingly, we found that transfer learning from self-supervised weights rarely improved performance, which is counter to other areas of computer vision. ConclusionsChoosing the right model for a given digital pathology dataset is nontrivial. ChampKit provides a valuable tool to fill this gap by enabling the evaluation of hundreds of existing (or user-defined) deep learning models across a variety of pathology tasks. Source code and data for the tool are freely accessible at https://github.com/SBU-BMI/champkit.

Bipartite quasi-synchronization of multiple neural networks with generalized cooperative-competitive topology

Bipartite quasi-synchronization of multiple neural networks with generalized cooperative-competitive topology

Deep learning modified neural networks with chicken swarm optimization-based lungs disease detection and severity classification

Many people around the world are affected by pulmonary disease as well as asthma, pneumonia, lung cancer, and tuberculosis. All of these conditions have one thing in common: airway obstruction. Lung illnesses are a worldwide issue in which chronic obstructive pulmonary disease, pneumonia, asthma, tuberculosis, and fibrosis are few of the most common. An accurate diagnosis of a lung problem is essential, and it has been pursued by many researchers using image processing and machine learning models. Multiple deep learning methods are used to predict lung disease; these include convolutional neural networks, vanilla neural networks, visual geometry group-based neural networks, and the capsule network. Medical data are scarce, so diagnosing pulmonary diseases using chest x-ray pictures from datasets with less than 1000 samples is considered to address the problem. Three deep learning multiple neural networks (DLMNNs) were generated using the chicken swarm optimization (CSO) approach, for which transfer learning was applied to evaluate the performance of each. First, we created an algorithm for segmenting CXR images, and then we compared it with other classification systems. Our results were compared with those of other methods using publicly available data from the Shenzhen and Montgomery lung datasets. However, our technique has a lower number of trainable parameters compared with the best-performing models trained on the Montgomery dataset. The DLMNN-CSO virtually matched the best performance on the Shenzhen dataset, although it was computationally less expensive than the other models. DLMNN-CSO’s validation loss was 0.4, whereas the validation loss for CNN, VDSNet, VGG, and DBN was 0.9, 0.8, 0.6, and 0.5, respectively.

An Ensembled Framework for Human Breast Cancer Survivability Prediction Using Deep Learning.

Breast cancer is categorized as an aggressive disease, and it is one of the leading causes of death. Accurate survival predictions for both long-term and short-term survivors, when delivered on time, can help physicians make effective treatment decisions for their patients. Therefore, there is a dire need to design an efficient and rapid computational model for breast cancer prognosis. In this study, we propose an ensemble model for breast cancer survivability prediction (EBCSP) that utilizes multi-modal data and stacks the output of multiple neural networks. Specifically, we design a convolutional neural network (CNN) for clinical modalities, a deep neural network (DNN) for copy number variations (CNV), and a long short-term memory (LSTM) architecture for gene expression modalities to effectively handle multi-dimensional data. The independent models' results are then used for binary classification (long term > 5 years and short term < 5 years) based on survivability using the random forest method. The EBCSP model's successful application outperforms models that utilize a single data modality for prediction and existing benchmarks.

Automatic semantic segmentation of the lumbar spine: Clinical applicability in a multi-parametric and multi-center study on magnetic resonance images

Significant difficulties in medical image segmentation include the high variability of images caused by their origin (multi-center), the acquisition protocols (multi-parametric), the variability of human anatomy, illness severity, the effect of age and gender, and notable other factors. This work addresses problems associated with the automatic semantic segmentation of lumbar spine magnetic resonance images using convolutional neural networks. We aimed to assign a class label to each pixel of an image, with classes defined by radiologists corresponding to structural elements such as vertebrae, intervertebral discs, nerves, blood vessels, and other tissues. The proposed network topologies represent variants of the U-Net architecture, and we used several complementary blocks to define the variants: three types of convolutional blocks, spatial attention models, deep supervision, and multilevel feature extractor. Here, we describe the topologies and analyze the results of the neural network designs that obtained the most accurate segmentation. Several proposed designs outperform the standard U-Net used as a baseline, primarily when used in ensembles, where the outputs of multiple neural networks are combined according to different strategies.

EDC-DTI: An end-to-end deep collaborative learning model based on multiple information for drug-target interactions prediction

Innovations in drug–target interactions (DTIs) prediction accelerate the progression of drug development. The introduction of deep learning models has a dramatic impact on DTIs prediction, with a distinct influence on saving time and money in drug discovery. This study develops an end-to-end deep collaborative learning model for DTIs prediction, called EDC-DTI, to identify new targets for existing drugs based on multiple drug-target-related information including homogeneous information and heterogeneous information by the way of deep learning. Our end-to-end model is composed of a feature builder and a classifier. Feature builder consists of two collaborative feature construction algorithms that extract the molecular properties and the topology property of networks, and the classifier consists of a feature encoder and a feature decoder which are designed for feature integration and DTIs prediction, respectively. The feature encoder, mainly based on the improved graph attention network, incorporates heterogeneous information into drug features and target features separately. The feature decoder is composed of multiple neural networks for predictions. Compared with six popular baseline models, EDC-DTI achieves highest predictive performance in the case of low computational costs. Robustness tests demonstrate that EDC-DTI is able to maintain strong predictive performance on sparse datasets. As well, we use the model to predict the most likely targets to interact with Simvastatin (DB00641), Nifedipine (DB01115) and Afatinib (DB08916) as examples. Results show that most of the predictions can be confirmed by literature with clear evidence.

Learning-based strategic operation of district cooling systems with ice storage systems

District cooling systems (DCS) can effectively satisfy the cooling demands of multiple buildings. However, since the peak cooling demands usually appear in working hours, DCS may further increase the load peak and off-peak difference in power systems. An ice storage system (ISS) can be combined with DCS to achieve peak-load shaving and reduce the operational cost of DCS. However, finding the optimal operational strategy for DCS with an ISS is challenging since the system involves many complex physical processes that are hard to describe explicitly. To address this issue, this paper proposes a learning-based approach to optimize the operation of the DCS and ISS. Based on historical operational data, the proposed approach first trains multiple neural networks to approximate the complex mappings from decision variables and environment parameters to critical variables. With these trained neural networks, the dynamics of the system can be explicitly described with no need for establishing physical models. Then, the optimization problem can be formulated by using these neural networks to replicate objectives and constraints. A projected gradient descent algorithm is further employed to solve the previous optimization problem. Simulation results based on an actual DCS show that the proposed learning-based method can achieve excellent energy efficiency while maintaining the quality of services.

Ultra-low latency recurrent neural network inference on FPGAs for physics applications with hls4ml

Recurrent neural networks have been shown to be effective architectures for many tasks in high energy physics, and thus have been widely adopted. Their use in low-latency environments has, however, been limited as a result of the difficulties of implementing recurrent architectures on field-programmable gate arrays (FPGAs). In this paper we present an implementation of two types of recurrent neural network layers—long short-term memory and gated recurrent unit—within the hls4ml framework. We demonstrate that our implementation is capable of producing effective designs for both small and large models, and can be customized to meet specific design requirements for inference latencies and FPGA resources. We show the performance and synthesized designs for multiple neural networks, many of which are trained specifically for jet identification tasks at the CERN Large Hadron Collider.

Deep metric learning for the classification of MALDI-TOF spectral signatures from multiple species of neotropical disease vectors

Deep Learning techniques have significant advantages for mass spectral classification, such as parallelized signal correction and feature extraction. Deep Metric Learning models combine Metric Learning to determine the degree of similarity or difference between a set of mass spectra with the generalization power of Deep Learning to improve feature extraction even further. The two most popular of these models combine multiple neural networks with identical architectures and are commonly called Siamese (SNN) and Triplet Neural Networks (TNN). Herein, using both SNNs and TNNs, we intended to taxonomically categorize two sets of previously-validated mass spectra that corresponded to 30 species of Neotropical arthropods in the Culicidae and Ixodidae families, some of which are disease vectors. The effectiveness of SNNs and TNNs to correctly classify 826 spectra from 12 mosquito species and 310 spectra from 18 species of hard ticks was highly effective, with both algorithms performing with minimal average loss during cross-validation. SNNs produced accuracy rates for ticks and mosquitoes of 91.22% and 94.46%, respectively, while accuracy rates of 93% and 99% were obtained with TNNs. Our results indicate that Deep Metric Learning is a practical machine learning tool for quickly and precisely classifying MALDI-TOF-generated mass spectra of Neotropical and public-health-relevant arthropod species.

Performance Analysis of Long Short-Term Memory Predictive Neural Networks on Time Series Data

Long short-term memory neural networks have been proposed as a means of creating accurate models from large time series data originating from various fields. These models can further be utilized for prediction, control, or anomaly-detection algorithms. However, finding the optimal hyperparameters to maximize different performance criteria remains a challenge for both novice and experienced users. Hyperparameter optimization algorithms can often be a resource-intensive and time-consuming task, particularly when the impact of the hyperparameters on the performance of the neural network is not comprehended or known. Teacher forcing denotes a procedure that involves feeding the ground truth output from the previous time-step as input to the current time-step during training, while during testing feeding back the predicted values. This paper presents a comprehensive examination of the impact of hyperparameters on long short-term neural networks, with and without teacher forcing, on prediction performance. The study includes testing long short-term memory neural networks, with two variations of teacher forcing, in two prediction modes, using two configurations (i.e., multi-input single-output and multi-input multi-output) on a well-known chemical process simulation dataset. Furthermore, this paper demonstrates the applicability of a long short-term memory neural network with a modified teacher forcing approach in a process state monitoring system. Over 100,000 experiments were conducted with varying hyperparameters and in multiple neural network operation modes, revealing the direct impact of each tested hyperparameter on the training and testing procedures.

Multi-phase synthetic contrast enhancement in interventional computed tomography for guiding renal cryotherapy

PurposeMinimally invasive treatments for renal carcinoma offer a low rate of complications and quick recovery. One drawback of the use of computed tomography (CT) for needle guidance is the use of iodinated contrast agents, which require an increased X-ray dose and can potentially cause adverse reactions. The purpose of this work is to generalise the problem of synthetic contrast enhancement to allow the generation of multiple phases on non-contrast CT data from a real-world, clinical dataset without training multiple convolutional neural networks.MethodsA framework for switching between contrast phases by conditioning the network on the phase information is proposed and compared with separately trained networks. We then examine how the degree of supervision affects the generated contrast by evaluating three established architectures: U-Net (fully supervised), Pix2Pix (adversarial with supervision), and CycleGAN (fully adversarial).ResultsWe demonstrate that there is no performance loss when testing the proposed method against separately trained networks. Of the training paradigms investigated, the fully adversarial CycleGAN performs the worst, while the fully supervised U-Net generates more realistic voxel intensities and performed better than Pix2Pix in generating contrast images for use in a downstream segmentation task. Lastly, two models are shown to generalise to intra-procedural data not seen during the training process, also enhancing features such as needles and ice balls relevant to interventional radiological procedures.ConclusionThe proposed contrast switching framework is a feasible option for generating multiple contrast phases without the overhead of training multiple neural networks, while also being robust towards unseen data and enhancing contrast in features relevant to clinical practice.

Deep Q-learning multiple networks based dynamic spectrum access with energy harvesting for green cognitive radio network

In cognitive radio (CR), dynamic spectrum access (DSA) can be used for secondary user (SU) to improve its spectrum efficiency under the limited spectrum resources. However, the SU has to consume more energy to sense the state of primary user (PU) before spectrum access. Therefore, high spectrum efficiency and lower energy consumption have become the requirements of CR. In this paper, we propose a sustainable green CR network combining DSA and energy harvesting, which can perform intelligent spectrum sensing, spectrum access and energy harvesting by a reward mechanism of deep Q-learning multiple networks (DQMN). The DQMN model consisting of multiple neural networks is designed to learn multiple learning objectives, including spectrum sensing, channel selection, communication power and access mode in DSA. A sleep mechanism is introduced to optimize the spectrum access mode and further reduce the energy consumption. The simulation results show that the DQMN-based DSA model can autonomously sense the channel state, harvest the energy, and then select the appropriate access strategy. Due to the energy harvesting and sleep mechanism, the proposed system model shows lower energy consumption and higher average effective throughput than the traditional models.

Neural network adaptive compensation control based on the backstepping method for nonlinear systems with time-varying delays

ABSTRACT Control of nonlinear systems with time-varying delays is a common problem, particularly when the time-varying delays are unknown. This paper presents an adaptive compensation control method based on multiple neural networks. Firstly, considering the nonlinear systems with time-varying delays, a neural network method is adopted to approximate the unknown nonlinear structure with time-varying delays. On this basis, a backstepping controller is designed. Considering the unknown time-varying delays, the control effect of the neural network controller based on the backstepping method is not optimal. To deal with the problem, a double-layer adaptive compensator for the neural network is proposed. Furthermore, for the neural network adaptive compensation controller based on the backstepping method, the stability is also proved by a Lyapunov method. Finally, simulation results demonstrate the effectiveness of the scheme, especially when the nonlinear systems have an unknown time-varying delay.

Dynamical analysis and vibration estimation of a flexible plate with enhanced active constrained layer damping treatment by combinatorial neural networks of surrogates

The effective estimation of the vibration of spacecrafts are frontier issues in the aerospace industry. But the vibration of aerospace components such as space solar panels and flexible manipulators under large overall rotating motion is usually very complex due to strong nonlinearity and often requires a lot of repetitive and burdensome calculations. Artificial neural network (ANN) is composed of interconnected neurons, and a surrogate model can be established to effectively predict the mechanical characteristics. In addition, the computational efficiency will be greatly improved if the object is changed from the model that is established by commercial finite element method (FEM) software or numerical calculations to the surrogates. However, the stochasticity of the estimation by single neural network is inevitable, and some neural networks much rely on the initial weight and threshold. This paper will implement a new combinatorial strategy that relies on multiple neural network models, which have different activation functions and regression feedback processes based on reasonable selection of sample points. To solve the elongated training time led by comparing these neural networks, genetic algorithm (GA) is used to optimize the iteration speed of one neural network when finding the optimal output value. Thus, better estimation accuracy will be achieved by designing an algorithm to combine these combinatorial neural networks of surrogates (CNNS). Based on the proposed method, the vibration estimation of a rotating flexible plate with enhanced active constrained layer damping (EACLD) treatment, and the vibration suppression capacity of the composite structure under various operating conditions will be investigated.

USING PHYSICS-INFORMED NEURAL NETWORKS TO SOLVE FOR PERMEABILITY FIELD UNDER TWO-PHASE FLOW IN HETEROGENEOUS POROUS MEDIA

Physics-informed neural networks (PINNs) have recently been applied to a wide range of computational physical problems. In this paper, we use PINNs to solve an inverse two-phase flow problem in heterogeneous porous media where only sparse direct and indirect measurements are available. The forward two-phase flow problem is governed by a coupled system of partial differential equations (PDEs) with initial and boundary conditions. As for inverse problems, the solutions are assumed to be known at scattered locations but some coefficients or variable functions in the PDEs are missing or incomplete. The idea is to train multiple neural networks representing the solutions and the unknown variable function at the same time such that both the underlying physical laws and the measurements can be honored. The numerical results show that our proposed method is able to recover the incomplete permeability field in different scenarios. Moreover, we show that the method can be used to forecast the future dynamics with the same format of loss function formulation. In addition, we employ a neural network structure inspired by the deep operator networks (DeepONets) to represent the solutions which can potentially shorten the time of the training process.