Types Of Neural Network Architectures Research Articles

Analysis of variables to determine their influence on renewable energy forecasting using ensemble methods

Forecasting is of great importance in the field of renewable energies because it allows us to know the quantity of energy that can be produced, and thus, to have an efficient management of energy sources. However, determining which prediction system is more adequate is very complex, as each energy infrastructure is different. This work studies the influence of some variables when making predictions using ensemble methods for different locations. In particular, the proposal analyzes the influence of the aspects: the variation of the sampling frequency of solar panel systems, the influence of the type of neural network architecture and the number of ensemble method blocks for each model. Following comprehensive experimentation across multiple locations, our study has identified the most effective solar energy prediction model tailored to the specific conditions of each energy infrastructure. The results offer a decisive framework for selecting the optimal system for accurate and efficient energy forecasting. The key point is the use of short time intervals, which is independent of type of prediction model and of their ensemble method.

Cancer detection and classification using a simplified binary state vector machine.

Cancer is an invasive and malignant growth of cells and is known to be one of the most fatal diseases. Its early detection is essential for decreasing the mortality rate and increasing the probability of survival. This study presents an efficient machine learning approach based on the state vector machine (SVM) to diagnose and classify tumors into malignant or benign cancer using the online lymphographic data. Further, two types of neural network architectures are also implemented to evaluate the performance of the proposed SVM-based approach. The optimal structures of the classifiers are obtained by varying the architecture, topology, learning rate, and kernel function and recording the results' accuracy. The classifiers are trained with the preprocessed data examples after noise removal and tested on the unknown cases to diagnose each example as positive or negative. Further, the positive cases are classified into different stages including metastases, malign lymph, and fibrosis. The results are evaluated against the feed-forward and generalized regression neural networks. It is found that the proposed SVM-based approach significantly improves the early detection and classification accuracy in comparison to the experienced physicians and the other machine learning approaches. The proposed approach is robust and can perform sub-class divisions for multipurpose tasks. Experimental results demonstrate that the two-class SVM gives the best results and can effectively be used for the classification of cancer. It has outperformed all other classifiers with an average accuracy of 94.90%.

Chatbots and Large Language Models in Radiology: A Practical Primer for Clinical and Research Applications.

Although chatbots have existed for decades, the emergence of transformer-based large language models (LLMs) has captivated the world through the most recent wave of artificial intelligence chatbots, including ChatGPT. Transformers are a type of neural network architecture that enables better contextual understanding of language and efficient training on massive amounts of unlabeled data, such as unstructured text from the internet. As LLMs have increased in size, their improved performance and emergent abilities have revolutionized natural language processing. Since language is integral to human thought, applications based on LLMs have transformative potential in many industries. In fact, LLM-based chatbots have demonstrated human-level performance on many professional benchmarks, including in radiology. LLMs offer numerous clinical and research applications in radiology, several of which have been explored in the literature with encouraging results. Multimodal LLMs can simultaneously interpret text and images to generate reports, closely mimicking current diagnostic pathways in radiology. Thus, from requisition to report, LLMs have the opportunity to positively impact nearly every step of the radiology journey. Yet, these impressive models are not without limitations. This article reviews the limitations of LLMs and mitigation strategies, as well as potential uses of LLMs, including multimodal models. Also reviewed are existing LLM-based applications that can enhance efficiency in supervised settings.

Convolution, aggregation and attention based deep neural networks for accelerating simulations in mechanics

Deep learning surrogate models are being increasingly used in accelerating scientific simulations as a replacement for costly conventional numerical techniques. However, their use remains a significant challenge when dealing with real-world complex examples. In this work, we demonstrate three types of neural network architectures for efficient learning of highly non-linear deformations of solid bodies. The first two architectures are based on the recently proposed CNN U-NET and MAgNET (graph U-NET) frameworks which have shown promising performance for learning on mesh-based data. The third architecture is Perceiver IO, a very recent architecture that belongs to the family of attention-based neural networks–a class that has revolutionised diverse engineering fields and is still unexplored in computational mechanics. We study and compare the performance of all three networks on two benchmark examples, and show their capabilities to accurately predict the non-linear mechanical responses of soft bodies.

Deep-Learning-Based Sequence Causal Long-Term Recurrent Convolutional Network for Data Fusion Using Video Data

The purpose of AI-Based schemes in intelligent systems is to advance and optimize system performance. Most intelligent systems adopt sequential data types derived from such systems. Realtime video data, for example, are continuously updated as a sequence to make necessary predictions for efficient system performance. The majority of deep-learning-based network architectures such as long short-term memory (LSTM), data fusion, two streams, and temporal convolutional network (TCN) for sequence data fusion are generally used to enhance robust system efficiency. In this paper, we propose a deep-learning-based neural network architecture for non-fix data that uses both a causal convolutional neural network (CNN) and a long-term recurrent convolutional network (LRCN). Causal CNNs and LRCNs use incorporated convolutional layers for feature extraction, so both architectures are capable of processing sequential data such as time series or video data that can be used in a variety of applications. Both architectures also have extracted features from the input sequence data to reduce the dimensionality of the data and capture the important information, and learn hierarchical representations for effective sequence processing tasks. We have also adopted a concept of series compact convolutional recurrent neural network (SCCRNN), which is a type of neural network architecture designed for processing sequential data combined by both convolutional and recurrent layers compactly, reducing the number of parameters and memory usage to maintain high accuracy. The architecture is challenge-able and suitable for continuously incoming sequence video data, and doing so allowed us to bring advantages to both LSTM-based networks and CNNbased networks. To verify this method, we evaluated it through a sequence learning model with network parameters and memory that are required in real environments based on the UCF-101 dataset, which is an action recognition data set of realistic action videos, collected from YouTube with 101 action categories. The results show that the proposed model in a sequence causal long-term recurrent convolutional network (SCLRCN) provides a performance improvement of at least 12% approximately or more to be compared with the existing models (LRCN and TCN).

Passing Vehicle Road Occupancy Detection Using the Magnetic Sensor Array

The increasing presence of vehicles on roads necessitates intelligent traffic management solutions in areas where video cameras cannot be utilized. Currently, there are limited choices for depersonalized vehicle reidentification systems. This paper introduces a system that later will be used for vehicle reidentification. The system uses anisotropic magnetoresistive sensors and is based on the hypothesis that each vehicle leaves unique magnetic signatures which can be used for comparison and matching. Vehicle location on the road perpendicular to sensor array detection methodology is presented in this work. An array of magnetic sensors is installed in asphalt across the vehicle’s driving direction. The system continuously measures Earth’s natural magnetic field and detects distortions when vehicles pass an array of a sensors and then logs magnetic signatures. Useful parameters from raw sensor axes are calculated– modules and derivatives. Applying signal-to-noise ratio calculation for module derivatives between ambient noise and signal gives important features for neural network input. Different types of neural network architectures and output result interpretation techniques are investigated. Further, after evaluating network output it is possible to label sensor nodes that are directly beneath the vehicle. Experiment results show that implemented algorithm is highly sufficient for valid sensors under the vehicle selection. Correct sensor selection is important for further re-identification algorithms.

Evaluation of neural networks for residential load forecasting and the impact of systematic feature identification

Energy systems face challenges due to climate change, distributed energy resources, and political agenda, especially distribution system operators (DSOs) responsible for ensuring grid stability. Accurate predictions of the electricity load can help DSOs better plan and maintain their grids. The study aims to test a systematic data identification and selection process to forecast the electricity load of Danish residential areas. The five-ecosystem CSTEP framework maps relevant independent variables on the cultural, societal, technological, economic, and political dimensions. Based on the literature, a recurrent neural network (RNN), long-short-term memory network (LSTM), gated recurrent unit (GRU), and feed-forward network (FFN) are evaluated and compared. The models are trained and tested using different data inputs and forecasting horizons to assess the impact of the systematic approach and the practical flexibility of the models. The findings show that the models achieve equal performances of around 0.96 adjusted R2 score and 4–5% absolute percentage error for the 1-h predictions. Forecasting 24 h gave an adjusted R2 of around 0.91 and increased the error slightly to 6–7% absolute percentage error. The impact of the systematic identification approach depended on the type of neural network, with the FFN showing the highest increase in error when removing the supporting variables. The GRU and LSTM did not rely on the identified variables, showing minimal changes in performance with or without them. The systematic approach to data identification can help researchers better understand the data inputs and their impact on the target variable. The results indicate that a focus on curating data inputs affects the performance more than choosing a specific type of neural network architecture.

Photometric redshifts from SDSS images with an interpretable deep capsule network

ABSTRACT Studies of cosmology, galaxy evolution, and astronomical transients with current and next-generation wide-field imaging surveys like the Rubin Observatory Legacy Survey of Space and Time are all critically dependent on estimates of photometric redshifts. Capsule networks are a new type of neural network architecture that is better suited for identifying morphological features of the input images than traditional convolutional neural networks. We use a deep capsule network trained on ugriz images, spectroscopic redshifts, and Galaxy Zoo spiral/elliptical classifications of ∼400 000 Sloan Digital Sky Survey galaxies to do photometric redshift estimation. We achieve a photometric redshift prediction accuracy and a fraction of catastrophic outliers that are comparable to or better than current methods for SDSS main galaxy sample-like data sets (r ≤ 17.8 and zspec ≤ 0.4) while requiring less data and fewer trainable parameters. Furthermore, the decision-making of our capsule network is much more easily interpretable as capsules act as a low-dimensional encoding of the image. When the capsules are projected on a two-dimensional manifold, they form a single redshift sequence with the fraction of spirals in a region exhibiting a gradient roughly perpendicular to the redshift sequence. We perturb encodings of real galaxy images in this low-dimensional space to create synthetic galaxy images that demonstrate the image properties (e.g. size, orientation, and surface brightness) encoded by each dimension. We also measure correlations between galaxy properties (e.g. magnitudes, colours, and stellar mass) and each capsule dimension. We publicly release our code, estimated redshifts, and additional catalogues at https://biprateep.github.io/encapZulate-1.

Intelligent Audio Signal Processing for Detecting Rainforest Species Using Deep Learning

Hearing a species in a tropical rainforest is much easier than seeing them. If someone is in the forest, he might not be able to look around and see every type of bird and frog that are there but they can be heard. A forest ranger might know what to do in these situations and he/she might be an expert in recognizing the different type of insects and dangerous species that are out there in the forest but if a common person travels to a rain forest for an adventure, he might not even know how to recognize these species, let alone taking suitable action against them. In this work, a model is built that can take audio signal as input, perform intelligent signal processing for extracting features and patterns, and output which type of species is present in the audio signal. The model works end to end and can work on raw input and a pipeline is also created to perform all the preprocessing steps on the raw input. In this work, different types of neural network architectures based on Long Short Term Memory (LSTM) and Convolution Neural Network (CNN) are tested. Both are showing reliable performance, CNN shows an accuracy of 95.62% and Log Loss of 0.21 while LSTM shows an accuracy of 93.12% and Log Loss of 0.17. Based on these results, it is shown that CNN performs better than LSTM in terms of accuracy while LSTM performs better than CNN in terms of Log Loss. Further, both of these models are combined to achieve high accuracy and low Log Loss. A combination of both LSTM and CNN shows an accuracy of 97.12% and a Log Loss of 0.16.

Forecasting time series using neural networks on the example of primary sales of a pharmaceutical company

This article is devoted to neural network forecasting of primary sales of a pharmaceutical company. The research includes an analysis of the main forecasting methods, various types of neural network architectures used to predict time series, as well as an analysis of neural network training methods. The primary sales of the pharmaceutical company AKRIKHIN JSC were considered as the applied basis of the study. As part of the study, an analysis of the revenue of the pharmaceutical company AKRIKHIN JSC from 2016 to 2021 was carried out, the selection of the main hyperparameters of the neural network was carried out and a forecast of primary sales for 4 weeks ahead was obtained. The practical significance of the research results is due to the fact that forecasting primary sales gives management an understanding of which direction the company is moving in and whether additional management decisions are needed to meet the planned values. This study can be used as an alternative method of sales forecasting, so the results will be in demand among analysts of pharmaceutical companies in sales planning departments.

Architectures for neural networks as surrogates for dynamic systems in chemical engineering

Surrogate models for dynamic systems in chemical engineering are increasingly of interest. Neural networks have already been applied in research, but it remains unclear which types of neural network architectures are actually required for practical systems. The focus here lies on recurrent neural networks of type Jordan, Elman, and LSTM layers. These are investigated for different types of data sets as training basis: batch trajectories, data of a proper excitation of a continuous process, and a typical operation trajectory of a large chemical plant. To ensure a rigorous investigation, hyperparameter tuning by Bayesian and Bandit optimization is included. As a first, a dynamic surrogate model using LSTM layers for a batch distillation system is presented, which is valid from start-up until shutdown. The evaluation shows further need for adjustments in data preparation and objective/loss function compared to the state of the art.

CapsTM: capsule network for Chinese medical text matching

BackgroundText Matching (TM) is a fundamental task of natural language processing widely used in many application systems such as information retrieval, automatic question answering, machine translation, dialogue system, reading comprehension, etc. In recent years, a large number of deep learning neural networks have been applied to TM, and have refreshed benchmarks of TM repeatedly. Among the deep learning neural networks, convolutional neural network (CNN) is one of the most popular networks, which suffers from difficulties in dealing with small samples and keeping relative structures of features. In this paper, we propose a novel deep learning architecture based on capsule network for TM, called CapsTM, where capsule network is a new type of neural network architecture proposed to address some of the short comings of CNN and shows great potential in many tasks.MethodsCapsTM is a five-layer neural network, including an input layer, a representation layer, an aggregation layer, a capsule layer and a prediction layer. In CapsTM, two pieces of text are first individually converted into sequences of embeddings and are further transformed by a highway network in the input layer. Then, Bidirectional Long Short-Term Memory (BiLSTM) is used to represent each piece of text and attention-based interaction matrix is used to represent interactive information of the two pieces of text in the representation layer. Subsequently, the two kinds of representations are fused together by BiLSTM in the aggregation layer, and are further represented with capsules (vectors) in the capsule layer. Finally, the prediction layer is a connected network used for classification. CapsTM is an extension of ESIM by adding a capsule layer before the prediction layer.ResultsWe construct a corpus of Chinese medical question matching, which contains 36,360 question pairs. This corpus is randomly split into three parts: a training set of 32,360 question pairs, a development set of 2000 question pairs and a test set of 2000 question pairs. On this corpus, we conduct a series of experiments to evaluate the proposed CapsTM and compare it with other state-of-the-art methods. CapsTM achieves the highest F-score of 0.8666.ConclusionThe experimental results demonstrate that CapsTM is effective for Chinese medical question matching and outperforms other state-of-the-art methods for comparison.

Survey on Early Detection of Alzheimer's Disease using Different Types of Neural Network Architecture

Alzheimer’s disease is a condition that leads to, progressive neurological brain disorder and destroys cells of the brain thereby causing an individual to lose their ability to continue daily activities and also hampers their mentality. Diagnostic symptoms are experienced by patients usually at later stages after irreversible neural damage occurs. Detection of AD is challenging because sometimes the signs that distinguish AD MRI data, can be found in MRI data of normal healthy brains of older people. Even though this disease is not completely curable, earlier detection can aid in promising treatment and prevent permanent damage to brain tissues. Age and genetics are the greatest risk factors for this disease. This paper presents the latest reports on AD detection based on different types of Neural Network Architectures.

AutoPPI: An Ensemble of Deep Autoencoders for Protein-Protein Interaction Prediction.

Proteins are essential molecules, that must correctly perform their roles for the good health of living organisms. The majority of proteins operate in complexes and the way they interact has pivotal influence on the proper functioning of such organisms. In this study we address the problem of protein–protein interaction and we propose and investigate a method based on the use of an ensemble of autoencoders. Our approach, entitled , adopts a strategy based on two autoencoders, one for each type of interactions (positive and negative) and we advance three types of neural network architectures for the autoencoders. Experiments were performed on several data sets comprising proteins from four different species. The results indicate good performances of our proposed model, with accuracy and AUC values of over 0.97 in all cases. The best performing model relies on a Siamese architecture in both the encoder and the decoder, which advantageously captures common features in protein pairs. Comparisons with other machine learning techniques applied for the same problem prove that outperforms most of its contenders, for the considered data sets.

Spatio-Temporal Traffic Flow Prediction in Madrid: An Application of Residual Convolutional Neural Networks

Due to the need to predict traffic congestion during the morning or evening rush hours in large cities, a model that is capable of predicting traffic flow in the short term is needed. This model would enable transport authorities to better manage the situation during peak hours and would allow users to choose the best routes for reaching their destinations. The aim of this study was to perform a short-term prediction of traffic flow in Madrid, using different types of neural network architectures with a focus on convolutional residual neural networks, and it compared them with a classical time series analysis. The proposed convolutional residual neural network is superior in all of the metrics studied, and the predictions are adapted to various situations, such as holidays or possible sensor failures.

Model order reduction based on Runge–Kutta neural networks

Abstract Model order reduction (MOR) methods enable the generation of real-time-capable digital twins, with the potential to unlock various novel value streams in industry. While traditional projection-based methods are robust and accurate for linear problems, incorporating machine learning to deal with nonlinearity becomes a new choice for reducing complex problems. These kinds of methods are independent to the numerical solver for the full order model and keep the nonintrusiveness of the whole workflow. Such methods usually consist of two steps. The first step is the dimension reduction by a projection-based method, and the second is the model reconstruction by a neural network (NN). In this work, we apply some modifications for both steps respectively and investigate how they are impacted by testing with three different simulation models. In all cases Proper orthogonal decomposition is used for dimension reduction. For this step, the effects of generating the snapshot database with constant input parameters is compared with time-dependent input parameters. For the model reconstruction step, three types of NN architectures are compared: multilayer perceptron (MLP), explicit Euler NN (EENN), and Runge–Kutta NN (RKNN). The MLPs learn the system state directly, whereas EENNs and RKNNs learn the derivative of system state and predict the new state as a numerical integrator. In the tests, RKNNs show their advantage as the network architecture informed by higher-order numerical strategy.

Book spine recognition with the use of deep neural networks

Nowadays deep neural networks play a significant part in various fields of human activity. Especially they benefit spheres dealing with large amounts of data and lengthy operations on obtaining and processing information from the visual environment. This article deals with the development of a convolutional neural network based on the YOLO architecture, intended for real-time book recognition. The creation of an original data set and the training of the deep neural network are described. The structure of the neural network obtained is presented and the most frequently used metrics for estimating the quality of the network performance are considered. A brief review of the existing types of neural network architectures is also made. YOLO architecture possesses a number of advantages that allow it to successfully compete with other models and make it the most suitable variant for creating an object detection network since it enables some of the common disadvantages of such networks to be significantly mitigated (such as recognition of similarly looking, same-color book coves or slanted books). The results obtained in the course of training the deep neural network allow us to use it as a basis for the development of the software for book spine recognition.

De-noising of transient electromagnetic data based on the long short-term memory-autoencoder

SUMMARYLate-time transient electromagnetic (TEM) data contain deep subsurface information and are important for resolving deeper electrical structures. However, due to their relatively small signal amplitudes, TEM responses later in time are often dominated by ambient noises. Therefore, noise removal is critical to the application of TEM data in imaging electrical structures at depth. De-noising techniques for TEM data have been developed rapidly in recent years. Although strong efforts have been made to improving the quality of the TEM responses, it is still a challenge to effectively extract the signals due to unpredictable and irregular noises. In this study, we develop a new type of neural network architecture by combining the long short-term memory (LSTM) network with the autoencoder structure to suppress noise in TEM signals. The resulting LSTM-autoencoders yield excellent performance on synthetic data sets including horizontal components of the electric field and vertical component of the magnetic field generated by different sources such as dipole, loop and grounded line sources. The relative errors between the de-noised data sets and the corresponding noise-free transients are below 1% for most of the sampling points. Notable improvement in the resistivity structure inversion result is achieved using the TEM data de-noised by the LSTM-autoencoder in comparison with several widely-used neural networks, especially for later-arriving signals that are important for constraining deeper structures. We demonstrate the effectiveness and general applicability of the LSTM-autoencoder by de-noising experiments using synthetic 1-D and 3-D TEM signals as well as field data sets. The field data from a fixed loop survey using multiple receivers are greatly improved after de-noising by the LSTM-autoencoder, resulting in more consistent inversion models with significantly increased exploration depth. The LSTM-autoencoder is capable of enhancing the quality of the TEM signals at later times, which enables us to better resolve deeper electrical structures.

Neural network analysis of mortality risk predictors in patients after acute coronary syndrome

Aim. To study the possibilities of neural network analysis of clinical and instrumental data to predict the mortality risk in patients after acute coronary syndrome (ACS).Material and methods. The study involved 400 patients after ACS which who observed for 62 months. The criterion for the complicated course of coronary artery disease (CAD) is the cardiovascular death. Group 1 consisted of 310 patients with uncomplicated course of CAD; group 2 — 90 patients with complicated course of CAD. To predict mortality risk, the machine learning method and neural network analysis was used. Machine learning was carried out with the inclusion of clinical, laboratory and instrumental (electrocardiography, echocardiography) parameters (49 in total). To solve the classification problems, two types of neural network architectures were used: Multilayer Perceptron (MLP) and Convolutional Neural Networks (CNN). The ratio in the examples for learning and validation was 340/60. The method of learning with a teacher was used on the available data in which the outcomes were known, and the neural network parameters were adjusted so as to minimize the error.Results. The following factors made the highest contribution to the mortality risk after ACS: age; history of MI and acute cerebrovascular accident; atrial fibrillation, class 2-3 heart failure; no history of percutaneous coronary intervention; stage 3 chronic kidney disease; reduced left ventricle ejection fraction. Most of the deaths occurred in the 2nd and 4th years of follow-up, which may be due to the low effectiveness of secondary prevention of CAD. CNN architecture had higher accuracy (sensitivity — 68%; specificity — 84%; area under curve=0,74). An advantage of CNN is its ability to analyze patterns over time using recurrent neural networks.Conclusion. Neural network analysis of clinical, laboratory and instrumental data allows configuring network parameters for mortality risk prediction. CNN predicts 5-year mortality risk after ACS with a sensitivity of 68% and a specificity of 84%.

A Kernel Density Estimation Based Loss Function and its Application to ASV-Spoofing Detection

Biometric systems are exposed to spoofing attacks which may compromise their security, and voice biometrics, also known as automatic speaker verification (ASV), is no exception. Replay, synthesis and voice conversion attacks cause false acceptances that can be detected by anti-spoofing systems. Recently, deep neural networks (DNNs) which extract embedding vectors have shown superior performance than conventional systems in both ASV and anti-spoofing tasks. In this work, we develop a new concept of loss function for training DNNs which is based on kernel density estimation (KDE) techniques. The proposed loss functions estimate the probability density function (pdf) of every training class in each mini-batch, and compute a log likelihood matrix between the embedding vectors and pdfs of all training classes within the mini-batch in order to obtain the KDE-based loss. To evaluate our proposal for spoofing detection, experiments were carried out on the recent ASVspoof 2019 corpus, including both logical and physical access scenarios. The experimental results show that training a DNN based anti-spoofing system with our proposed loss functions clearly outperforms the performance of the same system being trained with other well-known loss functions. Moreover, the results also show that the proposed loss functions are effective for different types of neural network architectures.