Multilayer Neural Network Architecture Research Articles

Neuromorphic overparameterisation and few-shot learning in multilayer physical neural networks

Physical neuromorphic computing, exploiting the complex dynamics of physical systems, has seen rapid advancements in sophistication and performance. Physical reservoir computing, a subset of neuromorphic computing, faces limitations due to its reliance on single systems. This constrains output dimensionality and dynamic range, limiting performance to a narrow range of tasks. Here, we engineer a suite of nanomagnetic array physical reservoirs and interconnect them in parallel and series to create a multilayer neural network architecture. The output of one reservoir is recorded, scaled and virtually fed as input to the next reservoir. This networked approach increases output dimensionality, internal dynamics and computational performance. We demonstrate that a physical neuromorphic system can achieve an overparameterised state, facilitating meta-learning on small training sets and yielding strong performance across a wide range of tasks. Our approach’s efficacy is further demonstrated through few-shot learning, where the system rapidly adapts to new tasks.

AN ANN MODEL FOR PREDICTING SLUMP OF CONCRETE CONTAINING CRUSHED GLASS AND NATURAL GRAVEL

This study modelled the slump of concrete containing crushed glass and Bida Natural Gravel (BNG) based on deep learning algorithm using the MATLAB neural network toolbox. A total of 240 (150mm × 150mm × 150mm) cubes were cast from 80 mixes generated randomly using Scheffe’s simplex lattice approach. Slump was measured for each of the experimental points of fresh concrete before filling in the moulds. The resulting batch for each mix was used as input data while the laboratory results for slump was used as output data for the ANN-model. Hence a shallow multilayer supervised Neural Network was developed to model these data. The developed model would be able to predict concrete slump containing 0% - 25% crushed glass as partial replacement for fine aggregate, water- cement ratio ranging from 0.45 – 0.65 and concrete grade M15 – M25. The architecture of the network contained 6 input parameters: water to cement ratio, water, cement, sand, crushed glass and BNG, 20 neurons in the hidden layer and slump in the outer layer. The adequacy of the developed model was measured using Mean Square Error (MSE) and Correlation Coefficient (R). Results showed that 6:20:1 model architecture for slump model had an MSE values for training, validation and testing as: 1.84e-2, 5.81e-3, 3.64e-3, 1.73e-3 respectively. Regression values for training, validation and testing are: 79%, 94%, 96% and 79%. The study concluded that a shallow multilayer Neural Network architecture with 20 neurons in the hidden layer is sufficient for predicting concrete slump.

RNN-Based Time Series Analysis for Wind Turbine Energy Forecasting

One significant source of renewable energy is wind power, which has the potential to generate sustainable energy. However, wind turbines have many challenges, such as high initial investment costs, the dynamic nature of wind speed, and the challenge of locating wind-efficient energy regions. Wind power predicting is crucial for effective planning of wind power generation, optimization of power generation, grid integration, and security of supply. Therefore, highly accurate forecasts ensure the efficient and sustainable operation of the wind energy sector and contribute to energy security, economic stability, and environmental sustainability. This study proposes a deep learning (DL) approach based on recurrent neural networks (RNNs) for long-term wind power forecasting utilizing climatic data. The input data that forms the basis of this study is obtained directly from a wind turbine system operating under real-world conditions. The proposed model in this study is based on a multilayer back-propagation neural network (RNN) architecture specifically designed to effectively handle complex data sets and time-dependent series. The architecture of the model is built on an RNN consisting of four separate layers, each with 50 hidden neurons, carefully structured to increase its capacity to capture complex features. To improve the robustness of the model and avoid overlearning, each RNN layer is followed by a dropout (regularizing) layer that randomly deactivates 20% of the neurons to enhance the generalization ability of the network. To finalize the prediction capability of the model, a linear function was chosen in the last layer to directly match the actual values. Evaluating the model performance metrics, the proposed architecture achieved a prediction accuracy of 91% R2 on the test dataset. The findings indicate that proposed method based on multilayer RNN can successfully capture the relationships between the sequential data of the wind turbine.

Advantages and machine learning role in assessing the enterprises competitiveness in the digital economy

This work examines the possibilities and advantages using neural networks to assess the competitive enterprise’s ability in the digital economy. This issue is relevant, considering the progressing digital economy development and the data accumulation on the enterprise’s activities. Analysis of this data with machine learning methods will make it possible to effectively assess the enterprises competitiveness and make effective management decisions that will bring the company to a high level not only in different regions, but also in the country. To achieve this goal, it is necessary to identify the main factors that have the greatest impact on the business. Such factors include, for example, the introduction electronic document management, business processes robotization using the RPA platform, or their automation using programming languages. The most popular and convenient programming languages for solving similar problems are Python, C#, PowerShell. The latter offers a good approach for communicating with Windows OS and automating system administration tasks. Among the main factors influencing competitiveness, a total of 5 most significant factors were identified for the task: the financial position of the enterprise, the image of the organization, the quality of services, the personnel qualifications level and organization’s digitalization level. Competitiveness classes are proposed to determine the enterprise competitiveness. Three classes were defined for classification: Below_Norm, Normal, Above_Norm. To solve the problem, a multilayer neural network architecture has been developed, the implementation of which is implemented in Python. The use of such a solution makes it possible to achieve an accuracy of more than 90% when solving the classification task. To confirm the neural network functionality, a method for constructing a confusion matrix has been implemented. This matrix shows the percentage of objects that are accurately classified according to the incorrectly detected. Attached are the results of the neural network when solving the problem of classifying competitiveness. According to the data obtained, it was concluded that the values fed to the input classifier may be classified inaccurately, because they are on the boundaries of the results between the observed classes. The data for training neural networks was generated randomly and had a matrix size of 30x5. Since the purpose of the work is to argue for a machine learning approach to solve determining competitiveness problem, there was no need to use real data. The k-nearest neighbors method was chosen as the classification method, since this method shows high accuracy in solving such problems.

PEB-DDI: A Task-Specific Dual-View Substructural Learning Framework for Drug-Drug Interaction Prediction.

Adverse drug-drug interactions (DDIs) pose potential risks in polypharmacy due to unknown physicochemical incompatibilities between co-administered drugs. Recent studies have utilized multi-layer graph neural network architectures to model hierarchical molecular substructures of drugs, achieving excellent DDI prediction performance. While extant substructural frameworks effectively encode interactions from atom-level features, they overlook valuable chemical bond representations within molecular graphs. More critically, given the multifaceted nature of DDI prediction tasks involving both known and novel drug combinations, previous methods lack tailored strategies to address these distinct scenarios. The resulting lack of adaptability impedes further improvements to model performance. To tackle these challenges, we propose PEB-DDI, a DDI prediction learning framework with enhanced substructure extraction. First, the information of chemical bonds is integrated and synchronously updated with the atomic nodes. Then, different dual-view strategies are selected based on whether novel drugs are present in the prediction task. Particularly, we constructed Molecular fingerprint-Molecular graph view for transduc- tive task, and Bipartite graph-Molecular graph view for inductive task. Rigorous evaluations on benchmark datasets underscore PEB-DDI's superior performance. Notably, on DrugBank, it achieves an outstanding accuracy rate of 98.18% when predicting previously unknown interactions among approved drugs. Even when faced with novel drugs, PEB-DDI consistently exhibits outstanding generalization capabilities with an accuracy rate of 88.06%, attributing to the proper migrating of molecular basic structure learning.

Comparison of multilayer perceptron neural network architecture in photovoltaic plants fault classification

The goal of this study is to assess the application of Multilayer Perceptron Artificial Neural Networks in fault classification within photovoltaic panels, focusing on key characteristics such as the number of neurons and layers, activation functions, training techniques, and the resulting accuracy. The study employs a comparative analysis approach, examining various characteristics and hyperparameters applied to Multilayer Perceptron Artificial Neural Networks for fault classification in photovoltaic panels. The research methodology involves reviewing publications from the past decade to gather data on these characteristics and their impact on fault analysis in photovoltaic generation systems. This study contributes to the originality of the field by providing a comprehensive comparison of various parameters and techniques used in Multilayer Perceptron Artificial Neural Networks for fault classification in photovoltaic panels. The findings offer valuable insights for researchers and practitioners in the renewable energy sector, aiding in the development of more efficient and reliable fault diagnosis systems for photovoltaic generation.

Predicting gene expression divergence between single-copy orthologs in two species.

Predicting gene expression divergence is integral to understanding the emergence of new biological functions and associated traits. Whereas several sophisticated methods have been developed for this task, their applications are either limited to duplicate genes or require expression data from more than two species. Thus, here we present PiXi, the first machine learning framework for predicting gene expression divergence between single-copy orthologs in two species. PiXi models gene expression evolution as an Ornstein-Uhlenbeck process, and overlays this model with multi-layer neural network, random forest, and support vector machine architectures for making predictions. It outputs the predicted class "conserved" or "diverged" for each pair of orthologs, as well as their predicted expression optima in the two species. We show that PiXi has high power and accuracy in predicting gene expression divergence between single-copy orthologs, as well as high accuracy and precision in estimating their expression optima in the two species, across a wide range of evolutionary scenarios, with the globally best performance achieved by a multi-layer neural network. Moreover, application of our best performing PiXi predictor to empirical gene expression data from single-copy orthologs residing at different loci in two species of Drosophila reveals that approximately 23% underwent expression divergence after positional relocation. Further analysis shows that several of these "diverged" genes are involved in the electron transport chain of the mitochondrial membrane, suggesting that new chromatin environments may impact energy production in Drosophila. Thus, by providing a toolkit for predicting gene expression divergence between single-copy orthologs in two species, PiXi can shed light on the origins of novel phenotypes across diverse biological processes and study systems.

PO-04-212 BRUGADA EKG CLASSIFICATION WITH SELF-SUPERVISED VICREG PERTAINING: A NOVEL ARTIFICIAL INTELLIGENCE ARCHITECTURE FOR RARE ARRHYTHMIA

Brugada Syndrome (BrS) is a rare inherited arrhythmia syndrome with an estimated prevalence of 2-5 per 10,000. Despite its low prevalence, it is responsible for an estimated 4% to 12% of all sudden deaths and in patients with structurally normal hearts. Because of the resulting limited number of available BrS EKG examples, Convoluted Neural Networks (CNNs), which require a large labeled dataset for training, are limited in their ability to automate the detection of BrS. To develop an Artificial Intelligence (AI) model for classifying BrS in the surface EKG by applying a novel self-supervised pretraining architecture on unrelated, unlabeled EKGs, compensating for the limited number of BrS examples. We pretrained a convolutional neural network on randomly selected 3,500,000 EKGs from adult NYU patients utilizing VICReg (Variance-Invariance-Covariance Regularization), a novel regularization architecture for self-supervised training allowing the model to learn embedding (vectors) for each EKG that are informative and unique (Figure 1). We then fine tuned the original model on a set containing 200 adult NYU patients with a MUSE (GE, USA) diagnosis of BrS not included in the pretraining phase, and evaluated the fine-tuned model for binary classification of Brugada. We used a modification of a multilayered convolutional neural network architecture similar to ConvNext, with a Softmax activation function for classification. Of the 200 EKGs with a MUSE label of BrS, 114 (57%) were confirmed as BrS by an expert Electrophysiologist and blended with a cohort of randomly selected 200 EKGs with non – BrS label. The combined cohort was divided into 80% training (with 20% validation) and 20% test sub-sets. The CNN classified BrS with an AUC of 0.953 (sensitivity 0.87, specificity 0.87), outperforming all previously described supervised learning models trained on larger datasets. We developed a novel deep learning model using VICReg architecture for self-supervised pretraining. We demonstrate the classification of BrS EKGs with high performance achieved with a remarkably small labeled dataset. Supervised pretraining with VICReg may be a useful architecture for improving model classification performance on rare conditions with limited availability of labeled data.

Scalable serial hardware architecture of multilayer perceptron neural network for automatic wheezing detection

This paper proposes a serial hardware architecture of a multilayer perceptron (MLP) neural network for real-time wheezing detection in respiratory sounds. As an established classification tool, the MLP has proven its ability to identify complex patterns within respiratory sounds. The proposed fully serial architecture uses a single calculation unit, independently of the number of neurons in the MLP network. It is also a fully scalable architecture that permits to implement MLP networks, of any size, easily and efficiently without modifying the design or wiring. The proposed serial architecture has been implemented on a low-cost and power-efficient field programmable gate array (FPGA) chip using a high-level programming tool. The respiratory sounds classification rates are evaluated in terms of sensitivity, specificity, performance, and accuracy. The proposed serial architecture reaches the same classification performances as the parallel one, but it presents the main advantage of using much fewer hardware resources.

Human activity recognition with self-attention

In this paper, a self-attention based neural network architecture to address human activity recognition is proposed. The dataset used was collected using smartphone. The contribution of this paper is using a multi-layer multi-head self-attention neural network architecture for human activity recognition and compared to two strong baseline architectures, which are convolutional neural network (CNN) and long-short term network (LSTM). The dropout rate, positional encoding and scaling factor are also been investigated to find the best model. The results show that proposed model achieves a test accuracy of 91.75%, which is a comparable result when compared to both the baseline models.

Tiny object detection model based on competitive multi-layer neural network (TOD-CMLNN)

Tiny Object Detection (TOD) is a fundamental and difficult task in computer vision. Current state-of-the-art detectors like RCNN, Fast RCNN, Faster RCNN, SSD, and YOLO can't find small objects using single-stage or multi-stage methods. With the exponential growth of deep learning, several researchers have drawn attention to advances in tiny object detection approaches. This study proposes a TOD-CMLNN (Tiny Object Detection Competitive Multi-Layer Neural Network) architecture with three sub components first competitive multi-layer network, second TOD auxiliary and third multi-level continue features aggregation for accurately detecting small objects. Competitive learning for object detection is the basis of the proposed architecture. Comparison study with existing RCNN, Fast RCNN, Faster RCNN, SSD and YOLO shows significant improvement in the results. TOD-CMLNN receives 72.46 % accuracy in terms of mAP which is impressive as compared to state-of-the-art detectors.

High-dimensional multi-objective optimization algorithm for combustion chamber of aero-engine based on artificial neural network-multi-objective particle swarm optimization

This paper considers optimizing the performance of high-temperature combustion chamber of an aero-engine based on a concentric and hierarchical model. First, sample data for the design variables are obtained based on Latin hypercube sampling method, and a one-dimensional program is used to obtain the true values of combustion efficiency and total pressure loss corresponding to each group of variables. The obtained data are then pre-processed to establish a dataset. Second, a multi-layer artificial neural network (ANN) architecture is designed and a surrogate model of the combustion-related performance of the combustor is established using a data-driven method. The results of global sensitivity analysis based on variance show that ratio of fuel flow to air flow (fuel–air ratio) and the total inlet pressure are the most important factors influencing the two objective functions. Finally, we optimize the multi-objective combustion-related performance of the surrogate model by applying the particle swarm optimization algorithm to it. The results of experiments show that the ANN-based model could accurately predict the efficiency of combustion and total pressure loss of the chamber, yielding root mean-squared errors of 0.0107 and 0.3032%, respectively. It also had better generalization ability than the cubic polynomial surrogate model. Compared with the cubic polynomial model, it generated an optimal Pareto solution set as prediction that had higher values in both objective functions. The proposed model might require better data that can be obtained using intelligent sampling methods so that deeper neural networks can be designed to reduce error and improve its optimization design.

Artificial intelligence applied in neoantigen identification facilitates personalized cancer immunotherapy.

The field of cancer neoantigen investigation has developed swiftly in the past decade. Predicting novel and true neoantigens derived from large multi-omics data became difficult but critical challenges. The rise of Artificial Intelligence (AI) or Machine Learning (ML) in biomedicine application has brought benefits to strengthen the current computational pipeline for neoantigen prediction. ML algorithms offer powerful tools to recognize the multidimensional nature of the omics data and therefore extract the key neoantigen features enabling a successful discovery of new neoantigens. The present review aims to outline the significant technology progress of machine learning approaches, especially the newly deep learning tools and pipelines, that were recently applied in neoantigen prediction. In this review article, we summarize the current state-of-the-art tools developed to predict neoantigens. The standard workflow includes calling genetic variants in paired tumor and blood samples, and rating the binding affinity between mutated peptide, MHC (I and II) and T cell receptor (TCR), followed by characterizing the immunogenicity of tumor epitopes. More specifically, we highlight the outstanding feature extraction tools and multi-layer neural network architectures in typical ML models. It is noted that more integrated neoantigen-predicting pipelines are constructed with hybrid or combined ML algorithms instead of conventional machine learning models. In addition, the trends and challenges in further optimizing and integrating the existing pipelines are discussed.

Data Augmentation of Micrographs and Prediction of Impact Toughness for Cast Austenitic Steel by Machine Learning

The Material Genome Initiative has been driven by high-throughput calculations, experiments, characterizations, and machine learning, which has accelerated the efficiency of the discovery of novel materials. However, the precise quantification of the material microstructure features and the construction of microstructure–property models are still challenging in optimizing the performance of materials. In this study, we proposed a new model based on machine learning to enhance the power of the data augmentation of the micrographs and construct a microstructure–property linkage for cast austenitic steels. The developed model consists of two modules: the data augmentation module and microstructure–property linkage module. The data augmentation module used a multi-layer convolution neural network architecture with diverse size filter to extract the microstructure features from irregular micrographs and generate new augmented microstructure images. The microstructure–property linkage module used a modified VGG model to establish the relationship between the microstructure and material property. Taking cast austenitic stainless steels after solution treating in different temperatures as an example, the results showed that the prediction accuracy of the developed machine learning model had been improved. The coefficient R2 of the model was 0.965, and the medians were only ±2 J different with the measured impact toughness.

An Optimized Multi-layer Spiking Neural Network implementation in FPGA Without Multipliers

This paper presents an expansion and evaluation of the hardware architecture for the Optimized Deep Event-driven Spiking Neural Network Architecture (ODESA). ODESA is a state-of-the-art, event-driven multi-layer Spiking Neural Network (SNN) architecture that offers an end-to-end, online, and local supervised training method. In previous work, ODESA was successfully implemented on Field-Programmable Gate Array (FPGA) hardware, showcasing its effectiveness in resource-constrained hardware environments. Building upon the previous implementation, this research focuses on optimizing the ODESA network hardware by introducing a novel approach. Specifically, we propose substituting the dot product multipliers in the Neurons with a low-cost shift-register design. This optimization strategy significantly reduces the hardware resources required for implementing a neuron, thereby enabling more complex SNNs to be accommodated within a single FPGA. Additionally, this optimization results in a reduction in power consumption, further enhancing the practicality and efficiency of the hardware implementation. To evaluate the effectiveness of the proposed optimization, extensive experiments and measurements were conducted. The results demonstrate the successful reduction in hardware resource utilization while maintaining the network's functionality and performance. Moreover, the power consumption reduction contributes to the overall energy efficiency of the hardware implementation.

Multilayer Neural Network Architecture for Predicting Stress/Anxiety in Modern Life: A Novel Approach using Facial Action Units

Multilayer Neural Network Architecture for Predicting Stress/Anxiety in Modern Life: A Novel Approach using Facial Action Units

Benchmarking a multi-layer approach and neural network architectures for defect detection in PBF-LB/M

The substitution of expensive non-destructive material testing by data-based process monitoring is intensively explored in quality assurance for additive manufactured components. Machine learning show promising results for defect detection but require conceptual adaption to layer wise manufacturing and line scanning patterns in laser powder bed fusion. A multi-layer approach to co-register µ-computer tomography measurements with process monitoring data is developed and a workflow for automatic data set generation is implemented. The objective of this research is to benchmark the volumetric multi-layer approach and specifically selected deep learning methods for defect detection. The volumetric approach shows superior results compared to single slice monitoring. All investigated structured neural network topologies deliver similar performance. • Machine learning identifies correlations between µ-CT porosity scans and melt pool monitoring • 3D-multi-layer-based machine learning outperforms single layer approach in defect detection • Saturation of detection performance is observed beyond optimal number of layers • Results hold independently for all investigated neural network topologies

Complementary Memtransistor-Based Multilayer Neural Networks for Online Supervised Learning Through (Anti-)Spike-Timing-Dependent Plasticity.

We propose a complete hardware-based architecture of multilayer neural networks (MNNs), including electronic synapses, neurons, and periphery circuitry to implement supervised learning (SL) algorithm of extended remote supervised method (ReSuMe). In this system, complementary (a pair of n- and p-type) memtransistors (C-MTs) are used as an electrical synapse. By applying the learning rule of spike-timing-dependent plasticity (STDP) to the memtransistor connecting presynaptic neuron to the output one whereas the contrary anti-STDP rule to the other memtransistor connecting presynaptic neuron to the teacher one, extended ReSuMe with multiple layers is realized without the usage of those complicated supervising modules in previous approaches. In this way, both the C-MT-based chip area and power consumption of the learning circuit for weight updating operation are drastically decreased comparing with the conventional single memtransistor (S-MT)-based designs. Two typical benchmarks, the linearly nonseparable benchmark XOR problem and Mixed National Institute of Standards and Technology database (MNIST) recognition have been successfully tackled using the proposed MNN system while impact of the nonideal factors of realistic devices has been evaluated.

An unfeasibility view of neural network learning

We define the notion of a continuously differentiable perfect learning algorithm for multilayer neural network architectures and show that such algorithms do not exist provided that the length of the data set exceeds the number of involved parameters and the activation functions are logistic, tanh or sin.

Double Moving Average Control Chart for Autocorrelated Data

The assumption of normality and independence is necessary for statistical inference of control charts. Misleading results are obtained if the traditional control chart technique is applied on the auto-correlated data. When data is correlated, a time series model is employed to produce an optimum output. The objective is to create a new control chart methodology that takes the autocorrelation of observations into account. Charts of Moving Average, Exponentially Weighted and Cumulative Sum better perform in the existence of autocorrelation data for small and moderate changes. The proposed methodology is highly skilled and receptive to minor, moderate and major changes in the process. Propsed DMA chart increases efficiency of average run length (ARL) chart for moving average (MA) to detect the small to medium magnitude shifts in the mean. The simulation also demonstrates that the DMA chart with spans of w=10 and 15 generally performs well in terms of average run length (ARL) as compared to clasical MA. This research may be extended to a multivariate autocorrelated statistical process control, but it can also be used to recognise and categorise seven categories of traditional control chart patterns, such as Downward, Upward Shift, Normal Trend, Cyclic, Systematic patterns, Increasing and Decresing Trend. In order to identify and categorize a set of subclasses of abnormal patterns, this model (multivariate autocorrelated statistical process control chart) should employ a multilayer feed forward Artificial Neural Network (ANN) architecture controlled by a back-propagation learning rule.