Multilayer Perceptron Architecture Research Articles

A comparative study of KAN transformer and traditional vision transformer for ultrasound image-based diagnosis

Abstract. In recent years, the MLP architecture has almost been monopolized in the field of deep learning, and its success is undeniable, but at the same time there are some problems. Kolmogorov-Arnold Network (KAN) is a new neural network architecture based on Kolmogorov-Arnold theory implementation. Compared to traditional MLPs, KANs have higher interpretability, faster training, and more efficient usability. In this paper, based on the theory of KAN, the author try to replace the Multi-Layer Perceptron (MLP) architecture in Vision Transformer (ViT) with the better performing KAN,author conducted scoring classification experiments on clinical acute respiratory distress syndrome (ARDS) and pneumonia image datasets provided by the Emergency Department of Changzheng Hospital in Shanghai, China, in order to validate the feasibility of the application of KAN+ViT in assisting clinical medical ultrasound diagnosis as well as its high efficiency in comparison with the traditional MLP+ViT structure. And classification experiments on the Cifar-10 dataset are used to validate the superiority of this new architecture over the traditional ViT architecture.

PPCon 1.0: Biogeochemical-Argo profile prediction with 1D convolutional networks

Abstract. Effective observation of the ocean is vital for studying and assessing the state and evolution of the marine ecosystem and for evaluating the impact of human activities. However, obtaining comprehensive oceanic measurements across temporal and spatial scales and for different biogeochemical variables remains challenging. Autonomous oceanographic instruments, such as Biogeochemical (BGC)-Argo profiling floats, have helped expand our ability to obtain subsurface and deep-ocean measurements, but measuring biogeochemical variables, such as nutrient concentration, still remains more demanding and expensive than measuring physical variables. Therefore, developing methods to estimate marine biogeochemical variables from high-frequency measurements is very much needed. Current neural network (NN) models developed for this task are based on a multilayer perceptron (MLP) architecture, trained over point-wise pairs of input–output features. Although MLPs can produce smooth outputs if the inputs change smoothly, convolutional neural networks (CNNs) are inherently designed to handle profile data effectively. In this study, we present a novel one-dimensional (1D) CNN model to predict profiles leveraging the typical shape of vertical profiles of a variable as a prior constraint during training. In particular, the Predict Profiles Convolutional (PPCon) model predicts nitrate, chlorophyll, and backscattering (bbp700) starting from the date and geolocation and from temperature, salinity, and oxygen profiles. Its effectiveness is demonstrated using a robust BGC-Argo dataset collected in the Mediterranean Sea for training and validation. Results, which include quantitative metrics and visual representations, prove the capability of PPCon to produce smooth and accurate profile predictions improving upon previous MLP applications.

Анализ угроз информационной безопасности органов государственной власти с помощью нейронных сетей

<p>Ensuring the information security of public authorities requires the use of specialized tools that take into account various sources of information threats and their constant changes. This study aims to develop a methodology for analyzing these threats using neural networks. The research uses methods such as machine learning and neural network analysis to systematize the data. The authors adapted the Multi-Layer Perceptron (MLP) architecture and configured the hyperparameters of the neural network to achieve their objectives. The neural network was trained using the Python programming language, and its effectiveness was evaluated using metrics such as accuracy, precision, recall, and F1 score. The results of the study included the development of a method for creating a data set that encompasses assessments of threats to the information security of various public authorities and their sources. Additionally, the study evaluated the effectiveness of neural networks in solving classification problems for public authorities. Finally, the study interpreted the results of neural network analysis to determine the resistance of public authorities against information security threats.</p>

NSEMBLE FINE TUNED MULTI LAYER PERCEPTRON FOR PREDICTIVE ANALYSIS OF WEATHER PATTERNS AND RAINFALL FORECASTING FROM SATELLITE DATA

The accurate prediction of weather patterns and rainfall forecasting is critical for various sectors, including agriculture, disaster management, and water resource planning. Traditional models often struggle to capture the complex interactions between atmospheric variables, particularly when integrating diverse types of satellite data (binary, categorical, and numerical). To address this challenge, an ensemble fine-tuned multi-layer perceptron (MLP) model is developed, combining the strengths of multiple machine learning techniques for more robust predictions. The primary problem is the difficulty in handling mixed data types while maintaining high prediction accuracy. Satellite data, including binary indicators (e.g., cloud presence), categorical features (e.g., cloud types), and numerical variables (e.g., temperature, humidity, and wind speed), provide rich information but require specialized processing for effective forecasting. The proposed method involves fine-tuning an ensemble of MLP models with backpropagation, dropout regularization, and batch normalization to reduce overfitting and enhance generalization. The ensemble integrates predictions from individual MLP models, each trained on different subsets of features (binary, categorical, numerical). This technique allows the model to leverage complementary strengths and produce more accurate rainfall forecasts. Satellite data is preprocessed and normalized before training, and categorical variables are one-hot encoded to ensure compatibility with the MLP architecture. Results from testing on historical satellite weather datasets demonstrate significant improvements in forecast accuracy. The ensemble MLP achieved an accuracy of 91.3%, with a precision of 90.7%, recall of 89.5%, and an F1-score of 90.1%. The model performed exceptionally well in identifying critical rainfall events, reducing false positives by 12% compared to traditional models.

Artificial Neural Networks for Classification Tasks: A Systematic Literature Review

Artificial neural networks (ANNs) have become indispensable tools for solving classification tasks across various domains. This systematic literature review explores the landscape of ANN utilization in classification, addressing three key research questions: the types of architectures employed, their accuracy, and the data utilized. The review encompasses 30 studies published between 2019 and 2024, revealing Convolutional Neural Networks (CNNs) as the predominant architecture in image-related tasks, followed by Multilayer Perceptron (MLP) architectures for general classification tasks. Feed Forward Neural Networks (FFNN) exhibited the highest average accuracy with a 97.12%, with specific studies achieving exceptional results across diverse classification tasks. Moreover, the review identifies digitized images as a commonly utilized data source, reflecting the broad applicability of ANNs in tasks such as medical diagnosis and remote sensing. The findings underscore the importance of machine learning approaches, highlight the robustness of ANNs in achieving high accuracy, and suggest avenues for future research to enhance interpretability, efficiency, and generalization capabilities, as well as address challenges related to data quality.

CGAformer: Multi-scale feature Transformer with MLP architecture for short-term photovoltaic power forecasting

Accurately predicting the output power of photovoltaic (PV) systems is an effective means to ensure the reliable and economical operation of grid-connected PV systems. Aiming at the characteristics of PV power generation such as strong volatility, high intermittency and obvious periodicity, a hybrid model named CGAformer based on One-Dimensional Convolutional Neural Networks (CNN1D), Global Additive Attention (GADAttention), and Auto-Correlation is proposed for short-term PV power generation prediction. The model uses CNN1D to extract local features and obtains global weights by improving the GADAttetion obtained by additive attention. Auto-Correlation integrates local features and global weights and identifies repeated patterns in the sequence to obtain highly coupled multi-scale features, and finally generates the final prediction results through Multilayer Perceptron (MLP). In order to verify the effectiveness of the model, this paper uses a historical dataset from a PV system located in Uluru, Australia for sufficient experiments. In the comparative experiments, The overall average RMSE and MAE of CGAformer are improved by 6.82% and 20.46% respectively compared with long short-term memory (LSTM). In addition, ablation experiments and seasonal analysis are used to verify the effectiveness of the model and its excellent generalization ability for different seasons.

Predictive modelling of cooling consumption in nursing homes using artificial neural networks: Implications for energy efficiency and thermal comfort

The growing need for cooling within the built environment, propelled by climate change and the expansion of nursing homes due to the increase in life expectancy, highlights the urgency of implementing energy-efficient strategies in buildings occupied by older populations. As of today, there remains a need for comprehensive research into the influence of indoor and outdoor conditions, building, operational, and occupant characteristics, on energy consumption specifically for nursing homes. This study develops a systemic artificial neural network-based model with a multi-layer perceptron architecture to assess HVAC energy implications during the cooling season for older populations. Using monitored data from eight nursing homes, the model includes cooling area, construction age, outdoor and indoor temperatures, and outdoor relative humidity as inputs, and cooling consumption as the output. Results show excellent predictive capability (R2=0.95), with mean error of −0.5 kWh, root mean squared error of 13.7 kWh, mean absolute error of 10.2 kWh, and relative error of 0.051. These outcomes are better compared to linear models (R2≈0.65) under the same data set. Adjusting operative temperatures adaptively can significantly enhance resident comfort and achieve up to 23.4 % energy savings, particularly in hotter, drier climates. These findings are of paramount importance for effective energy management in buildings.

Intelligent computing for the electro-osmotically modulated peristaltic pumping of blood-based nanofluid

The study delves into a novel application of artificial neural networks within the biomedical realm, specifically focusing on the peristaltic pumping of ionized blood-infused nanofluid using the Casson model. The investigation takes into account the impacts of electrokinetic forces, chemical reaction, thermal radiation and variable thermal conductivity. The endorsed nanofluid model encompasses both thermophoretic and Brownian diffusions. The Nernst-Planck and Poisson equations are used to characterize electroosmotic process. The mathematical model is then numerically solved using NDSolve in Mathematica. The variation in relevant hemodynamic characteristics concerning key flow parameters is explored through numerous graphical representations. Two distinct artificial neural networks have been developed to estimate values for rates of mass and heat transfer. The Levenberg-Marquardt algorithm is used as the training algorithm in network models with multi-layer perceptron architecture. Graphical inspection of the results demonstrates a decrease in blood flow with an elevated Helmholtz-Smoluchowski velocity. A lower pressure gradient is recorded with a higher electroosmotic parameter. Furthermore, an augmentation in the thermal conductivity parameter leads to a reduction in heat transfer rate at the wall. The concentration profile demonstrates an increase with a rise in the chemical reaction parameter. The results obtained from the neural network models exhibit an ideal harmony with the target values. The developed neural network models demonstrated the ability to predict rates of heat and mass transfer values with average deviations of −0.01% and 0.004%, respectively.

Advancing water disinfection strategies: assessing disinfection efficiency with a Bayesian Regularized artificial neural network model

ABSTRACT It is important to find the estimation methodology with the highest accuracy in order to determine the parameters of water disinfection and to provide the most ideal disinfection. In this study, the usability of artificial neural networks in predicting response disinfection efficiency in electrochemical water disinfection processes was investigated. An artificial neural network model was developed using a total of 17 data sets and Response Disinfection Efficiency values were estimated from the model. Current density, treatment time and interelectrode spacing values are defined as input parameters in the network model, which has a multilayer perceptron architecture with 10 neurons in its hidden layer. The coefficient of determination value for the developed model was 0.98682 and the average deviation rate was −0.1%. The study findings showed that neural networks are an ideal tool that can be used to predict response disinfection efficiency in electrochemical water disinfection processes.

PGNM: Using Physics-Informed Gated Recurrent Units Network Method to capture the dynamic data feature propagation process of PDEs

The multi-layer perceptron architecture in PINNs model severely limits the model’s ability to learn the temporal evolution of equation features. Instead, the GRU network is capable of capturing these complex temporal correlation features. This paper replaces the multi-layer perceptron structure of the PINNs model with the GRU network and called the modified model as physics-informed gated recurrent units network method (PGNM model). The PGNM model exhibits enhanced performance in assimilating insights from historical data and providing accurate predictions for future data. This paper compares the predictive performance of the proposed PGNM model and the classical PINNs model using L2 error and mean squared error (MSE) as metrics. Additionally, it evaluates how altering various parameter settings, such as the number of neurons, hidden layers and iterations, affects the predictive capabilities of both models. In conclusion, PGNM model shows significant improvement in prediction accuracy compared to PINNs model. Furthermore, PGNM model achieves better long-range prediction results than PINNs model when the training data does not include samples from the time period to be predicted.

Modelling and Output Power Estimation of a Combined Gas Plant and a Combined Cycle Plant Using an Artificial Neural Network Approach

Artificial neural networks (ANNs) have gained prominence among contemporary computing techniques due to their capacity to handle complicated stochastic datasets and nonlinear modelling in combined gas and combined cycle power (COGAS) plants. Researchers, academicians, and stakeholders have been unable to predict, ensure effective operation, and prevent power outages in COGAS due to the nonlinearity. The first implementation of the simultaneous adoption of three types of ANNs using Levenberg–Marquardt (LM), Bayesian regularisation (BR), and scaled conjugate gradient (SCG) configurations for training and assessing a combined cycle power plant output is presented. The dataset used in this research is a 9568-unit full combined cycle power plant basis load dataset, accessible through the public UCI Machine Learning Repository. It incorporates ambient temperature, exhaust vacuum, ambient pressure, and relative humidity as input parameters to predict the electric output power. The most accurate and dependable electric power predictions could be identified for 70% of the total data, of which 6698 were trained, 15% were tested, and 15% were validated (2870). By using the three training techniques, namely, LM, BR, and SCG, the parameterized networks are studied, increasing the number of hidden layers from 20 to 500. The lowest root-mean-square error value for a multilayer perceptron (MLP) architecture is 3.631%, which is lower than the values of 4.17%, 4.35%, and 4.63% for comparable MLP structures (20 to 500), documented in the literature. The LM and BR algorithms outperform SCG. These adopted algorithms could be a cutting-edge application in the power plant industry and other real-world applications for reliable solutions, to satisfy emerging societal needs with environmental benefits.

Molding robust S-box design based on linear fractional transformation and multilayer Perceptron: Applications to multimedia security

This study introduces a novel and refined approach for generating exceptionally efficient S-boxes. The proposed methodology employs a hybrid approach that combines linear fractional transformation (LFT) with a multilayer perceptron (MLP) architecture. This method makes use of a perceptron with three layers: input, hidden, and output. Each layer's neuron count is fine-tuned to conform to the S-box layout. In addition, a threshold nonlinear transformation is utilized to increase nonlinearity, and a novel algorithm for boosting nonlinearity is introduced. The utilization of both LFT and MLP approaches has led to the development of S-boxes that possess nearly ideal average nonlinearity values, surpassing those that have been presented in literature. Notably, one S-box achieved an exceptional nonlinearity score of 114.50. Furthermore, to demonstrate how well the S-box works, this study also employs it in an image encryption application.

Antimicrobial Activity Classification of Imidazolium Derivatives Predicted by Artificial Neural Networks

PurposeThis study assesses the Multilayer Perceptron (MLP) neural network, complemented by other Machine Learning techniques (CART, PCA), in predicting the antimicrobial activity of 140 newly designed imidazolium chlorides against Klebsiella pneumoniae before synthesis. Emphasis is on leveraging molecular properties for predictive analysis.MethodsClassification and regression decision trees (CART) identified the top 200 predictive molecular descriptors. Principal Component Analysis (PCA) reduced these descriptors to 5 components, retaining 99.57% of raw data information. Antimicrobial activity, categorized as high or low, was based on experimentally proven minimal inhibitory concentration (MIC), with a cut-point at MIC = 0.856 mol/L. A 12-fold cross-validation trained the MLP (architecture 5-12-2 with 5 Principal Components).ResultsThe MLP exhibited commendable performance, achieving almost 90% correct classifications across learning, validation, and test sets, outperforming models without PCA dimension reduction. Key metrics, including accuracy (0.907), sensitivity (0.905), specificity (0.909), and precision (0.891), were notably high. These results highlight the MLP model's efficacy with PCA as a high-quality classifier for determining antimicrobial activity.ConclusionsThe study concludes that the MLP neural network, along with CART and PCA, is a robust tool for predicting the antimicrobial activity class of imidazolium chlorides against Klebsiella pneumoniae. CART and PCA, used in this study, allowed input variable reduction without significant information loss. High classification accuracy and associated metrics affirm the method’s potential utility in pre-synthesis assessments, offering valuable insights for antimicrobial compound design.

MEAformer: An all-MLP transformer with temporal external attention for long-term time series forecasting

Transformer-based models have significantly improved performance in Long-term Time Series Forecasting (LTSF). These models employ various self-attention mechanisms to discover long-term dependencies. However, the computational efficiency is hampered by the inherent permutation invariance of self-attention, and they primarily focus on relationships within the sequence while neglecting potential relationships between different sample sequences. This limits the ability and flexibility of self-attention in LTSF. In addition, the Transformer's decoder outputs sequences in an autoregressive manner, leading to slow inference speed and error accumulation effects, especially for LTSF. Regarding the issues with Transformer-based models for LTSF, we propose a model better suited for LTSF, named MEAformer. MEAformer adopts a fully connected Multi-Layer Perceptron (MLP) architecture consisting of two types of layers: encoder layers and MLP layers. Unlike most encoder layers in Transformer-based models, the MEAformer replaces self-attention with temporal external attention. Temporal external attention explores potential relationships between different sample sequences in the training dataset. Compared to the quadratic complexity of self-attention mechanisms, temporal external attention has efficient linear complexity. Encoder layers can be stacked multiple times to capture time-dependent relationships at different scales. Furthermore, the MEAformer replaces the intricate decoder layers of the original model with more straightforward MLP layers. This modification aims to enhance inference speed and facilitate single-pass sequence generation, effectively mitigating the problem of error accumulation effects. Regarding long-term forecasting, MEAformer achieves state-of-the-art performance on six benchmark datasets, covering five real-world domains: energy, transportation, economy, weather, and disease. Code is available at: https://github.com/huangsiyuan924/MEAformer.

Graph Node Classification to Predict Autism Risk in Genes.

This study explores the genetic risk associations with autism spectrum disorder (ASD) using graph neural networks (GNNs), leveraging the Sfari dataset and protein interaction network (PIN) data. We built a gene network with genes as nodes, chromosome band location as node features, and gene interactions as edges. Graph models were employed to classify the autism risk associated with newly introduced genes (test set). Three classification tasks were undertaken to test the ability of our models: binary risk association, multi-class risk association, and syndromic gene association. We tested graph convolutional networks, Graph Sage, graph transformer, and Multi-Layer Perceptron (Baseline) architectures on this problem. The Graph Sage model consistently outperformed the other models, showcasing its utility in classifying ASD-related genes. Our ablation studies show that the chromosome band location and protein interactions contain useful information for this problem. The models achieved 85.80% accuracy on the binary risk classification, 81.68% accuracy on the multi-class risk classification, and 90.22% on the syndromic classification.

Robust Learning from Demonstration Based on GANs and Affine Transformation

Collaborative robots face barriers to widespread adoption due to the complexity of programming them to achieve human-like movement. Learning from demonstration (LfD) has emerged as a crucial solution, allowing robots to learn tasks directly from expert demonstrations, offering versatility and an intuitive programming approach. However, many existing LfD methods encounter issues such as convergence failure and lack of generalization ability. In this paper, we propose: (1) a generative adversarial network (GAN)-based model with multilayer perceptron (MLP) architecture, coupled with a novel loss function designed to mitigate convergence issues; (2) an affine transformation-based generalization method aimed at enhancing LfD tasks by improving their generalization performance; (3) a data preprocessing method tailored to facilitate deployment on robotics platforms. We conduct experiments on a UR5 robotic platform tasked with handwritten digit recognition. Our results demonstrate that our proposed method significantly accelerates generation speed, achieving a remarkable processing time of 23 ms, which is five times faster than movement primitives (MPs), while preserving key features from demonstrations. This leads to outstanding convergence and generalization performance.

Power Attack Vulnerability Assessment of Circuit-Level PRESENT Encryption IP Using Artificial Intelligence Mechanisms

Artificial Intelligence (AI) schemes eliminate the need for intellectual human knowledge of crypto algorithms to facilitate side-channel attacks on security implementations. Side channel attack analysis for circuit-level hardware (VLSI) implementations of symmetric-key block ciphers through artificial machine models are yet to be addressed. The proposed design of block cipher architecture, which is the implementation under attack features, secure adiabatic logic style the Charge Balancing Symmetric Pre-resolve Adiabatic Logic (CBSPAL). This style exhibits uniform power consumption through its inherent circuit arrangement, which makes it robust toward side-channel power attacks. This paper considers several Multilayer Perceptron (MLP) architectures to mount non-profiled side-channel attacks on round 1- PRESENT circuit-level implementation. Supervised learning of the MLP models through training with power waveforms employs binary LSB labeling. An attack on the custom dataset with 50,000 supply power traces collected from the CBSPAL implementation fails to retrieve secret information in any of the 8 bytes. The traditional Correlation Power Attack (CPA) was also not thriving with the proposed implementation. Comparison is made through a similar attack approach on benchmark AES (ASCAD dataset), which distinguishes the secret information throughout all the parameters of interest: normalized accuracy, normalized NMM accuracy and rank plots on byte 3.

Learning Deep Embedding with Acoustic and Phoneme Features for Speaker Recognition in FM Broadcasting

Text-independent speaker verification (TI-SV) is a crucial task in speaker recognition, as it involves verifying an individual’s claimed identity from speech of arbitrary content without any human intervention. The target for TI-SV is to design a discriminative network to learn deep speaker embedding for speaker idiosyncrasy. In this paper, we propose a deep speaker embedding learning approach of a hybrid deep neural network (DNN) for TI-SV in FM broadcasting. Not only acoustic features are utilized, but also phoneme features are introduced as prior knowledge to collectively learn deep speaker embedding. The hybrid DNN consists of a convolutional neural network architecture for generating acoustic features and a multilayer perceptron architecture for extracting phoneme features sequentially, which represent significant pronunciation attributes. The extracted acoustic and phoneme features are concatenated to form deep embedding descriptors for speaker identity. The hybrid DNN demonstrates not only the complementarity between acoustic and phoneme features but also the temporality of phoneme features in a sequence. Our experiments show that the hybrid DNN outperforms existing methods and delivers a remarkable performance in FM broadcasting TI-SV.

Video-MLP: Convolution-free, attention-free architecture for video classification

Convolutional neural networks (CNNs) and Transformer architectures have traditionally been recognized as the preferred models for addressing computer vision tasks. However, there has been a recent surge in the popularity of networks based on multi-layer perceptron (MLP) structures that do not rely on convolution or attention mechanisms. These MLP architectures have demonstrated exceptional performance in image classification tasks, exhibiting lower time complexity while maintaining high accuracy. In contrast, video classification tasks involve larger amounts of data and necessitate more intricate feature extraction, resulting in increased time and resource consumption. To enhance computational efficiency and minimize resource utilization, we propose a convolution-free and Transformer-free architecture for video classification called Video-MLP for video classification. Video-MLP utilizes a simple MLP structure to learn video features. Specifically, it comprises two types of layers: Spatial-Mixer and Temporal-Mixer, which respectively capture spatial and temporal information. The Spatial-Mixer extracts spatial information from each frame along the height and width dimensions, while the Temporal-Mixer models temporal information for the same spatial positions across frames. To improve the efficiency of spatial-temporal modeling in our network, we used a spatial-temporal information fusion approach to integrate information at different scales. Additionally, we grouped the input data along the time dimension and designed three different grouping schemes when extracting temporal information. The experimental results indicate that Video-MLP achieved accuracy rates of 87.2% on the Kinetics-400 dataset and 75.3% on the SomethingV2 dataset, outperforming models with equivalent computational complexity. Notably, Video-MLP achieved these results without using convolution and attention mechanisms, and without pre-training on large-scale image and video datasets.

S[formula omitted]WaveNet: A novel spectral–spatial wave network for hyperspectral image classification

Deep learning has made significant progress in hyperspectral image (HSI) classification, and its powerful ability to automatically learn abstract features is well recognized. Recently, the simple architecture of multi-layer perceptron (MLP) has been extensively employed to extract long-range dependencies of HSI and achieved impressive results. However, existing MLP-based models exhibit insufficient representation of spectral–spatial information in HSI and generally aggregate features with fixed weights, which limits their ability to capture semantic differences. To tackle these challenges, this paper proposes a novel spectral–spatial wave network (S2WaveNet) for HSI classification tasks to enhance the representation capability of spectral–spatial features in ground objects. Specifically, the spectral–spatial wave mixer (S2WaveMixer) block is designed as a key component to represent each HSI input as a wave function with amplitude and phase parts. Thus, it enables a deeper dynamic perception and facilitates the extraction of spectral–spatial feature variations of ground objects. The amplitude represents the original features and the phase term is a complex value changing based on the semantic contents of the input images. Furthermore, the inception unit is introduced into the S2WaveMixer block to consider spectral–spatial information at multiple granularity levels. Experiments conducted on five public datasets demonstrate the superiority of S2WaveNet in classification performance and generalization compared to competitors.