Standard Multilayer Perceptron Research Articles

Synergizing frequency domain and texture-aware implicit module for MRI arbitrary-scale super-resolution

Arbitrary-scale MR image super-resolution (SR) methods have emerged as a prominent research topic in medical imaging due to their ability to address the demand for continuous resolution in medical image analysis and diagnostics. Although several methods currently employ Implicit Neural Representations (INR) to achieve arbitrary-scale SR for MR images, they still face the following issues that constrain further enhancements in SR performance. (1) They solely perform feature extraction on MR images in the image domain, disregarding the frequency domain information in MR images, which results in aliasing artifacts and blurriness in the reconstructed SR images. (2) They utilize a standard multi-layer perceptron in INR, which tend to learn global structures (i.e., low-frequency components) and fail to capture and restore detailed texture information (i.e., high-frequency components) in MR images adequately. To address these issues, we propose a brand-new method called FTNet, which introduces the frequency domain information and employs the texture-aware implicit module for MR image arbitrary-scale SR. Specifically, we design an additional branch to extract the feature of MR images in the frequency domain to maximize removing aliasing artifacts. To further improve the quality of SR images, we introduce a texture extraction branch and a texture-aware implicit module to capture and restore detailed texture information within the MR images. Extensive experiments demonstrate the superiority of our network over state-of-the-art methods and its great potential for clinical practice.

Enhancing Skin Cancer Diagnosis Using Swin Transformer with Hybrid Shifted Window-Based Multi-head Self-attention and SwiGLU-Based MLP.

Skin cancer is one of the most frequently occurring cancers worldwide, and early detection is crucial for effective treatment. Dermatologists often face challenges such as heavy data demands, potential human errors, and strict time limits, which can negatively affect diagnostic outcomes. Deep learning-based diagnostic systems offer quick, accurate testing and enhanced research capabilities, providing significant support to dermatologists. In this study, we enhanced the Swin Transformer architecture by implementing the hybrid shifted window-based multi-head self-attention (HSW-MSA) in place of the conventional shifted window-based multi-head self-attention (SW-MSA). This adjustment enables the model to more efficiently process areas of skin cancer overlap, capture finer details, and manage long-range dependencies, while maintaining memory usage and computational efficiency during training. Additionally, the study replaces the standard multi-layer perceptron (MLP) in the Swin Transformer with a SwiGLU-based MLP, an upgraded version of the gated linear unit (GLU) module, to achieve higher accuracy, faster training speeds, and better parameter efficiency. The modified Swin model-base was evaluated using the publicly accessible ISIC 2019 skin dataset with eight classes and was compared against popular convolutional neural networks (CNNs) and cutting-edge vision transformer (ViT) models. In an exhaustive assessment on the unseen test dataset, the proposed Swin-Base model demonstrated exceptional performance, achieving an accuracy of 89.36%, a recall of 85.13%, a precision of 88.22%, and an F1-score of 86.65%, surpassing all previously reported research and deep learning models documented in the literature.

The Influence of Organizational Factors on the School’s Achievements

The purpose of this paper is to examine the influence of the most significant organizational factors on primary school achievements, with a note that the achievement of the school is expressed through the quality of outcome knowledge. The research was conducted on a sample of 460 employees, from 21 primary schools, of which 439 are professionals and 21 are principals in the South Bačka County, Republic of Serbia. The collected data were processed using the Gretl software and AMOS for modelling structural equations. More precisely, the research is based on the application of exploratory and confirmatory factor analysis. A neural network based on a standard multilayer perceptron model was used in the paper to test the validity of the obtained results of the AMOS model. The results of the research show that school management is the most important factor in school achievement and that this influence is most pronounced through teaching staff and school infrastructure. The results, also show that teachers’ competencies have the strongest direct influence on the quality of outcome knowledge. The results obtained indicate that decision-makers and creators of social policies must pay special attention to the selection of school principals as well as their professional education, while school principals to the selection of teachers. Future researchers are recommended to use the Sobel test to precisely determine the indirect influences of school management on school achievement.

Deep learning to develop zero-equation based turbulence model for CFD simulations of the built environment

This study aims to improve the accuracy and speed of predictions for thermal comfort and air quality in built environments by creating a coupled framework between computational fluid dynamics (CFD) simulations and deep learning models. The coupling approach is showcased by the development of a data-driven turbulence model. The new turbulence model is built using a deep learning neural network, whose mapping structure is based on a zero-equation turbulence model for built environment simulations, and is coupled with the CFD software OpenFOAM to create a hybrid framework. The neural network is a standard shallow multi-layer perceptron. The number of hidden layers and nodes per layer was optimized using Bayesan optimization algorithm. The framework is trained on an indoor environment case study, as well as tested on an indoor office simulation and an outdoor building array simulation. Results show that the deep learning based turbulence model is more robust and faster than traditional two-equation Reynolds average Navier-Stokes (RANS) turbulence models, while maintaining a similar level of accuracy. The model also outperforms the standard algebraic zero-equation model due to its superior ability to generalize to various flow scenarios. Despite some challenges, namely the mapping constraint, the limited training dataset size and the source of generation of training data, the hybrid framework demonstrates the viability of the coupling technique and serves as a starting point for future development of more reliable and advanced models.

Transmission line impulse response modelling using machine learning techniques

<p>Conventional methods of circuit simulation such as full-wave electromagnetic fieldsolvers can be very slow. Machine learning is an emerging technology in modelling, simulation, optimization, and design that present attractive alternatives to the conventional methodologies because they can be trained with a small amount of data, and then used to perform fast circuit predictions within the same design space. In this paper, we present applications of machine learning techniques for the modelling of transmission lines from their impulse reponses. The standard multilayer perceptron (MLP) neural network and the gaussian process (GP) regression techniques are demonstrated, and<br />both models are successfully implemented to model the impulse responses of transmission lines with great accuracies. We show that the GP outperforms the MLP in terms of prediction accuracies and that the GP is more data efficient than the MLP. This is beneficial considering that each training sample is expensive, making the GP a good candidate for the task, compared to the more popular MLP.</p>

Dynamically evolving deep neural networks with continuous online learning

In a big data environment, data streams are sequences of dynamically changing data with unlimited length; they are often associated with concept drift, caused by data distribution transition. The uncertainty of data streams challenges the prediction ability of traditional static algorithms and hampers their practical application. Although many deep neural network models can be tailored to perform in an online manner, they are not suitable for non-stationary data streams that continually change over time. In this work, we propose a deep neural network model that can dynamically adjust to data distribution change by self-growth, that is, by changing its structure from narrow to wide and/or from shallow to deep. Specifically, initialized from a standard multilayer perceptron, the model grows by adding neurons according to a local error and adding layers according to the change in Jenson–Shannon divergence. The growth involves an adaptive memory mechanism, designed to avoid catastrophic forgetting, which may occur when hidden layers are added. In experiments, the proposed model achieved remarkable improvements in adaptability and prediction performance across 12 regression datasets (including non-stationary and stationary data streams), compared with the state-of-the-art dynamic neural network and other online approaches. The influence of the parameters, timeliness, convergence, and size of the pretrained model are explored in the paper.

Pan evaporation estimation by relevance vector machine tuned with new metaheuristic algorithms using limited climatic data

This study investigates the feasibility of a relevance vector machine tuned with improved Manta-Ray foraging optimization (RVM-IMRFO) in predicting monthly pan evaporation using limited climatic input data (e.g. temperature). The accuracy of the RVM-IMRFO was evaluated by comparing it with RVM tuned by gray wolf optimization, RVM tuned with a whale optimization algorithm, and RVM tuned with Manta Ray foraging optimization concerning root mean square errors (RMSE), mean absolute errors (MAE), determination coefficient (R2) and Nash-Sutcliffe Efficiency (NSE) and new graphical inspection methods. The models were assessed using data acquired from two stations in China and data were divided into three equal parts. The models were tested using each data set. The application outcomes revealed that the proposed algorithm considerably improved the accuracy of a single RVM in monthly pan evaporation prediction by an average improvement in RMSE, MAE, R2, and NSE as 27.65%, 27.53%, 8.40% and 8.63%, respectively. It is also found that the proposed algorithm showed significant dominance over others models with respect to improvement in overall mean values of RMSE, MAE, R2, and NSE statistics from 34.7–38.2 to 18.2–19.5, 36.2–36.4 to 19.1–18.5, 12.5–13.8 to 3.6–3.7, and 12.4–14.6 to 3.6–3.9%, for both climatic stations, respectively. Importing extraterrestrial radiation and periodicity component (month number of the data) into the model inputs improved the prediction accuracy of the implemented models. The outcomes revealed that the RVM-IMRFO performed superior to the other methods in predicting monthly pan evaporation using only temperature data which is essential, especially in developing countries where other climatic data are missing or unavailable. The RVM model was also compared with standard multi-layer perceptron neural networks (MLPNN) and found that the first acts better than the latter in monthly pan evaporation prediction.

Investigating the Use of Machine Learning Models to Understand the Drugs Permeability Across Placenta

Owing to limited drug testing possibilities in pregnant population, the development of computational algorithms is crucial to predict the fate of drugs in the placental barrier; it could serve as an alternative to animal testing. The ability of a molecule to effectively cross the placental barrier and reach the fetus determines the drug’s toxicological effects on the fetus. In this regard, our study aims to predict the permeability of molecules across the placental barrier. Based on publicly available datasets, several machine learning models are comprehensively analysed across different fingerprints and toolkits to find the best suitable models. Several dataset analysis models are utilised to study the data diversity. Further, this study demonstrates the application of neural network-based models to effectively predict the permeability. K-nearest neighbour (KNN), standard vector classifier (SVC) and Multi-layer perceptron (MLP) are found to be the best-performing models with a prediction percentage of 82%, 86.4% and 90.8%, respectively. Different models are compared to predict the chosen set of drugs, drugs like Aliskiren, some insulin secretagogues and glucocorticoids are found to be negative while predicting the permeability.

An integrated data-driven surrogate model and thermofluid network-based model of a 620 MWe utility-scale boiler

An integrated data-driven surrogate model and one-dimensional (1-D) process model of a 620 [ MWe] utility scale boiler is presented. A robust and computationally inexpensive computational fluid dynamic (CFD) model of the utility boiler was utilized to generate the solution dataset for surrogate model training and testing. Both a standard multi-layer perceptron (MLP) and mixture density network (MDN) machine learning architectures are compared for use as a surrogate model to predict the furnace heat loads and the flue gas inlet conditions to the convective pass. A hyperparameter search was performed to find the best MLP and MDN architecture. The MDN was selected for surrogate model integration since it showed comparable accuracy and provides the ability to predict the associated uncertainties. Validation of the integrated model against plant data was performed for a wide range of loads, and critical results were predicted within 5–8% of the measured results. The validated model was subsequently used to investigate the effects of using a poor-quality fuel for the 100% maximum continuous rating load case. The uncertainties predicted by the surrogate model were propagated through the integrated model using the Monte Carlo technique, adding valuable insight into the operational limits of the power plant and the uncertainties associated with it.

Exploiting Dual-Attention Networks for Explainable Recommendation in Heterogeneous Information Networks.

The aim of explainable recommendation is not only to provide recommended items to users, but also to make users aware of why these items are recommended. Traditional recommendation methods infer user preferences for items using user-item rating information. However, the expressive power of latent representations of users and items is relatively limited due to the sparseness of the user-item rating matrix. Heterogeneous information networks (HIN) provide contextual information for improving recommendation performance and interpreting the interactions between users and items. However, due to the heterogeneity and complexity of context information in HIN, it is still a challenge to integrate this contextual information into explainable recommendation systems effectively. In this paper, we propose a novel framework-the dual-attention networks for explainable recommendation (DANER) in HINs. We first used multiple meta-paths to capture high-order semantic relations between users and items in HIN for generating similarity matrices, and then utilized matrix decomposition on similarity matrices to obtain low-dimensional sparse representations of users and items. Secondly, we introduced two-level attention networks, namely a local attention network and a global attention network, to integrate the representations of users and items from different meta-paths for obtaining high-quality representations. Finally, we use a standard multi-layer perceptron to model the interactions between users and items, which predict users' ratings of items. Furthermore, the dual-attention mechanism also contributes to identifying critical meta-paths to generate relevant explanations for users. Comprehensive experiments on two real-world datasets demonstrate the effectiveness of DANER on recommendation performance as compared with the state-of-the-art methods. A case study illustrates the interpretability of DANER.

Data-Free and Data-Efficient Physics-Informed Neural Network Approaches to Solve the Buckley–Leverett Problem

Physics-informed neural networks (PINNs) are an emerging technology in the scientific computing domain. Contrary to data-driven methods, PINNs have been shown to be able to approximate and generalize well a wide range of partial differential equations (PDEs) by imbedding the underlying physical laws describing the PDE. PINNs, however, can struggle with the modeling of hyperbolic conservation laws that develop shocks, and a classic example of this is the Buckley–Leverett problem for fluid flow in porous media. In this work, we explore specialized neural network architectures for modeling the Buckley–Leverett shock front. We present extensions of the standard multilayer perceptron (MLP) that are inspired by the attention mechanism. The attention-based model was, compared to the multilayer perceptron model, and the results show that the attention-based architecture is more robust in solving the hyperbolic Buckley–Leverett problem, more data-efficient, and more accurate. Moreover, by utilizing distance functions, we can obtain truly data-free solutions to the Buckley–Leverett problem. In this approach, the initial and boundary conditions (I/BCs) are imposed in a hard manner as opposed to a soft manner, where labeled data are provided on the I/BCs. This allows us to use a substantially smaller NN to approximate the solution to the PDE.

Flexible Dual-Branched Message-Passing Neural Network for a Molecular Property Prediction.

A molecule is a complex of heterogeneous components, and the spatial arrangements of these components determine the whole molecular properties and characteristics. With the advent of deep learning in computational chemistry, several studies have focused on how to predict molecular properties based on molecular configurations. MA message-passing neural network provides an effective framework for capturing molecular geometric features with the perspective of a molecule as a graph. However, most of these studies assumed that all heterogeneous molecular features, such as atomic charge, bond length, or other geometric features, always contribute equivalently to the target prediction, regardless of the task type. In this study, we propose a dual-branched neural network for molecular property prediction based on both the message-passing framework and standard multilayer perceptron neural networks. Our model learns heterogeneous molecular features with different scales, which are trained flexibly according to each prediction target. In addition, we introduce a discrete branch to learn single-atom features without local aggregation, apart from message-passing steps. We verify that this novel structure can improve the model performance. The proposed model outperforms other recent models with sparser representations. Our experimental results indicate that, in the chemical property prediction tasks, the diverse chemical nature of targets should be carefully considered for both model performance and generalizability. Finally, we provide the intuitive analysis between the experimental results and the chemical meaning of the target.

AtomSets as a hierarchical transfer learning framework for small and large materials datasets

Predicting properties from a material’s composition or structure is of great interest for materials design. Deep learning has recently garnered considerable interest in materials predictive tasks with low model errors when dealing with large materials data. However, deep learning models suffer in the small data regime that is common in materials science. Here we develop the AtomSets framework, which utilizes universal compositional and structural descriptors extracted from pre-trained graph network deep learning models with standard multi-layer perceptrons to achieve consistently high model accuracy for both small compositional data (<400) and large structural data (>130,000). The AtomSets models show lower errors than the graph network models at small data limits and other non-deep-learning models at large data limits. They also transfer better in a simulated materials discovery process where the targeted materials have property values out of the training data limits. The models require minimal domain knowledge inputs and are free from feature engineering. The presented AtomSets model framework can potentially accelerate machine learning-assisted materials design and discovery with less data restriction.

A Hybrid-Domain Deep Learning-Based BCI For Discriminating Hand Motion Planning From EEG Sources.

In this paper, a hybrid-domain deep learning (DL)-based neural system is proposed to decode hand movement preparation phases from electroencephalographic (EEG) recordings. The system exploits information extracted from the temporal-domain and time-frequency-domain, as part of a hybrid strategy, to discriminate the temporal windows (i.e. EEG epochs) preceding hand sub-movements (open/close) and the resting state. To this end, for each EEG epoch, the associated cortical source signals in the motor cortex and the corresponding time-frequency (TF) maps are estimated via beamforming and Continuous Wavelet Transform (CWT), respectively. Two Convolutional Neural Networks (CNNs) are designed: specifically, the first CNN is trained over a dataset of temporal (T) data (i.e. EEG sources), and is referred to as T-CNN; the second CNN is trained over a dataset of TF data (i.e. TF-maps of EEG sources), and is referred to as TF-CNN. Two sets of features denoted as T-features and TF-features, extracted from T-CNN and TF-CNN, respectively, are concatenated in a single features vector (denoted as TTF-features vector) which is used as input to a standard multi-layer perceptron for classification purposes. Experimental results show a significant performance improvement of our proposed hybrid-domain DL approach as compared to temporal-only and time-frequency-only-based benchmark approaches, achieving an average accuracy of [Formula: see text]%.

Neural network-crow search model for the prediction of functional properties of nano TiO2 coated cotton composites

This paper presents a new hybrid approach for the prediction of functional properties i.e., self-cleaning efficiency, antimicrobial efficiency and ultraviolet protection factor (UPF), of titanium dioxide nanoparticles (TiO2 NPs) coated cotton fabric. The proposed approach is based on feedforward artificial neural network (ANN) model called a multilayer perceptron (MLP), trained by an optimized algorithm known as crow search algorithm (CSA). ANN is an effective and widely used approach for the prediction of extremely complex problems. Various studies have been proposed to improve the weight training of ANN using metaheuristic algorithms. CSA is a latest and an effective metaheuristic method relies on the intelligent behavior of crows. CSA has been never proposed to improve the weight training of ANN. Therefore, CSA is adopted to optimize the initial weights and thresholds of the ANN model, in order to improve the training accuracy and prediction performance of functional properties of TiO2 NPs coated cotton composites. Furthermore, our proposed algorithm i.e., multilayer perceptron with crow search algorithm (MLP-CSA) was applied to map out the complex input–output conditions to predict the optimal results. The amount of chemicals and reaction time were selected as input variables and the amount of titanium dioxide coated on cotton, self-cleaning efficiency, antimicrobial efficiency and UPF were evaluated as output results. A sensitivity analysis was carried out to assess the performance of CSA in prediction process. MLP-CSA provided excellent result that were statistically significant and highly accurate as compared to standard MLP model and other metaheuristic algorithms used in the training of ANN reported in the literature.

Silvi-Net – A dual-CNN approach for combined classification of tree species and standing dead trees from remote sensing data

Forest managers and nature conservationists rely on precise mapping of single trees from remote sensing data for efficient estimation of forest attributes. In recent years, additional quantification of dead wood in particular has garnered interest. However, tree-level approaches utilizing segmented single trees are still limited in accuracy and their application is therefore mostly restricted to research studies. Furthermore, the combined classification of presegmented single trees with respect to tree species and health status is important for practical use but has been insufficiently investigated so far. Therefore, we introduce Silvi-Net, an approach based on convolutional neural networks (CNNs) fusing airborne lidar data and multispectral (MS) images for 3D object classification. First, we segment single 3D trees from the lidar point cloud, render multiple silhouette-like side-view images, and enrich them with calibrated laser echo characteristics. Second, projected outlines of the segmented trees are used to crop and mask the MS orthomosaic and to generate MS image patches for each tree. Third, we independently train two ResNet-18 networks to learn meaningful features from both datasets. This optimization process is based on pretrained CNN weights and recursive retraining of model parameters. Finally, the extracted features are fused for a final classification step based on a standard multi-layer perceptron and majority voting. We analyzed the network’s performance on data captured in two study areas, the Chernobyl Exclusion Zone (ChEZ) and the Bavarian Forest National Park (BFNP). For both study areas, the lidar point density was approximately 55 points/m2 and the ground sampling distance values of the true orthophotos were 10 cm (ChEZ) and 20 cm (BFNP). In general, the trained models showed high generalization capacity on independent test data, achieving an overall accuracy (OA) of 96.1% for the classification of pines, birches, alders, and dead trees (ChEZ) - and 91.5% for coniferous, deciduous, snags, and dead trees (BFNP). Interestingly, lidar-based imagery increased the OA by 2.5% (ChEZ) and 5.9% (BFNP) compared to experiments only utilizing MS imagery. Moreover, Silvi-Net also demonstrated superior OA compared to the baseline method PointNet++ by 11.3% (ChEZ) and 2.2% (BFNP). Overall, the effectiveness of our approach was proven using 2D and 3D datasets from two natural forest areas (400–530 trees/ha), acquired with different sensor models, and varying geometric and spectral resolution. Using the technique of transfer learning, Silvi-Net facilitates fast model convergence, even for datasets with a reduced number of samples. Consequently, operators can generate reliable maps that are of major importance in applications such as automated inventory and monitoring projects.

Application of combined model of stepwise regression analysis and artificial neural network in data calibration of miniature air quality detector

In this paper, six types of air pollutant concentrations are taken as the research object, and the data monitored by the micro air quality detector are calibrated by the national control point measurement data. We use correlation analysis to find out the main factors affecting air quality, and then build a stepwise regression model for six types of pollutants based on 8 months of data. Taking the stepwise regression fitting value and the data monitored by the miniature air quality detector as input variables, combined with the multilayer perceptron neural network, the SRA-MLP model was obtained to correct the pollutant data. We compared the stepwise regression model, the standard multilayer perceptron neural network and the SRA-MLP model by three indicators. Whether it is root mean square error, average absolute error or average relative error, SRA-MLP model is the best model. Using the SRA-MLP model to correct the data can increase the accuracy of the self-built point data by 42.5% to 86.5%. The SRA-MLP model has excellent prediction effects on both the training set and the test set, indicating that it has good generalization ability. This model plays a positive role in scientific arrangement and promotion of miniature air quality detectors. It can be applied not only to air quality monitoring, but also to the monitoring of other environmental indicators.

How machine learning conquers the unitary limit

Machine learning has become a premier tool in physics and other fields of science. It has been shown that the quantum mechanical scattering problem cannot only be solved with such techniques, but it was argued that the underlying neural network develops the Born series for shallow potentials. However, classical machine learning algorithms fail in the unitary limit of an infinite scattering length. The unitary limit plays an important role in our understanding of bound strongly interacting fermionic systems and can be realized in cold atom experiments. Here, we develop a formalism that explains the unitary limit in terms of what we define as unitary limit surfaces. This not only allows to investigate the unitary limit geometrically in potential space, but also provides a numerically simple approach towards unnaturally large scattering lengths with standard multilayer perceptrons. Its scope is therefore not limited to applications in nuclear and atomic physics, but includes all systems that exhibit an unnaturally large scale.

MCS-MCMC for Optimising Architectures and Weights of Higher Order Neural Networks

The standard method to train the Higher Order Neural Networks (HONN) is the well-known Backpropagation (BP) algorithm. Yet, the current BP algorithm has several limitations including easily stuck into local minima, particularly when dealing with highly non-linear problems and utilise computationally intensive training algorithms. The current BP algorithm is also relying heavily on the initial weight values and other parameters picked. Therefore, in an attempt to overcome the BP drawbacks, we investigate a method called Modified Cuckoo Search-Markov chain Monté Carlo for optimising the weights in HONN and boost the learning process. This method, which lies in the Swarm Intelligence area, is notably successful in optimisation task. We compared the performance with several HONN-based network models and standard Multilayer Perceptron on four (4) time series datasets: Temperature, Ozone, Gold Close Price and Bitcoin Closing Price from various repositories. Simulation results indicate that this swarm-based algorithm outperformed or at least at par with the network models with current BP algorithm in terms of lower error rate.

A hybrid machine learning framework for analyzing human decision-making through learning preferences

Multiple criteria decision aiding (MCDA) is a family of analytic approaches to depicting the rationale of human decisions. To better interpret the contributions of individual attributes to the decision maker, the conventional MCDA approaches assume that the attributes are monotonic and preference independence. However, the capacity in describing the decision maker’s preferences is sacrificed as a result of model simplification. To meet the decision maker’s demand for more accurate and interpretable decision models, we propose a novel hybrid method, namely Neural Network-based Multiple Criteria Decision Aiding (NN-MCDA), which combines MCDA model and machine learning to achieve better prediction performance while capturing the relationships between individual attributes and the prediction. NN-MCDA uses a linear component (in an additive form of a set of polynomial functions) to characterize such relationships through providing explicit non-monotonic marginal value functions, and a nonlinear component (in a standard multilayer perceptron form) to capture the implicit high-order interactions among attributes and their complex nonlinear transformations. We demonstrate the effectiveness of NN-MCDA with extensive simulation studies and three real-world datasets. The study sheds light on how to improve the prediction performance of MCDA models using machine learning techniques, and how to enhance the interpretability of machine learning models using MCDA approaches.