Single Layer Feedforward Neural Network Research Articles

Diversifying Multi-Head Attention in the Transformer Model

Recent studies have shown that, due to redundancy, some heads of the Transformer model can be pruned without diminishing the efficiency of the model. In this paper, we propose a constrained optimization algorithm based on Hebbian learning, which trains specific layers in the Transformer architecture in order to enforce diversification between the different heads in the multi-head attention module. The diversification of the heads is achieved through a single-layer feed-forward neural network that is added to the Transformer architecture and is trained with the proposed algorithm. We utilize the algorithm in three different architectural variations of the baseline Transformer model. In addition to the diversification of the heads, the proposed methodology can be used to prune the heads that capture redundant information. Experiments on diverse NLP tasks, including machine translation, text summarization, question answering and large language modeling, show that our proposed approach consistently improves the performance of baseline Transformer models.

Heat transfer evaluation of combined nanofluids in vertical annulus: An experimental and artificial neural network approach

Enhancing the thermal properties of the working fluid is a practical way to improve heat transfer performance. One of the suggested fluids for this purpose is nanofluids, which are suspensions of nanoparticles within a base fluid. Nanofluids have the potential to operate as a primary coolant or as an emergency coolant in a nuclear reactor. This paper investigates the thermal performance of a homogeneous hybrid nanofluid of Al2O3 and TiO2 in deionized water. The thermo-hydraulic performance of the nanofluid is analyzed in a high-pressure test loop capable of 25 bar pressure. The test section has an annular geometry, generating a cosine heat flux pattern. This flux pattern is representative of nuclear reactor cores. The experimental study is conducted to investigate the effects of nanoparticle concentration and Reynolds number on the heat transfer performance of nanofluids. The results show that increasing these two parameters significantly improved heat transfer. Specifically, raising the nanoparticle concentration from 0% to 1% leads to a reduction in surface temperature of 19% and 8% at Reynolds numbers of 825 and 4128, respectively. The calculated Nusselt numbers are compared with correlations from previous studies. Furthermore, an artificial neural network model is developed to predict heat transfer data for the studied system. The model consists of a single-layer feed-forward neural network trained with the stochastic gradient descent algorithm. To determine the optimal network, the number of neurons is optimized and evaluated based on the network's predictability. The neural network model demonstrates substantial improvements over previous experimental correlations (MAPE <2.5% and MSE <0.001).

Efficient Ensemble Broad Learning System Based on Dropout and DropConnect

Broad Learning System is an emerged efficient algorithm for training single hidden layer feedforward neural network (SLFN) with fast speed and good generalization ability. However, it is very difficult to determine the appropriate broad learning system structure and broad learning system may perform overfitting due to the dependence between nodes in processing fully connected network. In order to deal with these problems, efficient ensemble broad learning system based on Dropout and DropConnect are proposed in this paper. The proposed Dropout Ensemble Broad Learning System randomly discards hidden nodes to improve diversity between individuals and reduce the synergy between nodes to improve prediction stability. The DropConnect Ensemble Broad Learning System randomly drops connect weights to generate more complementary models by adding input attribute disturbance. The experimental results and statistical analysis on UCI data sets confirm that the proposed method can solve the problem of model overfitting caused by the strong dependence between the nodes of ensemble broad learning system and also show that the proposed approaches outperform the original BLS on prediction stability and classification accuracy.

Prediksi Harga Kelapa Sawit Menggunakan Metode Extreme Learning Machine

Palm oil is one of the keys to the Indonesian economy and the main commodity for attracting foreign investment. The palm oil and palm kernel industry generates most of the foreign currency from palm oil. The price of palm oil often goes up and down every month resulting in instability in the income received by people who own oil palm plantations. The aim of predicting palm oil prices is to carry out appropriate planning or steps for palm oil business actors. One way to overcome this problem is to make predictions. One method that can make predictions is the Extreme Learning Machine (ELM). ELM is an artificial neural network method used to predict palm oil prices. The ELM method is a feedforward method with a single hidden layer which is better known as a single hidden layer feedforward neural network (SLFNs). In this research, the best implementation was 5 inputs with 20 neurons in the hidden layer with output in the form of palm oil price predictions. Based on the tests carried out, the research produced the smallest error rate of 0.0027111424247658633 using 20 neurons in the hidden layer so that the latest data prediction test results for 5 price rotations in September rotation 1 were 1400.314191, September rotation 2 were 1846.798921, September rotation 3 amounted to 1505.430419, September rotation 4 amounted to 2301.853412, September rotation 5 amounted to 2645.082489 in palm oil price predictions.

Extreme Learning Machine Classification Method based on Twin Strategy Salp Swarm Algorithm

Extreme Learning Machines (ELM) are a type of Single Hidden Layer Feedforward Neural Network (SLFN). In recent years, they have attracted attention for their powerful approximation ability and fast learning speed. Compared with traditional neural network algorithms, ELM have the advantages of simple structure, fast learning speed, and good generalization performance. However, since the input weights and biases of ELM are randomly generated, there may be some suboptimal or unnecessary input weights and biases. In addition, ELM may require more hidden nodes, which may slow down its response to unknown test data. To address these issues, a twin strategy Salp Swarm Algorithm (TSSA) is proposed to increase the population diversity to improve the convergence speed of the algorithm through chaotic initialization, while shock inertia weights and learning paradigms are introduced into the follower position updating to enhance the stochasticity of the particles. Classification tests are conducted on six datasets using the proposed TSSA, and the experimental results show that the proposed TSSA-ELM has higher accuracy and better generalization performance than some existing ELM variants.

Industrial AI in condition-based maintenance: A case study in wooden piece manufacturing

The article presents a case study applying industrial artificial intelligence to Condition-Based Maintenance in a wooden piece manufacturing company. The study focuses on the extraction system that transports wood residue to a warehouse, supplying a biomass plant for cold and heat generation in the factory. The objective is to predict the temperature of the ten induction motors in the extraction system using an Extreme Learning Machines-based methodology, enabling dynamic model prediction. Data from IoT sensors measuring the motors’ intensity, temperature, and humidity are collected every minute, pre-processed, and stored in a database. The pre-processing includes a single novel algorithm to detect and eliminate data containing possible sensor blockages. The results demonstrate an implementable methodology utilizing single-layer feedforward neural networks, prioritizing fast training while maintaining sufficient accuracy for detecting deviations in motor behaviour. The research offers valuable insights for preventive maintenance applications in similar industrial settings.

Comparative analysis of feed-forward neural network and second-order polynomial regression in textile wastewater treatment efficiency

&lt;abstract&gt;&lt;p&gt;This study refines a single-layer Feed-Forward Neural Network (FFNN) for the treatment of textile dye wastewater, concentrating on percentage decolorization (%DEC) and percentage chemical oxygen demand (%COD) reduction. The optimized neural network configuration comprises four input and one output neuron, fine-tuned based on the mean squared error (MSE). The training phase demonstrates a consistent MSE decline, reaching its lowest at epoch 209 for %DEC and epoch 34 for %COD, with corresponding MSEs of $1.799 \times 10^{-5}$ and $ 1.4 \times 10^{-3} $, respectively. The maximum absolute errors for %DEC and %COD were found to be $ 4.0787 $ and $ 2.4486 $, while the mean absolute errors were $ 0.4821 $ and $ 0.7256 $, respectively. In contrast to second-degree polynomial regression, the FFNN model exhibits enhanced predictive accuracy, as indicated by higher $ R^2 $ values of $ 0.99363 $ for %DEC and $ 0.99716 $ for %COD, and reduced error metrics.&lt;/p&gt;&lt;/abstract&gt;

A fast online stacked regressor to handle concept drifts

The number of algorithms based on Extreme Learning Machine (ELM), which train Single Layer Feedforward Neural Networks (SLFN), has increased in the past years due to their fast training stages and good generalization performances. Many variants have been proposed, which deal with outliers, with online learning and other characteristics. Recently, Fast Deep Stacked Network (FDSN) was proposed, which is an alternative to large ELM networks with similar performances using less memory than ELM-based algorithms. Regarding online methods, which deal with data that arrives over time considering concept drift, ensemble approaches are one of the most promising categories of these methods. FDSN can be viewed as an ensemble of SLFN, where the network output is the output of the most recent SLFN. In this paper, we propose Online Sequential FDSN (OSFDSN), which is similar to FDSN, but each of its SLFN modules has a weighted contribution to the network output. These weights are dynamically calculated, according to the most recent data. We compare our method with some other similar techniques with implementations publicly available. Our conducted experiments show that the proposed technique is statistically equivalent to the others when considering RMSE, while having a faster updating scheme.

Micromechanics-informed parametric deep material network for physics behavior prediction of heterogeneous materials with a varying morphology

Deep Material Network (DMN) has recently emerged as a data-driven surrogate model for heterogeneous materials. Given a particular microstructural morphology, the effective linear and nonlinear behaviors can be successfully approximated by such physics-based neural-network like architecture. In this work, a novel micromechanics-informed parametric DMN (MIpDMN) architecture is proposed for multiscale materials with a varying microstructure characterized by several parameters. A single-layer feedforward neural network is used to account for the dependence of DMN fitting parameters on the microstructural ones. Micromechanical constraints are prescribed both on the architecture and the outputs of this new neural network. The proposed MIpDMN is also recast in a multiple physics setting, where physical properties other than the mechanical ones can also be predicted. In the numerical simulations conducted on three parameterized microstructures, MIpDMN demonstrates satisfying generalization capabilities when morphology varies. The effective behaviors of such parametric multiscale materials can thus be predicted and encoded by MIpDMN with high accuracy and efficiency.

Evolutionary Optimization of Convolutional Extreme Learning Machine for Remaining Useful Life Prediction

Remaining useful life (RUL) prediction is a key enabler for making optimal maintenance strategies. Data-driven approaches, especially employing neural networks (NNs) such as multi-layer perceptrons (MLPs) and convolutional neural networks (CNNs), have gained increasing attention in the field of RUL prediction. Most of the past research has mainly focused on minimizing the RUL prediction error by training NNs with back-propagation (BP), which in general requires an extensive computational effort. However, in practice, such BP-based NNs (BPNNs) may not be affordable in industrial contexts that normally seek to save cost by minimizing access to expensive computing infrastructures. Driven by this motivation, here, we propose: (1) to use a very fast learning scheme called extreme learning machine (ELM) for training two different kinds of feed-forward neural networks (FFNNs), namely a single-layer feed-forward neural network (SL-FFNN) and a Convolutional ELM (CELM); and (2) to optimize the architecture of those networks by applying evolutionary computation. More specifically, we employ a multi-objective optimization (MOO) technique to search for the best network architectures in terms of trade-off between RUL prediction error and number of trainable parameters, the latter being correlated with computational effort. In our experiments, we test our methods on a widely used benchmark dataset, the C-MAPSS, on which we search such trade-off solutions. Compared to other methods based on BPNNs, our methods outperform a MLP and show a similar level of performance to a CNN in terms of prediction error, while using a much smaller (up to two orders of magnitude) number of trainable parameters.

Mixture extreme learning machine algorithm for robust regression

The extreme learning machine (ELM) is a well-known approach for training single hidden layer feedforward neural networks (SLFNs) in machine learning. However, ELM is most effective when used for regression on datasets with simple Gaussian distributed error because it often employs a squared loss in its objective function. In contrast, real-world data is often collected from unpredictable and diverse contexts, which may contain complex noise that cannot be characterized by a single distribution. To address this challenge, we propose a robust mixture ELM algorithm, called Mixture-ELM, that enhances modeling capability and resilience to both Gaussian and non-Gaussian noise. The Mixture-ELM algorithm uses an adjusted objective function that blends Gaussian and Laplacian distributions to approximate any continuous distribution and match the noise. The Gaussian mixture accurately models the residual distribution, while the inclusion of the Laplacian distribution addresses the limitations of the Gaussian distribution in identifying outliers. We derive a solution to the novel objective function using the expectation maximization (EM) and iteratively reweighted least squares (IRLS) algorithms. We evaluate the effectiveness of the algorithm through numerical simulation and experiments on benchmark datasets, thereby demonstrating its superiority over other state-of-the-art machine learning methods in terms of robustness and generalization.

Identification Of Hijaiyah Letters Image Using Extreme Learning Machine Method

The development of digital image processing technology has many benefits, one of which is the identification of an object, such as the identification of the image of the hijaiyah letter. Hijaiyah letters are the letters of the arabic alphabet as the original language of the Qur'an. In essence, humans have the ability to recognize and distinguish hijaiyah letter patterns from one another, but this is not the case with computers, using digital images and machine learning, in this study an identification concept was built by recognizing the image of hijaiyah letters using one of the machine learning methods. namely the extreme learning machine (ELM) method. Extreme learning machine (ELM) is a feedforward neural network with one hidden layer or better known as single hidden layer feedforward neural networks (SLFNs). Therefore, the purpose of this study is that the computer can identify objects as well as human capabilities and see how accurate the results obtained in the ELM method are. The digital image identification process using the extreme learning machine (ELM) method is carried out in two stages, namely training and testing, where previously the preprocessing process was carried out first by changing the color of the RGB image to HSV and processing the color v, then segmentation was carried out with the aim of separating the objects (foreground) with the background, then to make it easier to recognize the pattern, a morphological process is carried out. From the simulation carried out on the test data, the results obtained an average accuracy of 90% with an error of 10%.

PREDICTING CONSTRUCTION CREW PRODUCTIVITY FOR CONCRETE POURING OPERATIONS OF RC COLUMNS USING ANN

In current practice, construction crew productivity (CCP) assessment solely relies on traditional methods like personal judgements or past published data. However, a systematic approach is required to measure and predict the CCP. Modelling with artificial neural networks (ANNs) can be useful for examining interrelationships between factors affecting CCP. The present study aims to predict CCP for concrete pouring operation of reinforced concrete (RC) columns using ANN. To achieve this, 14 factors like crew size, average age of crew, average experience of crew, extent of supervision, working hours, lead distance, working height, location of columns, number of concrete bucket carriers, method of concrete supply, temperature, wind speed, column length, and column width were identified from literature study and industry experts. These identified factors were considered as inputs, while CCP was the output variable for ANN model. The study collected 187 samples from 11 residential and commercial construction sites located in four cities of India. A single-layer feed-forward back propagation neural network was used for prediction. The developed model can estimate productivity rates reasonably with least range of errors. Outcomes of study can be useful to improve the crew’s productivity at construction sites for concrete pouring works of RC columns.

Functional extreme learning machine.

Extreme learning machine (ELM) is a training algorithm for single hidden layer feedforward neural network (SLFN), which converges much faster than traditional methods and yields promising performance. However, the ELM also has some shortcomings, such as structure selection, overfitting and low generalization performance. This article a new functional neuron (FN) model is proposed, we takes functional neurons as the basic unit, and uses functional equation solving theory to guide the modeling process of FELM, a new functional extreme learning machine (FELM) model theory is proposed. The FELM implements learning by adjusting the coefficients of the basis function in neurons. At the same time, a simple, iterative-free and high-precision fast parameter learning algorithm is proposed. The standard data sets UCI and StatLib are selected for regression problems, and compared with the ELM, support vector machine (SVM) and other algorithms, the experimental results show that the FELM achieves better performance.

A self-organizing modular neural network based on empirical mode decomposition with sliding window for time series prediction

Time series is mostly with a chaotic nature and non-stationary characteristic in real-word, which makes it difficult to be modeled and predicted accurately. To solve this problem, we introduce a novel self-organizing modular neural network based on the empirical mode decomposition with the sliding window mechanism (SWEMD-MNN) for time series prediction. In SWEMD-MNN, the improved empirical mode decomposition with sliding window (SWEMD) is developed to decompose time series online, which can effectively alleviate the limitation that the traditional EMD-based models cannot handle the long term or online problem and end effect. Thus, SWEMD-MNN can decompose time series based on time characteristic effectively and dynamically, and improve the prediction accuracy of the classical modular neural networks dividing time series based on sample space. Then time subseries are dynamically assigned to the subnetworks with a single layer feedforward neural network using the sample entropy and Euclidean distance for learning. Experimental investigations using benchmark chaotic and real-world time series show that SWEMD-MNN can decompose time series effectively and dynamically, and provides a better prediction accuracy than the fully coupled networks and other MNN models for time series prediction.

Plausible presence of new state in neutron stars with masses above [formula omitted

We investigate the neutron star (NS) equation of state (EOS) by incorporating multi-messenger data of GW170817, PSR J0030 + 0451, PSR J0740 + 6620, and state-of-the-art theoretical progresses, including the information from chiral effective field theory (χEFT) and perturbative quantum chromodynamics (pQCD) calculation. Taking advantage of the various structures sampling by a single-layer feed-forward neural network model embedded in the Bayesian nonparametric inference, the structure of NS matter’s sound speed cs is explored in a model-agnostic way. It is found that a peak structure is common in the cs2 posterior, locating at (2.4-4.8)ρsat (nuclear saturation density) and cs2 exceeds c2/3 at 90% credibility. The non-monotonic behavior suggests evidence of the state deviating from the hadronic matter inside the very massive NSs. Assuming the new/exotic state is featured as it is softer than typical hadronic models or even with hyperons, we find that a sizable (⩾10-3M⊙) exotic core, likely made of quark matter, is plausible for the NS with a gravitational mass above about 0.98MTOV, where MTOV represents the maximum gravitational mass of a non-rotating cold NS. The inferred MTOV=(2.18-0.13+0.27)M⊙ (90% credibility) is well consistent with the value of (2.17-0.12+0.15)M⊙ estimated independently with GW170817/GRB 170817A/AT2017gfo assuming a temporary supramassive NS remnant formed after the merger. PSR J0740 + 6620, the most massive NS detected so far, may host a sizable exotic core with a probability of ≈0.36.

STWD-SFNN: Sequential three-way decisions with a single hidden layer feedforward neural network

The three-way decisions strategy has been employed to construct network topology in a single hidden layer feedforward neural network (SFNN). However, this model has a general performance, and does not consider the process costs, since it has fixed threshold parameters. Inspired by the sequential three-way decisions (STWD), this paper proposes STWD with an SFNN (STWD-SFNN) to enhance the performance of networks on structured datasets. STWD-SFNN adopts multi-granularity levels to dynamically learn the number of hidden layer nodes from coarse to fine, and set the sequential threshold parameters. Specifically, at the coarse granular level, STWD-SFNN handles easy-to-classify instances by applying strict threshold conditions, and with the increasing number of hidden layer nodes at the fine granular level, STWD-SFNN focuses more on disposing of the difficult-to-classify instances by applying loose threshold conditions, thereby realizing the classification of instances. Moreover, STWD-SFNN considers and reports the process cost produced from each granular level. The experimental results verify that STWD-SFNN has a more compact network on structured datasets than other SFNN models, and has better generalization performance than the competitive models. All models and datasets can be downloaded from https://github.com/wuc567/Machine-learning/tree/main/STWD-SFNN.

An Accelerated Maximally Split ADMM for a Class of Generalized Ridge Regression.

Ridge regression (RR) has been commonly used in machine learning, but is facing computational challenges in big data applications. To meet the challenges, this article develops a highly parallel new algorithm, i.e., an accelerated maximally split alternating direction method of multipliers (A-MS-ADMM), for a class of generalized RR (GRR) that allows different regularization factors for different regression coefficients. Linear convergence of the new algorithm along with its convergence ratio is established. Optimal parameters of the algorithm for the GRR with a particular set of regularization factors are derived, and a selection scheme of the algorithm parameters for the GRR with general regularization factors is also discussed. The new algorithm is then applied in the training of single-layer feedforward neural networks. Experimental results on performance validation on real-world benchmark datasets for regression and classification and comparisons with existing methods demonstrate the fast convergence, low computational complexity, and high parallelism of the new algorithm.

A novel DCNN-ELM hybrid framework for face mask detection

The Coronavirus disease (2019) has caused massive destruction of human lives and capital around the world. The latest variant Omicron is proved to be the most infectious of all its previous counterparts – Alpha, Beta and Delta. Various measures are identified, tested and implemented to minimize the attack on humans. Face masks are one of those measures that are shown to be very effective in containing the infection. However, it requires continuous monitoring for law enforcement. In the present manuscript, a detailed research investigation using different ablation studies is carried out to develop the framework for face mask recognition using pre-trained deep convolution neural networks (DCNN) models used in conjunction with a fast single layer feed-forward neural network (SLFNN) commonly known as Extreme Learning Machine (ELM) as classification technique. The ELM is well known for its real time data processing capabilities and has been successfully applied both for regression and classification problems of image processing and biomedical domain. It is for the first time that in this paper we have proposed the use of ELM as classifier for face mask detection. As a precursor to this, for feature selection, six pre-trained DCNNs such as Xception, Vgg16, Vgg19, ResNet50, ResNet 101 and ResNet152 are tested for this purpose. The best testing accuracy is obtained in case of ResNet152 transfer learning model used with ELM as the classifier. The performance evaluation through different ablation studies on testing accuracy explicitly proves that ResNet152 - ELM hybrid architecture is not only the best among the selected transfer learning models but also proves so when it is compared with several other classifiers used for the face mask detection operation. Through this investigation, novelty of the use of ResNet152 + ELM for face mask detection framework in real time domain is established.

Retracted: Potential Fault Diagnosis Method and Classification Accuracy Detection of IGBT Device Based on Improved Single Hidden Layer Feedforward Neural Network.

[This retracts the article DOI: 10.1155/2021/6036118.].