Learning In Feedforward Neural Networks Research Articles

Emergence and reconfiguration of modular structure for artificial neural networks during continual familiarity detection.

Advances in artificial intelligence enable neural networks to learn a wide variety of tasks, yet our understanding of the learning dynamics of these networks remains limited. Here, we study the temporal dynamics during learning of Hebbian feedforward neural networks in tasks of continual familiarity detection. Drawing inspiration from network neuroscience, we examine the network's dynamic reconfiguration, focusing on how network modules evolve throughout learning. Through a comprehensive assessment involving metrics like network accuracy, modular flexibility, and distribution entropy across diverse learning modes, our approach reveals various previously unknown patterns of network reconfiguration. We find that the emergence of network modularity is a salient predictor of performance and that modularization strengthens with increasing flexibility throughout learning. These insights not only elucidate the nuanced interplay of network modularity, accuracy, and learning dynamics but also bridge our understanding of learning in artificial and biological agents.

The Southern Ocean carbon sink has been overestimated in the past three decades

Employing machine learning methods for mapping surface ocean pCO2 has reduced the uncertainty in estimating sea-air CO2 flux. However, a general discrepancy exists between the Southern Ocean carbon sinks derived from pCO2 products and those from biogeochemistry models. Here, by performing a boosting ensemble learning feed-forward neural networks method, we have identified an underestimation of the surface Southern Ocean pCO2 due to notably uneven density of pCO2 measurements between summer and winter, which resulted in about 16% overestimating of Southern Ocean carbon sink over the past three decades. In particular, the Southern Ocean carbon sink since 2010 was notably overestimated by approximately 29%. This overestimation can be mitigated by a winter correction in algorithms, with the average Southern Ocean carbon sink during 1992-2021 corrected to −0.87 PgC yr−1 from the original −1.01 PgC yr−1. Furthermore, the most notable underestimation of surface ocean pCO2 mainly occurred in regions south of 60°S and was hiding under ice cover. As the surface ocean pCO2 under sea ice coverage in the winter is much higher than the atmosphere, if sea ice melts completely, there could be a further reduction of about 0.14 PgC yr−1 in the Southern Ocean carbon sink.

Intelligent Lighting Control Systems for Energy Savings in Hospital Buildings Using Artificial Neural Networks

Lighting control systems are essential in modern building automation and smart homes, efficiently managing illumination to enhance energy conservation and user comfort. This project tackles energy consumption challenges in hospital buildings by introducing Intelligent Lighting Control Systems (ILCS) that take natural light and occupancy into account, driven by Artificial Neural Networks (ANN) and diverse machine learning algorithms. In our study, we collected sensor data, processed it, and designed a lighting control system employing a feedforward neural network and various machine learning algorithms. Surprisingly, our research found that a linear regression algorithm surpassed the ANN-based system in this context. We implemented a prototype, tested it on hardware, and obtained the expected results. This research marks progress towards optimizing energy use in hospital buildings and contributing to sustainability endeavors. By combining ILCS and machine learning, it offers a promising approach for more efficient and eco-friendly lighting systems

Fossil energy market price prediction by using machine learning with optimal hyper-parameters: A comparative study

Fossil energy markets are important commodities, and their price fluctuations impact worldwide economy and financial markets. Hence, it is essential to forecast the prices of fossil energy commodities. In this study, various machine learning systems are optimized by using Bayesian optimization method and implemented to forecast prices in 12 different fossil energy markets including crude oil, natural gas, propane, kerosene, gasoline, heating oil, and coal based on price information from all fossil energy markets. The optimized machine learning systems considered in modeling and simulations are Gaussian regression process, support vector regression, regression trees, k-nearest neighbor algorithm, and deep feedforward neural networks. The simulation results show that Gaussian regression process is the best system to forecast fossil energy markets. In addition, the deep learning feedforward neural networks system yields to stable forecasts. Furthermore, the prediction of the prices in natural gas, coal, and propane markets is a difficult task compared to the prediction of the prices in crude oil markets. The understanding of the effectiveness of machine learning systems in pricing different fossil energy markets can help international economic agents establish appropriate investment strategies.

Topological learning in multiclass data sets.

We specialize techniques from topological data analysis to the problem of characterizing the topological complexity (as defined in the body of the paper) of a multiclass data set. As a by-product, a topological classifier is defined that uses an open subcovering of the data set. This subcovering can be used to construct a simplicial complex whose topological features (e.g., Betti numbers) provide information about the classification problem. We use these topological constructs to study the impact of topological complexity on learning in feedforward deep neural networks (DNNs). We hypothesize that topological complexity is negatively correlated with the ability of a fully connected feedforward deep neural network to learn to classify data correctly. We evaluate our topological classification algorithm on multiple constructed and open-source data sets. We also validate our hypothesis regarding the relationship between topological complexity and learning in DNN's on multiple data sets.

A deep learning feed-forward neural network framework for the solutions to singularly perturbed delay differential equations

In this paper, we explore a deep learning feedforward artificial neural network (ANN) framework as a numerical tool for approximating the solutions to singularly perturbed delay differential equations (SPDDE). Our approach trains the network with fewer uniform data points along with linear interpolation as a dependent variable. More importantly, the exact solution is not used for training the network. A mean square or Euclidean norm type total loss function is utilized to assess the deep learning network’s performance. In the training and testing phase, our network adjusts itself to fit the tangent and the curvature by minimizing the total loss function and fine-tuning the network’s hyperparameters during the backpropagation stage. Our focus in this contribution is to investigate an answer to the question- “whether neural networks can learn to solve the differential equations with both delay and boundary layer behavior”? Traditional numerical methods are known to perform poorly in approximating the solutions to SPDDE due to boundary layer behavior, which is the sharp change in the gradient of the solution inside a region of very small width. Our proposed network architecture is used to investigate the above question for three varieties of SPDDE. The numerical results demonstrate that the fine-tuned adaptive deep learning architecture can effectively approximate the solution to SPDDE for varieties of delay and perturbation parameters. It is expected that the current method can be extendable to approximate 2D convection-diffusion partial differential equations with a boundary layer.

Machine learning-based prediction of shear strength of steel reinforced concrete columns subjected to axial compressive load and seismic lateral load

The shear strength of steel reinforced concrete (SRC) columns is a crucial basis in their seismic design and evaluation of structural performance. Most of the existing research predicts the lateral load carrying capacity based on regression equations, but lacks accuracy. To this end, in this study, Gaussian process regression (GPR), Least Squares Boosting (LSBoosting), Support Vector Regression (SVR), Feedforward Neural Network (FNN) and other machine learning (ML) algorithms were employed to develop the shear strength model of SRC columns subjected to axial compressive load and seismic lateral load. A large experimental database with 395 samples was established to train 11 input features. The original database was divided into 5 datasets based on the mixed, flexural, flexural-shear, shear, and bond failure modes. The failure mechanism factors were introduced for better evaluation. Specifically, an effective data splitting strategy-bootstrapping was used to train the models, and Bayesian optimization was attempted to improve the prediction accuracy. Results demonstrate that GPR has the highest prediction accuracy with the most failure modes, and the factors affecting the shear strength are explained reasonably by correlation analysis and Partial Dependence Plot. Through comprehensive comparison, the GPR model developed in this paper is advanced in predicting the shear strength of SRC columns.

Automatic Hybrid Access Control in SCADA-Enabled IIoT Networks Using Machine Learning.

The recent advancements in the Internet of Things have made it converge towards critical infrastructure automation, opening a new paradigm referred to as the Industrial Internet of Things (IIoT). In the IIoT, different connected devices can send huge amounts of data to other devices back and forth for a better decision-making process. In such use cases, the role of supervisory control and data acquisition (SCADA) has been studied by many researchers in recent years for robust supervisory control management. Nevertheless, for better sustainability of these applications, reliable data exchange is crucial in this domain. To ensure the privacy and integrity of the data shared between the connected devices, access control can be used as the front-line security mechanism for these systems. However, the role engineering and assignment propagation in access control is still a tedious process as its manually performed by network administrators. In this study, we explored the potential of supervised machine learning to automate role engineering for fine-grained access control in Industrial Internet of Things (IIoT) settings. We propose a mapping framework to employ a fine-tuned multilayer feedforward artificial neural network (ANN) and extreme learning machine (ELM) for role engineering in the SCADA-enabled IIoT environment to ensure privacy and user access rights to resources. For the application of machine learning, a thorough comparison between these two algorithms is also presented in terms of their effectiveness and performance. Extensive experiments demonstrated the significant performance of the proposed scheme, which is promising for future research to automate the role assignment in the IIoT domain.

Lithium-Ion Battery State of Charge Estimation Using Deep Neural Network

In an electric vehicle (EV), the battery management system (BMS) is crucial for managing the health and safety of the battery. The accurate estimation of battery state of charge (SOC) offers critical information about the battery’s remaining capacity. The SOC of the battery mainly depends on its non-linear internal parameters, battery chemistry, ambient temperature, aging factor etc. So, accurate SOC estimation is still a significant challenge. Many researchers have developed several model-based methods that are more complex to develop. Another approach is a data-driven based SOC estimation algorithm, which is less complex but requires more data and it may be inaccurate. In this context, this paper presents a robust and accurate SOC estimation algorithm for a Lithium-ion battery using a deep learning feed-forward neural network (DLFFNN) approach. The proposed algorithm accurately characterizes the battery’s non-linear behavior. To develop a robust SOC estimation algorithm, data is collected at different temperatures with 5% error in data (4 mV-voltage, 110 mA-current, 5∘C temperature) is added to battery datasets. The obtained results demonstrated that the performance of the proposed DLFNN is robust and accurate on different drive cycles with 1.14% Root mean squared error (RMSE), 0.66% mean absolute error (MAE), and 6.65% maximum error (MAX).

Deep learning Feedforward Neural Network in predicting model of Environmental risk factors in the Sohar region

AQI (Air Quality Index) is the standard degree that guides us to measure air pollution levels such as PM2.5, O3, NO2, and SO2 to show the state of air quality. Polluted gas causes much damage and problems to people, plants, and the environment because of its negative impact. Data mining successfully examines an enormous cluster of data to recognize associations, determine relations between variables, and predict future values. In this paper, an experimental study was performed on analyzing the previous dataset of (PM2.5 and O3) for accurately predicting AQI using deep learning Feedforward Neural network techniques. The deep learning (Feedforward Neural Network (FFNN) predicting models are employed to evaluate based on R, R², MSE, MAE, and RMSE criteria using historical data from (the Ministry of Environment-Oman). Different epochs and a different number of hidden layers are deployed to improve and boost performance. In FFNN, the epochs number increase by 50,100 and 500 while the hidden layer utilized to 1,5 and 10. This optimization technique exceeds the performance from R=0.892 to R=0.992 in predicting the level of (PM2.5) and the (O3) from R=0.864 to R=0.999. The results show that the Sohar Region in a safe level of AQI.

Drought monitoring by downscaling GRACE-derived terrestrial water storage anomalies: A deep learning approach

The current study proposes a new method to downscale the monthly GRACE-derived Terrestrial Water Storage Anomaly (TWSA) to 10 km spatial resolution over Iran using deep learning methods. First, the growing neural gas (GNG) method is utilized to cluster the TWSA data to find similar patterns. The Silhouette Chou Davies (SCD) validity measure, a combination of three robust cluster validity measures, is used to evaluate the performance of the GNG method. Then, the feasibility of deep learning Convolutional Long Short-Term Memory (ConvLSTM), shallow learning Feed Forward Neural Networks (FFNN), and Random Forest (RF) are examined in downscaling GRACE-derived TWSA using only remote sensing images. The results indicate that the deep learning method outperforms the RF and FFNN models by 7 % and 18 %, respectively. Then, the Ground Water Storage (GWS) data are isolated from TWSA, showing an agreement between the GRACE-derived GWS and ground-based measurements with R2 = 0.76. Additionally, the downscaled TWSA is used to generate annual drought frequency and change detection maps for the GWS from 2002 to 2016. Annual standardized precipitation index (SPI) drought frequency maps are also produced to gain deeper insight into the water resource scarcity of Iran. The results indicate that Iran has experienced water resource overexploitation since 2011 for agricultural activities, while meteorological drought is a trigger and intensifier for the water crisis in Iran. According to the results, Iran experienced exceptional drought in some regions, and the GWS has decreased all over the country. In addition, further analysis of Iran’s water resources is provided until 2022. The current study calls for integrated sustainable water resources management in Iran; otherwise, irreversible water and environmental problems with higher frequencies will be expected in the near future.

A Deep-Learning Approach to Load Modeling in Modern Power Distribution System

Modern Power Distribution Networks (MPDNs) are no longer passive because Distributed Generations (DGs) are integrated with them to enhance system reliability and power quality. For this reason, load modeling has to be updated to capture the new dynamics of active DNs. This paper presents a composite load modeling for a grid-connected photovoltaic (PV) distribution network using the Levenberg-Marquardt algorithm in the deep learning feed-forward neural network approach. Load modeling is constructing a relationship between input excitation(s) and output response(s); it can be used for simulation studies, stability analysis, and control/protection design. A grid-connected PV distribution network was modeled in Matlab/Simulink and generates data for training and model estimation. The estimated model was tested and validated using a laboratory experimental test bed. Results of the model exhibit a good fitness of 99.8% and 97.2% in active and reactive power models respectively during training. While 97.84% and 94.65% respectively were obtained during testing. The estimation errors were found to be 0.0025 and 0.0049 for active and reactive powers respectively with 0.0473 and 0.0701 corresponding errors in testing.

Grafting constructive algorithm in feedforward neural network learning

Grafting constructive algorithm in feedforward neural network learning

An accuracy-maximization learning framework for supervised and semi-supervised imbalanced data

While we attempt to develop the balanced error rate (BER) minimization learning framework for randomized learning of feedforward neural networks to deal with imbalanced datasets, it remains unclear whether the BER minimization learning framework can be effectively extended into its semi-supervised version. This paper proposes a new concept of accuracy maximization for randomized learning methods on imbalanced datasets for the first time, and theoretically proves that it is equivalent to the minimization of the generalized BER for the use of the selected neural networks. In particular, accuracy maximization can be easily extended to semi-supervised scenarios as its semi-supervised version is proved to be linearly dependent on its original. In this paper, based on the proposed accuracy maximization concept, we propose an accuracy-maximization learning framework, and further develop a new accuracy-maximization extreme learning machine AMELM by taking Extreme Learning Machine (ELM) as a typical randomized learning method for feedforward neural networks so as to handle challenging data issues such as the class imbalance and label scarcity. It is worth noting that the proposed accuracy maximization based framework is not only suitable for ELM, but can be extended to different randomized learning methods, such as Random Vector Functional Link Network (RVFL), and Schmidt Neural Network (SNN) for supervised and semi-supervised imbalanced data. The efficacy of AMELM is tested on extensive benchmark datasets. Experimental results show that AMELM can achieve satisfactory performances on labeled or partially labeled imbalanced data. Also, AMELM obtains at least comparable classification performance to other baseline methods yet has fewer hyperparameters to tune, showing its potential for practical applications.

On the evaluation of the interfacial tension of immiscible binary systems of methane, carbon dioxide, and nitrogen-alkanes using robust data-driven approaches

Gas injection has emerged over the recent decades as a promising technology to enhance oil recovery in various fields worldwide. The efficiency and success of a gas injection operation can be assessed through a number of vital experimental studies. Interfacial Tension (IFT) between the injected gas and the displacing fluid is a key parameter playing an eminent role in the foregoing studies. The main scope of this work is making a progress in modeling the IFTs between diverse n-alkanes and Methane (CH4), Carbon Dioxide (CO2), and Nitrogen (N2) natural gases. For this purpose, two smart AI-based approaches of Cascaded Feedforward Neural Network (CFNN) and Decision Tree Learning (DT) were used to simultaneously model the IFTs between foregoing immiscible binary systems as a function of pressure, temperature, the gases properties, and the properties of the liquid. Several statistical measures and graphical descriptions were employed to aid the accuracy analysis of the proposed models. Both developed CFNN and DT networks represented desirable close-to-reality predictions in all binary systems. Besides, CFNN established itself as the most robust model for all studies binary systems with RMSE values of 0.5924, 0.5649, and 0.5870 mN/m, and R2 values of 0.9902, 0.9910, and 0.9904 for the train, test, and overall data, respectively.

A Hybrid Model of Extreme Learning Machine Based on Bat and Cuckoo Search Algorithm for Regression and Multiclass Classification

Extreme learning machine (ELM), as a new simple feedforward neural network learning algorithm, has been extensively used in practical applications because of its good generalization performance and fast learning speed. However, the standard ELM requires more hidden nodes in the application due to the random assignment of hidden layer parameters, which in turn has disadvantages such as poorly hidden layer sparsity, low adjustment ability, and complex network structure. In this paper, we propose a hybrid ELM algorithm based on the bat and cuckoo search algorithm to optimize the input weight and threshold of the ELM algorithm. We test the numerical experimental performance of function approximation and classification problems under a few benchmark datasets; simulation results show that the proposed algorithm can obtain significantly better prediction accuracy compared to similar algorithms.

Controlling a Small Mobile 3-Pi Robot Movement in a Maze Via the Neural Network Using Back-Propagation Learning Method

Abstract The contribution is focused on technical implementation of controlling a small mobile 3Pi robot in a maze along a predefined guide line where the control of the acquired direction of the robot’s movement was provided by a neural network. The weights (memory) of the neuron were calculated using a feedforward neural network learning via the Back-propagation method. This article fastens on the paper by the title “Movement control of a small mobile 3-pi robot in a maze using artificial neural network”, where Hebbian learning was used for a single-layer neural network. The reflectance infra-red sensors performed as input sensors. The result of this research is the evaluation based on the experiments that served to compare different training sets with the learning methods when moving a mobile robot in a maze.

A constructive approach to data-driven randomized learning for feedforward neural networks

There is an issue with the way in which feedforward neural networks with random hidden nodes generate random parameters in order to obtain a good projection space. Typically, random weights and biases are both drawn from the same interval, which is misguided as they have different functions. Recently, more sophisticated methods of random parameters generation have been developed, such as the data-driven approach, where the sigmoids are placed in randomly selected regions of the input space and then their slopes are adjusted to the local fluctuations of the target function. In this work, we propose a new constructive data-driven method that builds iteratively the network architecture. This method successively generates new candidate hidden nodes and accepts them when the training error falls significantly. The threshold of acceptance is adapted throughout training, accepting at the beginning of the training process only those nodes which lead to the largest reductions in error. In the next stages, the threshold is successively reduced to accept only those nodes which model the target function details more accurately. This leads to a more compact network architecture, as it includes only ”significant” nodes. It is worth noting that redundant, random nodes, which are usually generated by existing randomized learning methods, are not accepted by the proposed method. We empirically compared our approach with several alternative methods, including its predecessor, competitive randomized learning solutions, a gradient-based network and a generalized additive model. We found that our proposed approach outperformed its competitors in terms of fitting accuracy.

Topology of Learning in Feedforward Neural Networks.

Understanding how neural networks learn remains one of the central challenges in machine learning research. From random at the start of training, the weights of a neural network evolve in such a way as to be able to perform a variety of tasks, such as classifying images. Here, we study the emergence of structure in the weights by applying methods from topological data analysis. We train simple feedforward neural networks on the MNIST data set and monitor the evolution of the weights. When initialized to zero, the weights follow trajectories that branch off recurrently, thus generating trees that describe the growth of the effective capacity of each layer. When initialized to tiny random values, the weights evolve smoothly along 2-D surfaces. We show that natural coordinates on these learning surfaces correspond to important factors of variation.

An Intelligent Task Scheduling Mechanism for Autonomous Vehicles via Deep Learning

With the rapid development of the Internet of Things (IoT) and artificial intelligence, autonomous vehicles have received much attention in recent years. Safe driving is one of the essential concerns of self-driving cars. The main problem in providing better safe driving requires an efficient inference system for real-time task management and autonomous control. Due to limited battery life and computing power, reducing execution time and resource consumption can be a daunting process. This paper addressed these challenges and developed an intelligent task management system for IoT-based autonomous vehicles. For each task processing, a supervised resource predictor is invoked for optimal hardware cluster selection. Tasks are executed based on the earliest hyper period first (EHF) scheduler to achieve optimal task error rate and schedule length performance. The single-layer feedforward neural network (SLFN) and lightweight learning approaches are designed to distribute each task to the appropriate processor based on their emergency and CPU utilization. We developed this intelligent task management module in python and experimentally tested it on multicore SoCs (Odroid Xu4 and NVIDIA Jetson embedded platforms). Connected Autonomous Vehicles (CAV) and Internet of Medical Things (IoMT) benchmarks are used for training and testing purposes. The proposed modules are validated by observing the task miss rate, resource utilization, and energy consumption metrics compared with state-of-art heuristics. SLFN-EHF task scheduler achieved better results in an average of 98% accuracy, and in an average of 20–27% reduced in execution time and 32–45% in task miss rate metric than conventional methods.