Learning In Feedforward Neural Networks Research Articles

Competitive plasticity to reduce the energetic costs of learning.

The brain is not only constrained by energy needed to fuel computation, but it is also constrained by energy needed to form memories. Experiments have shown that learning simple conditioning tasks which might require only a few synaptic updates, already carries a significant metabolic cost. Yet, learning a task like MNIST to 95% accuracy appears to require at least 108 synaptic updates. Therefore the brain has likely evolved to be able to learn using as little energy as possible. We explored the energy required for learning in feedforward neural networks. Based on a parsimonious energy model, we propose two plasticity restricting algorithms that save energy: 1) only modify synapses with large updates, and 2) restrict plasticity to subsets of synapses that form a path through the network. In biology networks are often much larger than the task requires, yet vanilla backprop prescribes to update all synapses. In particular in this case, large savings can be achieved while only incurring a slightly worse learning time. Thus competitively restricting plasticity helps to save metabolic energy associated to synaptic plasticity. The results might lead to a better understanding of biological plasticity and a better match between artificial and biological learning. Moreover, the algorithms might benefit hardware because also electronic memory storage is energetically costly.

Resonant-mode metasurface thermal super mirror by deep learning-assisted optimization algorithms

A “super-mirror” having ultrahigh infrared reflectance is achieved by an optimized photonic contrast grating metasurface. Finding ways to achieve this exceptional performance can be enabled by implementing global optimization and machine learning elements, such as Bayesian optimization and genetic algorithm. Here, we acquired an optimized grating design made of high-index germanium, which excites resonances that result in ultralow emittance at certain wavelengths. Our optimizations assisted in the discovery of hybridized coupling of Fabry-Pérot modes and guided modes in a monolithic microscale multilayered coating. We demonstrate constraints in the given geometric variable ranges improves the overall performance of algorithms. We also show the enhanced performance of a deep learning Feedforward Neural Network, which is implemented as the inverse design using the network trained with dataset obtained from Bayesian optimization and Genetic Algorithm approaches. The performance of the Feedforward Neural Network-assisted design produced normal emissivity difference by only +3.5 %, with lower sensitivity to grating dimensional parameter variations. The improvement is achieved by predicting and better understanding of the optical physics of resonant gratings. The proposed few-layer grating coating can be applied to space components, enclosures, and vessels to suppress thermal radiative heat loss.

Adjusted SpikeProp algorithm for recurrent spiking neural networks with LIF neurons

A problem related to the development of a supervised learning method for recurrent spiking neural networks is addressed in the paper. The widely used Leaky-Integrate-and-Fire model has been adopted as a spike neuron model. The proposed method is based on a known SpikeProp algorithm. In detail, the developed method enables gradient descent learning of recurrent or multi-layer feedforward spiking neural networks. The research included an extended verification study for the classical XOR classification problem. In addition, the developed learning method has been used to provide a spiking neural black-box model of fast processes occurring in a pressurised water nuclear reactor. The obtained simulation results demonstrate satisfactory effectiveness of the proposed approach.

Emergence and reconfiguration of modular structure for artificial neural networks during continual familiarity detection.

Advances in artificial intelligence enable neural networks to learn a wide variety of tasks, yet our understanding of the learning dynamics of these networks remains limited. Here, we study the temporal dynamics during learning of Hebbian feedforward neural networks in tasks of continual familiarity detection. Drawing inspiration from network neuroscience, we examine the network's dynamic reconfiguration, focusing on how network modules evolve throughout learning. Through a comprehensive assessment involving metrics like network accuracy, modular flexibility, and distribution entropy across diverse learning modes, our approach reveals various previously unknown patterns of network reconfiguration. We find that the emergence of network modularity is a salient predictor of performance and that modularization strengthens with increasing flexibility throughout learning. These insights not only elucidate the nuanced interplay of network modularity, accuracy, and learning dynamics but also bridge our understanding of learning in artificial and biological agents.

The Southern Ocean carbon sink has been overestimated in the past three decades

Employing machine learning methods for mapping surface ocean pCO2 has reduced the uncertainty in estimating sea-air CO2 flux. However, a general discrepancy exists between the Southern Ocean carbon sinks derived from pCO2 products and those from biogeochemistry models. Here, by performing a boosting ensemble learning feed-forward neural networks method, we have identified an underestimation of the surface Southern Ocean pCO2 due to notably uneven density of pCO2 measurements between summer and winter, which resulted in about 16% overestimating of Southern Ocean carbon sink over the past three decades. In particular, the Southern Ocean carbon sink since 2010 was notably overestimated by approximately 29%. This overestimation can be mitigated by a winter correction in algorithms, with the average Southern Ocean carbon sink during 1992-2021 corrected to −0.87 PgC yr−1 from the original −1.01 PgC yr−1. Furthermore, the most notable underestimation of surface ocean pCO2 mainly occurred in regions south of 60°S and was hiding under ice cover. As the surface ocean pCO2 under sea ice coverage in the winter is much higher than the atmosphere, if sea ice melts completely, there could be a further reduction of about 0.14 PgC yr−1 in the Southern Ocean carbon sink.

Intelligent Lighting Control Systems for Energy Savings in Hospital Buildings Using Artificial Neural Networks

Lighting control systems are essential in modern building automation and smart homes, efficiently managing illumination to enhance energy conservation and user comfort. This project tackles energy consumption challenges in hospital buildings by introducing Intelligent Lighting Control Systems (ILCS) that take natural light and occupancy into account, driven by Artificial Neural Networks (ANN) and diverse machine learning algorithms. In our study, we collected sensor data, processed it, and designed a lighting control system employing a feedforward neural network and various machine learning algorithms. Surprisingly, our research found that a linear regression algorithm surpassed the ANN-based system in this context. We implemented a prototype, tested it on hardware, and obtained the expected results. This research marks progress towards optimizing energy use in hospital buildings and contributing to sustainability endeavors. By combining ILCS and machine learning, it offers a promising approach for more efficient and eco-friendly lighting systems

Fossil energy market price prediction by using machine learning with optimal hyper-parameters: A comparative study

Fossil energy markets are important commodities, and their price fluctuations impact worldwide economy and financial markets. Hence, it is essential to forecast the prices of fossil energy commodities. In this study, various machine learning systems are optimized by using Bayesian optimization method and implemented to forecast prices in 12 different fossil energy markets including crude oil, natural gas, propane, kerosene, gasoline, heating oil, and coal based on price information from all fossil energy markets. The optimized machine learning systems considered in modeling and simulations are Gaussian regression process, support vector regression, regression trees, k-nearest neighbor algorithm, and deep feedforward neural networks. The simulation results show that Gaussian regression process is the best system to forecast fossil energy markets. In addition, the deep learning feedforward neural networks system yields to stable forecasts. Furthermore, the prediction of the prices in natural gas, coal, and propane markets is a difficult task compared to the prediction of the prices in crude oil markets. The understanding of the effectiveness of machine learning systems in pricing different fossil energy markets can help international economic agents establish appropriate investment strategies.

Topological learning in multiclass data sets.

We specialize techniques from topological data analysis to the problem of characterizing the topological complexity (as defined in the body of the paper) of a multiclass data set. As a by-product, a topological classifier is defined that uses an open subcovering of the data set. This subcovering can be used to construct a simplicial complex whose topological features (e.g., Betti numbers) provide information about the classification problem. We use these topological constructs to study the impact of topological complexity on learning in feedforward deep neural networks (DNNs). We hypothesize that topological complexity is negatively correlated with the ability of a fully connected feedforward deep neural network to learn to classify data correctly. We evaluate our topological classification algorithm on multiple constructed and open-source data sets. We also validate our hypothesis regarding the relationship between topological complexity and learning in DNN's on multiple data sets.

A deep learning feed-forward neural network framework for the solutions to singularly perturbed delay differential equations

In this paper, we explore a deep learning feedforward artificial neural network (ANN) framework as a numerical tool for approximating the solutions to singularly perturbed delay differential equations (SPDDE). Our approach trains the network with fewer uniform data points along with linear interpolation as a dependent variable. More importantly, the exact solution is not used for training the network. A mean square or Euclidean norm type total loss function is utilized to assess the deep learning network’s performance. In the training and testing phase, our network adjusts itself to fit the tangent and the curvature by minimizing the total loss function and fine-tuning the network’s hyperparameters during the backpropagation stage. Our focus in this contribution is to investigate an answer to the question- “whether neural networks can learn to solve the differential equations with both delay and boundary layer behavior”? Traditional numerical methods are known to perform poorly in approximating the solutions to SPDDE due to boundary layer behavior, which is the sharp change in the gradient of the solution inside a region of very small width. Our proposed network architecture is used to investigate the above question for three varieties of SPDDE. The numerical results demonstrate that the fine-tuned adaptive deep learning architecture can effectively approximate the solution to SPDDE for varieties of delay and perturbation parameters. It is expected that the current method can be extendable to approximate 2D convection-diffusion partial differential equations with a boundary layer.

Machine learning-based prediction of shear strength of steel reinforced concrete columns subjected to axial compressive load and seismic lateral load

The shear strength of steel reinforced concrete (SRC) columns is a crucial basis in their seismic design and evaluation of structural performance. Most of the existing research predicts the lateral load carrying capacity based on regression equations, but lacks accuracy. To this end, in this study, Gaussian process regression (GPR), Least Squares Boosting (LSBoosting), Support Vector Regression (SVR), Feedforward Neural Network (FNN) and other machine learning (ML) algorithms were employed to develop the shear strength model of SRC columns subjected to axial compressive load and seismic lateral load. A large experimental database with 395 samples was established to train 11 input features. The original database was divided into 5 datasets based on the mixed, flexural, flexural-shear, shear, and bond failure modes. The failure mechanism factors were introduced for better evaluation. Specifically, an effective data splitting strategy-bootstrapping was used to train the models, and Bayesian optimization was attempted to improve the prediction accuracy. Results demonstrate that GPR has the highest prediction accuracy with the most failure modes, and the factors affecting the shear strength are explained reasonably by correlation analysis and Partial Dependence Plot. Through comprehensive comparison, the GPR model developed in this paper is advanced in predicting the shear strength of SRC columns.

Automatic Hybrid Access Control in SCADA-Enabled IIoT Networks Using Machine Learning.

The recent advancements in the Internet of Things have made it converge towards critical infrastructure automation, opening a new paradigm referred to as the Industrial Internet of Things (IIoT). In the IIoT, different connected devices can send huge amounts of data to other devices back and forth for a better decision-making process. In such use cases, the role of supervisory control and data acquisition (SCADA) has been studied by many researchers in recent years for robust supervisory control management. Nevertheless, for better sustainability of these applications, reliable data exchange is crucial in this domain. To ensure the privacy and integrity of the data shared between the connected devices, access control can be used as the front-line security mechanism for these systems. However, the role engineering and assignment propagation in access control is still a tedious process as its manually performed by network administrators. In this study, we explored the potential of supervised machine learning to automate role engineering for fine-grained access control in Industrial Internet of Things (IIoT) settings. We propose a mapping framework to employ a fine-tuned multilayer feedforward artificial neural network (ANN) and extreme learning machine (ELM) for role engineering in the SCADA-enabled IIoT environment to ensure privacy and user access rights to resources. For the application of machine learning, a thorough comparison between these two algorithms is also presented in terms of their effectiveness and performance. Extensive experiments demonstrated the significant performance of the proposed scheme, which is promising for future research to automate the role assignment in the IIoT domain.

Lithium-Ion Battery State of Charge Estimation Using Deep Neural Network

In an electric vehicle (EV), the battery management system (BMS) is crucial for managing the health and safety of the battery. The accurate estimation of battery state of charge (SOC) offers critical information about the battery’s remaining capacity. The SOC of the battery mainly depends on its non-linear internal parameters, battery chemistry, ambient temperature, aging factor etc. So, accurate SOC estimation is still a significant challenge. Many researchers have developed several model-based methods that are more complex to develop. Another approach is a data-driven based SOC estimation algorithm, which is less complex but requires more data and it may be inaccurate. In this context, this paper presents a robust and accurate SOC estimation algorithm for a Lithium-ion battery using a deep learning feed-forward neural network (DLFFNN) approach. The proposed algorithm accurately characterizes the battery’s non-linear behavior. To develop a robust SOC estimation algorithm, data is collected at different temperatures with 5% error in data (4 mV-voltage, 110 mA-current, 5∘C temperature) is added to battery datasets. The obtained results demonstrated that the performance of the proposed DLFNN is robust and accurate on different drive cycles with 1.14% Root mean squared error (RMSE), 0.66% mean absolute error (MAE), and 6.65% maximum error (MAX).

Deep learning Feedforward Neural Network in predicting model of Environmental risk factors in the Sohar region

AQI (Air Quality Index) is the standard degree that guides us to measure air pollution levels such as PM2.5, O3, NO2, and SO2 to show the state of air quality. Polluted gas causes much damage and problems to people, plants, and the environment because of its negative impact. Data mining successfully examines an enormous cluster of data to recognize associations, determine relations between variables, and predict future values. In this paper, an experimental study was performed on analyzing the previous dataset of (PM2.5 and O3) for accurately predicting AQI using deep learning Feedforward Neural network techniques. The deep learning (Feedforward Neural Network (FFNN) predicting models are employed to evaluate based on R, R², MSE, MAE, and RMSE criteria using historical data from (the Ministry of Environment-Oman). Different epochs and a different number of hidden layers are deployed to improve and boost performance. In FFNN, the epochs number increase by 50,100 and 500 while the hidden layer utilized to 1,5 and 10. This optimization technique exceeds the performance from R=0.892 to R=0.992 in predicting the level of (PM2.5) and the (O3) from R=0.864 to R=0.999. The results show that the Sohar Region in a safe level of AQI.

Drought monitoring by downscaling GRACE-derived terrestrial water storage anomalies: A deep learning approach

The current study proposes a new method to downscale the monthly GRACE-derived Terrestrial Water Storage Anomaly (TWSA) to 10 km spatial resolution over Iran using deep learning methods. First, the growing neural gas (GNG) method is utilized to cluster the TWSA data to find similar patterns. The Silhouette Chou Davies (SCD) validity measure, a combination of three robust cluster validity measures, is used to evaluate the performance of the GNG method. Then, the feasibility of deep learning Convolutional Long Short-Term Memory (ConvLSTM), shallow learning Feed Forward Neural Networks (FFNN), and Random Forest (RF) are examined in downscaling GRACE-derived TWSA using only remote sensing images. The results indicate that the deep learning method outperforms the RF and FFNN models by 7 % and 18 %, respectively. Then, the Ground Water Storage (GWS) data are isolated from TWSA, showing an agreement between the GRACE-derived GWS and ground-based measurements with R2 = 0.76. Additionally, the downscaled TWSA is used to generate annual drought frequency and change detection maps for the GWS from 2002 to 2016. Annual standardized precipitation index (SPI) drought frequency maps are also produced to gain deeper insight into the water resource scarcity of Iran. The results indicate that Iran has experienced water resource overexploitation since 2011 for agricultural activities, while meteorological drought is a trigger and intensifier for the water crisis in Iran. According to the results, Iran experienced exceptional drought in some regions, and the GWS has decreased all over the country. In addition, further analysis of Iran’s water resources is provided until 2022. The current study calls for integrated sustainable water resources management in Iran; otherwise, irreversible water and environmental problems with higher frequencies will be expected in the near future.

A Deep-Learning Approach to Load Modeling in Modern Power Distribution System

Modern Power Distribution Networks (MPDNs) are no longer passive because Distributed Generations (DGs) are integrated with them to enhance system reliability and power quality. For this reason, load modeling has to be updated to capture the new dynamics of active DNs. This paper presents a composite load modeling for a grid-connected photovoltaic (PV) distribution network using the Levenberg-Marquardt algorithm in the deep learning feed-forward neural network approach. Load modeling is constructing a relationship between input excitation(s) and output response(s); it can be used for simulation studies, stability analysis, and control/protection design. A grid-connected PV distribution network was modeled in Matlab/Simulink and generates data for training and model estimation. The estimated model was tested and validated using a laboratory experimental test bed. Results of the model exhibit a good fitness of 99.8% and 97.2% in active and reactive power models respectively during training. While 97.84% and 94.65% respectively were obtained during testing. The estimation errors were found to be 0.0025 and 0.0049 for active and reactive powers respectively with 0.0473 and 0.0701 corresponding errors in testing.

Grafting constructive algorithm in feedforward neural network learning

Grafting constructive algorithm in feedforward neural network learning

An accuracy-maximization learning framework for supervised and semi-supervised imbalanced data

While we attempt to develop the balanced error rate (BER) minimization learning framework for randomized learning of feedforward neural networks to deal with imbalanced datasets, it remains unclear whether the BER minimization learning framework can be effectively extended into its semi-supervised version. This paper proposes a new concept of accuracy maximization for randomized learning methods on imbalanced datasets for the first time, and theoretically proves that it is equivalent to the minimization of the generalized BER for the use of the selected neural networks. In particular, accuracy maximization can be easily extended to semi-supervised scenarios as its semi-supervised version is proved to be linearly dependent on its original. In this paper, based on the proposed accuracy maximization concept, we propose an accuracy-maximization learning framework, and further develop a new accuracy-maximization extreme learning machine AMELM by taking Extreme Learning Machine (ELM) as a typical randomized learning method for feedforward neural networks so as to handle challenging data issues such as the class imbalance and label scarcity. It is worth noting that the proposed accuracy maximization based framework is not only suitable for ELM, but can be extended to different randomized learning methods, such as Random Vector Functional Link Network (RVFL), and Schmidt Neural Network (SNN) for supervised and semi-supervised imbalanced data. The efficacy of AMELM is tested on extensive benchmark datasets. Experimental results show that AMELM can achieve satisfactory performances on labeled or partially labeled imbalanced data. Also, AMELM obtains at least comparable classification performance to other baseline methods yet has fewer hyperparameters to tune, showing its potential for practical applications.

On the evaluation of the interfacial tension of immiscible binary systems of methane, carbon dioxide, and nitrogen-alkanes using robust data-driven approaches

Gas injection has emerged over the recent decades as a promising technology to enhance oil recovery in various fields worldwide. The efficiency and success of a gas injection operation can be assessed through a number of vital experimental studies. Interfacial Tension (IFT) between the injected gas and the displacing fluid is a key parameter playing an eminent role in the foregoing studies. The main scope of this work is making a progress in modeling the IFTs between diverse n-alkanes and Methane (CH4), Carbon Dioxide (CO2), and Nitrogen (N2) natural gases. For this purpose, two smart AI-based approaches of Cascaded Feedforward Neural Network (CFNN) and Decision Tree Learning (DT) were used to simultaneously model the IFTs between foregoing immiscible binary systems as a function of pressure, temperature, the gases properties, and the properties of the liquid. Several statistical measures and graphical descriptions were employed to aid the accuracy analysis of the proposed models. Both developed CFNN and DT networks represented desirable close-to-reality predictions in all binary systems. Besides, CFNN established itself as the most robust model for all studies binary systems with RMSE values of 0.5924, 0.5649, and 0.5870 mN/m, and R2 values of 0.9902, 0.9910, and 0.9904 for the train, test, and overall data, respectively.

A Hybrid Model of Extreme Learning Machine Based on Bat and Cuckoo Search Algorithm for Regression and Multiclass Classification

Extreme learning machine (ELM), as a new simple feedforward neural network learning algorithm, has been extensively used in practical applications because of its good generalization performance and fast learning speed. However, the standard ELM requires more hidden nodes in the application due to the random assignment of hidden layer parameters, which in turn has disadvantages such as poorly hidden layer sparsity, low adjustment ability, and complex network structure. In this paper, we propose a hybrid ELM algorithm based on the bat and cuckoo search algorithm to optimize the input weight and threshold of the ELM algorithm. We test the numerical experimental performance of function approximation and classification problems under a few benchmark datasets; simulation results show that the proposed algorithm can obtain significantly better prediction accuracy compared to similar algorithms.

Controlling a Small Mobile 3-Pi Robot Movement in a Maze Via the Neural Network Using Back-Propagation Learning Method

Abstract The contribution is focused on technical implementation of controlling a small mobile 3Pi robot in a maze along a predefined guide line where the control of the acquired direction of the robot’s movement was provided by a neural network. The weights (memory) of the neuron were calculated using a feedforward neural network learning via the Back-propagation method. This article fastens on the paper by the title “Movement control of a small mobile 3-pi robot in a maze using artificial neural network”, where Hebbian learning was used for a single-layer neural network. The reflectance infra-red sensors performed as input sensors. The result of this research is the evaluation based on the experiments that served to compare different training sets with the learning methods when moving a mobile robot in a maze.