Traditional Learning Algorithms Research Articles

A review of research on reinforcement learning algorithms for multi-agents

In recent years, multi-agent reinforcement learning techniques have been widely used and evolved in the field of artificial intelligence. However, traditional reinforcement learning methods have limitations such as long training time, large sample data requirements, and highly delayed rewards. Therefore, this paper systematically and specifically studies the MARL algorithm. Firstly, this paper uses Citespace software to visually analyze the existing literature on multi-agent reinforcement learning and briefly indicates the research hotspots and key research directions in this field. Secondly, the applications of traditional reinforcement learning algorithms under two task objects, namely single-agent and multi-agent systems, are described in detail. Then, the paper highlights the diverse applications, challenges, and corresponding solutions of MARL algorithmic techniques in the field of MAS. Finally, the paper points out future research directions based on the existing limitations of the algorithm. Through this paper, readers will gain a systematic and in-depth understanding of MARL algorithms and how they can be utilized to better address the various challenges posed by MAS.

An Improved Dyna-Q Algorithm Inspired by the Forward Prediction Mechanism in the Rat Brain for Mobile Robot Path Planning.

The traditional Model-Based Reinforcement Learning (MBRL) algorithm has high computational cost, poor convergence, and poor performance in robot spatial cognition and navigation tasks, and it cannot fully explain the ability of animals to quickly adapt to environmental changes and learn a variety of complex tasks. Studies have shown that vicarious trial and error (VTE) and the hippocampus forward prediction mechanism in rats and other mammals can be used as key components of action selection in MBRL to support "goal-oriented" behavior. Therefore, we propose an improved Dyna-Q algorithm inspired by the forward prediction mechanism of the hippocampus to solve the above problems and tackle the exploration-exploitation dilemma of Reinforcement Learning (RL). This algorithm alternately presents the potential path in the future for mobile robots and dynamically adjusts the sweep length according to the decision certainty, so as to determine action selection. We test the performance of the algorithm in a two-dimensional maze environment with static and dynamic obstacles, respectively. Compared with classic RL algorithms like State-Action-Reward-State-Action (SARSA) and Dyna-Q, the algorithm can speed up spatial cognition and improve the global search ability of path planning. In addition, our method reflects key features of how the brain organizes MBRL to effectively solve difficult tasks such as navigation, and it provides a new idea for spatial cognitive tasks from a biological perspective.

SFL-TUM: Energy efficient SFRL method for large scale AI model's task offloading in UAV-assisted MEC networks

The convergence of mobile edge computing (MEC) network with unmanned aerial vehicles (UAVs) presents an auspicious opportunity to revolutionize wireless communication and facilitate high-speed internet access in remote regions for mobile devices (MDs) as well as large scale artificial intelligence (AI) models. However, the substantial amount of data produced by the UAVs-assisted MEC network necessitates the integration of efficient distributed learning techniques in AI models. In recent times, distributed learning algorithms, including federated reinforcement learning (FRL) and split learning (SL), have been explored for the purpose of learning machine learning (ML) models that are distributed by sharing model parameters, as opposed to large raw data-sets as seen in traditional centralized learning algorithms. To implement the hybrid method, the model is first trained locally on each UAV-assisted MEC network using SL. Subsequently, the model parameters that have been encrypted are sent to a central server for federated averaging. Finally, after the model has been updated, it is distributed to each UAV-assisted MEC network for local fine-tuning. Our simulations indicate that the proposed split and federated reinforcement learning (SFRL) framework yields comparable high-test accuracy performance while consuming less energy compared to extant distributed learning algorithms. Furthermore, the SFRL algorithm efficiently realizes energy-efficient selection between the SL and FRL methods under different distributions. Numerical results shows that the proposed scheme improves the accuracy by 29.31% and reduced the energy consumption by around 67.34% and time delay by about 7.37%. as compared to the existing baseline schemes.

A Static Security Region Analysis of New Power Systems Based on Improved Stochastic–Batch Gradient Pile Descent

The uncertainty in the new power system has increased, leading to limitations in traditional stability analysis methods. Therefore, considering the perspective of the three-dimensional static security region (SSR), we propose a novel approach for system static stability analysis. To address the slow training speed of traditional deep learning algorithms using batch gradient descent (BGD), we introduce an improved stochastic–batch gradient descent (S-BGD) search method that combines the advantages of stochastic gradient descent (SGD) in fast training. This method ensures both speed and precision in parameter training. Moreover, to tackle the problem of getting trapped in local optima and saddle points during parameter training, we draw inspiration from kinematic theory and propose a gradient pile (GP) training method. By utilizing accumulated gradients as parameter corrections, this method effectively avoids getting stuck in local optima and saddle points, thereby enhancing precision. Finally, we apply the proposed methods to construct the static security region for the IEEE-118 new power system using its data as samples, demonstrating the effectiveness of our approach.

An improved deep hashing model for image retrieval with binary code similarities

The exponential growth of data raises an unprecedented challenge in data analysis: how to retrieve interesting information from such large-scale data. Hash learning is a promising solution to address this challenge, because it may bring many potential advantages, such as extremely high efficiency and low storage cost, after projecting high-dimensional data to compact binary codes. However, traditional hash learning algorithms often suffer from the problem of semantic inconsistency, where images with similar semantic features may have different binary codes. In this paper, we propose a novel end-to-end deep hashing method based on the similarities of binary codes, dubbed CSDH (Code Similarity-based Deep Hashing), for image retrieval. Specifically, it extracts deep features from images to capture semantic information using a pre-trained deep convolutional neural network. Additionally, a hidden and fully connected layer is attached at the end of the deep network to derive hash bits by virtue of an activation function. To preserve the semantic consistency of images, a loss function has been introduced. It takes the label similarities, as well as the Hamming embedding distances, into consideration. By doing so, CSDH can learn more compact and powerful hash codes, which not only can preserve semantic similarity but also have small Hamming distances between similar images. To verify the effectiveness of CSDH, we evaluate CSDH on two public benchmark image collections, i.e., CIFAR-10 and NUS-WIDE, with five classic shallow hashing models and six popular deep hashing ones. The experimental results show that CSDH can achieve competitive performance to the popular deep hashing algorithms.

New ensemble learning algorithm based on classification certainty and semantic correlation

A new bootstrap-aggregating (bagging) ensemble learning algorithm is proposed based on classification certainty and semantic correlation to improve the classification accuracy of ensemble learning. First, two predetermined thresholds are introduced to construct the long and short-text sample subsets, and different deep learning methods are compared to construct the optimal base classifier groups for each sample subsets. Then, the random sampling method employed in traditional bagging classification algorithms is improved, and a threshold group based random sampling method is proposed to obtain long and short training sample subsets of each iteration. Finally, the sample classification certainty of the base classifiers for different categories is defined, and the semantic correlation information is integrated with the traditional weighted voting classifier ensemble method to avoid the loss of important information during the sampling process. The experimental results on multiple datasets demonstrate that the algorithm significantly improves text classification accuracy and outperforms typical deep learning algorithms. The proposed algorithm achieves the improvements of approximately 0.082, 0.061 and 0.019 on CNews dataset when the F1 measurement is used over the traditional ensemble learning algorithms such as random forest, M_ADA_A_SMV and CNN_SVM_LR. Moreover, it achieves the best F1 values of 0.995, 0.985, and 0.989 on the datasets of Spam, CNews, and SogouCS datasets, respectively, when compared with the ensemble learning algorithms using different base classifiers.

Energy Management Strategy Based on Reinforcement Learning and Frequency Decoupling for Fuel Cell Hybrid Powertrain

This study presents a Two-Layer Deep Deterministic Policy Gradient (TL-DDPG) energy management strategy for Hydrogen fuel cell hybrid train, that aims to solve the problem that traditional reinforcement learning strategies require high initial values and are difficult to optimize global variables. Augmenting the optimization capabilities of the inner layer, a frequency decoupling algorithm integrates into the outer layer, furnishing a fitting initial value for strategy optimization. This addition aims to bolster the stability of fuel cell output, thereby enhancing the overall efficiency of the hybrid power system. In comparison with the traditional reinforcement learning algorithm, the proposed approach demonstrates notable improvements: a reduction in hydrogen consumption per 100 km by 16.3 kg, a 9.7% increase in the output power stability of the fuel cell, and a 1.8% enhancement in its efficiency.

Advancing spacecraft rendezvous and docking through safety reinforcement learning and ubiquitous learning principles

As spacecraft rendezvous and docking missions become increasingly complex, the need for advanced solutions has surged. In recent years, the application of reinforcement learning techniques to tackle spacecraft rendezvous guidance challenges has emerged as a prominent international trend. Vital to ensuring the secure rendezvous and docking of spacecraft is the task of obstacle avoidance. However, traditional reinforcement learning algorithms lack the ability to enforce safety constraints within the exploration space, which presents a formidable obstacle in the design of spacecraft rendezvous guidance strategies. In response to this challenge, a spacecraft rendezvous guidance methodology founded on safety reinforcement learning is proposed. Firstly, a Markov model is crafted for autonomous spacecraft rendezvous in scenarios involving collision avoidance. A reward system, contingent on obstacle warnings and collision avoidance constraints, is introduced to establish a safety reinforcement learning framework for devising spacecraft rendezvous guidance strategies. Secondly, within the framework of safety reinforcement learning, two deep reinforcement learning (DRL) algorithms, Proximal Policy Optimisation (PPO) and Deep Deterministic Policy Gradient (DDPG), are leveraged to generate these guidance strategies. Experimental findings validate the effectiveness of this approach in successfully executing obstacle avoidance and achieving rendezvous with remarkable precision. Furthermore, through an analysis of the performance and generalization capabilities of these two algorithms, the efficacy of the proposed methodology is further underscored. This fusion of advanced space guidance technology with the principles of Ubiquitous Learning marks a significant step forward in the quest for safer and more efficient spacecraft rendezvous and docking operations.

Short-term load prediction with LSTM and FCNN models Based on attention mechanisms

To address the significant load prediction discrepancies in microgrid economic dispatch, which result from substantial load fluctuations, a method for near-term load forecasting leveraging the attention mechanism is proposed, utilizing both LSTM and FCNN models. In this method, the attention mechanism is introduced to enhance the LSTM model’s ability to recognize and utilize key information, and the LSTM is used to process historical load data with time continuity selected by a sliding window, whereas the FCNN is used to process daily static features, such as the day’s maximum temperature, minimum temperature, and weather conditions. The outputs of the two networks are subsequently fused into a fully connected layer to generate the final load forecast. Training leverages six months of actual power load values and regional meteorological factor data. The forecast outcomes are then evaluated against both traditional and machine learning algorithms. The experimental findings highlight the superior prediction accuracy of the proposed model.

Content Modeling in Smart Learning Environments: A systematic literature review

Educational content has become a key element for improving the quality and effectiveness of teaching. Many studies have been conducted on user and knowledge modeling using machine-learning algorithms in smart-learning environments. However, few studies have focused on content modeling to estimate content indicators based on student interaction. This study presents a systematic literature review of content modeling using machine learning algorithms in smart learning environments. Two databases were used: Scopus and Web of Science (WoS), with studies conducted until August 2023. In addition, a manual search was performed at conferences and in relevant journals in the area. The results showed that assessment was the most used content in the papers, with difficulty and discrimination as the most common indicators. Item Response Theory (IRT) is the most commonly used technique; however, some studies have used different traditional learning algorithms such as Random Forest, Neural Networks, and Regression. Other indicators, such as time, grade, and number of attempts, were also estimated. Owing to the few studies on content modeling using machine learning algorithms based on interactions, this study presents new lines of research based on the results obtained in the literature review.

Based on an Overview of Crack Detection

This Crack detection, as an important part of concrete structural health monitoring, aims to reflect the stress state and damage degree of the structure. Traditional concrete crack detection mainly relies on manual visual identification, which has low detection efficiency and accuracy. In addition, manual visual detection has problems such as being greatly affected by lighting conditions, being unable to cover overhead locations such as bridge towers and high piers, and being highly subjective. In recent years, in order to solve the above problems, researchers at home and abroad have developed concrete crack detection equipment based on digital image technology, such as inspection vehicles equipped with high-resolution cameras, unmanned aerial vehicles, crawling robots, and so on. Meanwhile, efficient and accurate crack detection algorithms are the basis for realizing accurate crack identification. How to balance the detection speed and accuracy has been the focus of academic attention. This paper reviews the research progress of concrete crack detection equipment based on digital image technology at home and abroad in recent years, including camera platform, calibration method, preprocessing, traditional and deep learning algorithms, crack feature extraction, image stitching, and 3D representation and monitoring of cracks. In addition, the deficiencies in the research are summarized.

Learning strategies for underwater robot autonomous manipulation control

Autonomous manipulation operations represent the high intelligent coordination from robotic vision and control, it is also a symbol of the advances of robotic intelligence. The limitations of visual sensing and the increasingly complex experimental conditions make autonomous manipulation operations more difficult, particularly for deep reinforcement learning methods, which can enhance robotic control intelligence but require a lot of training process. Due to the high-dimensional continuous state space and continuous action space characteristics of underwater operations, this paper adopts a policy-based reinforcement learning method as the foundational approach. To address the issues of instability and low convergence efficiency in traditional policy-based reinforcement learning algorithms during the learning process, this paper proposes a novel policy learning method. This method adopts the Proximal Policy Optimization algorithm (PPOClip) and optimizes it through an actor-critic network. The aim is to improve the stability and effectiveness of convergence in the learning process. In the underwater training environment, a new reward shaping scheme has been designed to address the issue of reward sparsity during the training process. The manually crafted dense reward function is utilized as attractive and repulsive potential functions for goal manipulation and obstacle avoidance. On the highly complex underwater manipulation and training environment, transferred learning algorithm has been established to reduce the training times and compensate the differences between the simulation and experiment. Simulations and tank experiments have verified the performance of the proposed strategy learning method.

Novel cloud-AIoT fault diagnosis for industrial diesel generators based hybrid deep learning CNN-BGRU algorithm

The industrial diesel generator is essential to provide electric power during emergencies and requires accurate failure diagnosis in preventative maintenance services. This research presents a novel AIoT system-based hybrid deep learning algorithm for online monitoring of the anomaly conditions of 125 kW/250 kW industrial generators. Firstly, the IoT modules with different precise sensors are developed to be installed on 125 kW and 250 kW industrial generators to simulate other faults conditions in the laboratory. Then, the supervised learning operations are deployed to train deep learning algorithms with collected labeled data. The proposed algorithm, a hybrid convolution neural network bidirectional gate recurrent unit (CNN-BGRU), is deployed to extract deep features from long sequential historical signals and classify the anomaly conditions. The experiments in different 10 s, 20 s, and 30 s sampling frequencies are performed to evaluate and analyze with various benchmarks, which demonstrate efficiencies of the proposed CNN-BGRU method with other traditional deep learning algorithms, including RNN, CNN, GRU, LSTM, BRNN, and BGRU methods. The proposed hybrid method accomplishes the supreme augmentation of 29.43 % loss, 36.55 % validating loss, 25.86 % MAE, 27.00 % validating MAE, 16.78 % MSE, 20.64 % validating MSE, 6.36 % Pre, 25.34 % validating Pre, 17.30 % Rec, and 26.91 % validating Rec compared with state-of-the-art approaches. The AIoT system-based hybrid CNN-BGRU algorithm provides better improvement and higher accuracy in fault diagnosis of firefighting pumps for maintenance service in Industry 4.0.

IInception-CBAM-IBiGRU based fault diagnosis method for asynchronous motors

Aiming at the problems of insufficient extraction of asynchronous motor fault features by traditional deep learning algorithms and poor diagnosis of asynchronous motor faults in robust noise environments, this paper proposes an end-to-end fault diagnosis method for asynchronous motors based on IInception-CBAM-IBiGRU. The method first uses a signal-to-grayscale image conversion method to convert one-dimensional vibration signals into two-dimensional images and initially extracts shallow features through two-dimensional convolution; then the Improved Inception (IInception) module is used as a residual block to learning features at different scales with a residual structure, and extracts its important feature information through the Convolutional Block Attention Module (CBAM) to extract important feature information and adjust the weight parameters; then the feature information is input to the Improved Bi-directional Gate Recurrent Unit (IBiGRU) to extract its timing features further; finally, the fault identification is achieved by the SoftMax function. The primary hyperparameters in the model are optimized by the Weighted Mean Of Vectors Algorithm (INFO). The experimental results show that the method is effective in fault diagnosis of asynchronous motors, with an accuracy rate close to 100%, and can still maintain a high accuracy rate under the condition of low noise ratio, with good robustness and generalization ability.

Can Generative AI Models Extract Deeper Sentiments as Compared to Traditional Deep Learning Algorithms?

Can Generative AI Models Extract Deeper Sentiments as Compared to Traditional Deep Learning Algorithms?

Stretchable spring‐sheathed yarn sensor for 3D dynamic body reconstruction assisted by transfer learning

AbstractA wearable sensing system that can reconstruct dynamic 3D human body models for virtual cloth fitting is highly important in the era of information and metaverse. However, few research has been conducted regarding conformal sensors for accurately measuring the human body circumferences for dynamic 3D human body reshaping. Here, we develop a stretchable spring‐sheathed yarn sensor (SSYS) as a smart ruler, for precisely measuring the circumference of human bodies and long‐term tracking the movement for the dynamic 3D body reconstruction. The SSYS has a robust property, high resilience, high stability (>18 000), and ultrafast response (12 ms) to external deformation. It is also washable, wearable, tailorable, and durable for long‐time wearing. Moreover, geometric, and mechanical behaviors of the SSYS are systematically investigated both theoretically and experimentally. In addition, a transfer learning algorithm that bridges the discrepancy of real and virtual sensing performance is developed, enabling a small body circumference measurement error of 1.79%, noticeably lower than that of traditional learning algorithm. Furtherly, 3D human bodies that are numerically consistent with the actual bodies are reconstructed. The 3D dynamic human body reconstruction based on the wearing sensing system and transfer learning algorithm enables excellent virtual fitting and shirt customization in a smart and highly efficient manner. This wearable sensing technology shows great potential in human‐computer interaction, intelligent fitting, specialized protection, sports activities, and human physiological health tracking.image

Duty Cycle Scheduling in Wireless Sensor Networks Using an Exploratory Strategy-Directed MADDPG Algorithm

This paper presents an in-depth study of the application of Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithms with an exploratory strategy for duty cycle scheduling (DCS) in the wireless sensor networks (WSNs). The focus is on optimizing the performance of sensor nodes in terms of energy efficiency and event detection rates under varying environmental conditions. Through a series of simulations, we investigate the impact of key parameters such as the sensor specificity constant α and the Poisson rate of events on the learning and operational efficacy of sensor nodes. Our results demonstrate that the MADDPG algorithm with an exploratory strategy outperforms traditional reinforcement learning algorithms, particularly in environments characterized by high event rates and the need for precise energy management. The exploratory strategy enables a more effective balance between exploration and exploitation, leading to improved policy learning and adaptation in dynamic and uncertain environments. Furthermore, we explore the sensitivity of different algorithms to the tuning of the sensor specificity constant α, revealing that lower values generally yield better performance by reducing energy consumption without significantly compromising event detection. The study also examines the algorithms' robustness against the variability introduced by different event Poisson rates, emphasizing the importance of algorithm selection and parameter tuning in practical WSN applications. The insights gained from this research provide valuable guidelines for the deployment of sensor networks in real-world scenarios, where the trade-off between energy consumption and event detection is critical. Our findings suggest that the integration of exploratory strategies in MADDPG algorithms can significantly enhance the performance and reliability of sensor nodes in WSNs.

Integrating tabular data through image conversion for enhanced diagnosis: A novel intelligent decision support system for stratifying obstructive sleep apnoea patients using convolutional neural networks.

High-dimensional databases make it difficult to apply traditional learning algorithms to biomedical applications. Recent developments in computer technology have introduced deep learning (DL) as a potential solution to these difficulties. This study presents a novel intelligent decision support system based on a novel interpretation of data formalisation from tabular data in DL techniques. Once defined, it is used to diagnose the severity of obstructive sleep apnoea, distinguishing between moderate to severe and mild/no cases. The study uses a complete database extract from electronic health records of 2472 patients, including anthropometric data, habits, medications, comorbidities, and patient-reported symptoms. The novelty of this methodology lies in the initial processing of the patients' data, which is formalised into images. These images are then used as input to train a convolutional neural network (CNN), which acts as the inference engine of the system. The initial tests of the system were performed on a set of 247 samples from the Pulmonary Department of the Álvaro Cunqueiro Hospital in Vigo (Galicia, Spain), with an AUC value of ≈ 0.8. This study demonstrates the benefits of an intelligent decision support system based on a novel data formalisation approach that allows the use of advanced DL techniques starting from tabular data. In this way, the ability of CNNs to recognise complex patterns using visual elements such as gradients and contrasts can be exploited. This approach effectively addresses the challenges of analysing large amounts of tabular data and reduces common problems such as bias and variance, resulting in improved diagnostic accuracy.

Electric vehicle charging navigation strategy in coupled smart grid and transportation network: A hierarchical reinforcement learning approach

Most of the existing electric vehicle (EV) charging navigation methods do not simultaneously take into account the electric vehicle charging destination optimization and path planning. Moreover, they are unable to provide online real-time decision-making under a variety of uncertain factors. To address these problems, this paper first establishes a bilevel stochastic optimization model for EV charging navigation considering various uncertainties, and then proposes an EV charging navigation method based on the hierarchical enhanced deep Q network (HEDQN) to solve the above stochastic optimization model in real-time. The proposed HEDQN contains two enhanced deep Q networks, which are utilized to optimize the charging destination and charging route path of EVs, respectively. Finally, the proposed method is simulated and validated in two urban transportation networks. The simulation results demonstrate that compared with the Dijkstra shortest path algorithm, single-layer deep reinforcement learning algorithm, and traditional hierarchical deep reinforcement learning algorithm, the proposed HEDQN algorithm can effectively reduce the total charging cost of electric vehicles and realize online real-time charging navigation of electric vehicles, that shows excellent generalization ability and scalability.

Continual Reinforcement Learning for Quadruped Robot Locomotion.

The ability to learn continuously is crucial for a robot to achieve a high level of intelligence and autonomy. In this paper, we consider continual reinforcement learning (RL) for quadruped robots, which includes the ability to continuously learn sub-sequential tasks (plasticity) and maintain performance on previous tasks (stability). The policy obtained by the proposed method enables robots to learn multiple tasks sequentially, while overcoming both catastrophic forgetting and loss of plasticity. At the same time, it achieves the above goals with as little modification to the original RL learning process as possible. The proposed method uses the Piggyback algorithm to select protected parameters for each task, and reinitializes the unused parameters to increase plasticity. Meanwhile, we encourage the policy network exploring by encouraging the entropy of the soft network of the policy network. Our experiments show that traditional continual learning algorithms cannot perform well on robot locomotion problems, and our algorithm is more stable and less disruptive to the RL training progress. Several robot locomotion experiments validate the effectiveness of our method.