Learning Agent Research Articles

Adaptive Learning in AI Agents for the Metaverse: The ALMAA Framework

This study investigates the adaptability of Artificial Intelligence (AI) agents in the Metaverse, focusing on their ability to enhance responsiveness, decision-making, and engagement through the proposed Adaptive Learning Model for AI Agents (ALMAA) framework. The research does not introduce new interventions to existing platforms like Epic Games or AltspaceVR but instead analyzes how their operations align with adaptive learning principles. By examining these platforms, the study demonstrates the alignment between real-world practices and theoretical constructs, offering insights into how adaptive AI systems operate in dynamic virtual environments. Case observations highlight key metrics such as user interaction efficiency, contextual decision accuracy, and predictive engagement strategies. The data, derived from detailed user interaction logs and feedback reports, underscore the practical application of adaptive learning in optimizing user satisfaction and system performance. Statistical analyses reveal notable gains in response speed, predictive precision, and user engagement, validating the theoretical framework’s relevance. This paper positions the ALMAA framework as a critical lens for understanding and analyzing adaptive AI in virtual settings. It emphasizes theoretical exploration rather than experimental application, providing a foundation for future research into scalable, user-centered AI systems tailored for the Metaverse’s evolving demands.

Teachers and students at the crossroads: unpacking L1 use in CLIL in China’s higher education

ABSTRACT Grounded in the concept of translanguaging, the use of first language (L1) in foreign language (L2) classrooms has sparked considerable debate within Content and Language Integrated Learning (CLIL), particularly in under-researched non-Western contexts such as China. A significant research gap remains regarding the contexts, purposes, and factors influencing L1 use. This qualitative study, situated within English language education in Chinese higher education, addresses this gap with a focus on one purposively selected CLIL classroom. Conversation analysis of classroom observations indicates that translanguaging functions as a strategic, context-driven choice shaped by teacher awareness and learner agency rather than as a fallback to L1. In-depth interviews with the teacher and students reveal that L1 use, framed within Cultural-Historical Activity Theory, is mediated by physical and conceptual tools and peer interactions, with effects observed both inside and outside the classroom. The study also highlights how the division of labour and institutional, educational, or ideological rules shape the balance between L1 and L2 use and create tensions between traditional L2 immersion expectations and learners’ practical needs. These findings call for a more nuanced understanding of translanguaging and suggest the reconsideration of strict L2-only policies to enhance learner engagement and comprehension in CLIL.

Reinforcement-Learning-Based Synthesis of Custom Approximate Parallel Prefix Adders

Approximate hardware units compute a sufficiently accurate result rather than a fully accurate one using fewer transistors or equivalently logic gates than their accurate counterparts. This approach significantly saves energy while maintaining acceptable application-level quality. In this work, we focus on synthesizing custom approximate parallel prefix adders tailored to specific applications. The introduced reinforcement learning (RL) framework co-optimizes application performance and hardware complexity. An RL agent learns approximate addition strategies by exploring the entire design space and receiving feedback from hardware synthesis and application performance. Experimental results demonstrate that the synthesized adders can reduce area and power consumption by 12% and 10%, respectively, on average without compromising the error behavior of the application, or can significantly improve the application’s error metrics without degrading area and power consumption for a variety of practical applications.

A deep reinforcement learning-based optimization approach for containerized microservice scheduling in Hybrid Fog/Cloud environments

The deployment of microservices in Hybrid Fog/Cloud (HFC) environments for Internet of Things (IoT) applications presents a significant challenge in efficiently scheduling containerized services across distributed resources. While existing studies have explored microservice scheduling, a comprehensive approach that considers resource constraints, workflow dependencies, and dynamic hybrid environments remains elusive. This paper introduces a novel Deep Reinforcement Learning-based Algorithm (DRLA) for containerized microservice scheduling in HFC environments. DRLA utilizes a multi-constrained Binary Quadratic Program (BQP) model to optimize execution time, resource consumption, and occupancy rates while considering microservice dependencies and resource capabilities. The algorithm leverages two Deep Reinforcement Learning (DRL) agents, DQN and REINFORCE, to learn and adapt to the dynamic nature of the HFC federation. Experimental evaluations using five real-world Business Process (BP) use cases demonstrate that DRLA outperforms existing scheduling approaches such as default Kubernetes, Reward Sharing Deep Q-Learning (RSDQL), and Deep Reinforcement Learning (DRL) schedulers. Compared to existing schedulers, ours delivers optimal or near-optimal solutions, demonstrating significant improvements in key performance values. Indeed, DRLA achieves average optimality gaps of just 0.16% and 0.21%, significantly outperforming the Kubernetes scheduler, which exhibits a gap of 2.88%. It is worth noting that other similar algorithms see a far higher increase in optimality gaps. This highlights DRLA’s excellent performance and ability to schedule containerized microservices in hybrid fog/cloud environments, resulting in near-optimal solutions for a variety of use cases.

Enhancing fog load balancing through lifelong transfer learning of reinforcement learning agents

Enhancing fog load balancing through lifelong transfer learning of reinforcement learning agents

Safe Reinforcement Learning With Dual Robustness.

Reinforcement learning (RL) agents are vulnerable to adversarial disturbances, which can deteriorate task performance or break down safety specifications. Existing methods either address safety requirements under the assumption of no adversary (e.g., safe RL) or only focus on robustness against performance adversaries (e.g., robust RL). Learning one policy that is both safe and robust under any adversaries remains a challenging open problem. The difficulty is how to tackle two intertwined aspects in the worst cases: feasibility and optimality. The optimality is only valid inside a feasible region (i.e., robust invariant set), while the identification of maximal feasible region must rely on how to learn the optimal policy. To address this issue, we propose a systematic framework to unify safe RL and robust RL, including the problem formulation, iteration scheme, convergence analysis and practical algorithm design. The unification is built upon constrained two-player zero-sum Markov games, in which the objective for protagonist is twofold. For states inside the maximal robust invariant set, the goal is to pursue rewards under the condition of guaranteed safety; for states outside the maximal robust invariant set, the goal is to reduce the extent of constraint violation. A dual policy iteration scheme is proposed, which simultaneously optimizes a task policy and a safety policy. We prove that the iteration scheme converges to the optimal task policy which maximizes the twofold objective in the worst cases, and the optimal safety policy which stays as far away from the safety boundary. The convergence of safety policy is established by exploiting the monotone contraction property of safety self-consistency operators, and that of task policy depends on the transformation of safety constraints into state-dependent action spaces. By adding two adversarial networks (one is for safety guarantee and the other is for task performance), we propose a practical deep RL algorithm for constrained zero-sum Markov games, called dually robust actor-critic (DRAC). The evaluations with safety-critical benchmarks demonstrate that DRAC achieves high performance and persistent safety under all scenarios (no adversary, safety adversary, performance adversary), outperforming all baselines by a large margin.

Envisioning Hopeful Futures: Designing Racially Just Online Learning

This autoethnographic study examines the influence of my positionality in designing a hopeful, racially just online learning. Hope stems from an innate belief in the possibilities for a brighter future. Therefore, hope involves embracing strengths. In this paper, I explore the influence of my positionality on my online course design through autoethnography. I analyze how feminism has impacted my online course design and strategies for cultivating teaching, social, and cognitive presence for racially just online learning. Racially just online learning aims for representational justice, recognitive justice, and representational justice to restore to people their history, culture, language, and dignity. I describe how my online course design is rooted in human-centered learning design with empathy where learners’ cultural capital, lived experiences, and funds of knowledge are valued and honored. In this trusting, caring online community, open educational practices and appreciative inquiry empower learners with learner agency for a hopeful and racially just future.

Privacy-preserving data mining optimization for big data analytics using deep reinforcement learning

The rapid growth of big data analytics has heightened concerns about data privacy, necessitating the development of advanced privacy-preserving techniques. This research addresses the challenge of optimizing privacy-preserving data mining (PPDM) for big data analytics through the innovative application of deep reinforcement learning (DRL). We propose a novel framework that integrates DRL to dynamically balance privacy and utility, ensuring robust data protection while maintaining analytical accuracy. The framework employs a reinforcement learning agent to adaptively select optimal privacy-preserving strategies based on the evolving data environment and user requirements, while ensuring compliance with the latest security and privacy standards such as ISO/IEC 27001:2023. Experimental results demonstrate significant improvements in both privacy protection and data utility, surpassing traditional PPDM methods. Our findings highlight the potential of DRL in enhancing privacy-preserving mechanisms, offering a scalable and efficient solution for secure big data analytics.

The Evolutionary Dynamics of Soft-Max Policy Gradient in Multi-Agent Settings

Policy gradient is one of the most famous algorithms in reinforcement learning. This paper studies the mean dynamics of the soft-max policy gradient algorithm and its properties in multi-agent settings by resorting to evolutionary game theory and dynamical system tools. Unlike most multi-agent reinforcement learning algorithms, whose mean dynamics are a slight variant of the replicator dynamics not affecting the properties of the original dynamics, the soft-max policy gradient dynamics presents a structure significantly different from that of the replicator. In particular, we show that the soft-max policy gradient dynamics in a given game are equivalent to the replicator dynamics in an auxiliary game obtained by a non-convex transformation of the payoffs of the original game. Such a structure gives the dynamics several non-standard properties. The first property we study concerns the convergence to the best response. In particular, while the continuous-time mean dynamics always converge to the best response, the crucial question concerns the convergence speed. Precisely, we show that the space of initializations can be split into two complementary sets such that the trajectories initialized from points of the first set (said good initialization region) directly move to the best response. In contrast, those initialized from points of the second set (said bad initialization region) move first to a series of sub-optimal strategies and then to the best response. Interestingly, in multi-agent adversarial machine learning environments, we show that an adversary can exploit this property to make any current strategy of the learning agent using the soft-max policy gradient fall inside a bad initialization region, thus slowing its learning process and exploiting that policy. When the soft-max policy gradient dynamics is studied in multi-population games, modeling the learning dynamics in self-play, we show that the dynamics preserve the volume of the set of initial points. This property proves that the dynamics cannot converge when the only equilibrium of the game is fully mixed, as the volume of the set of initial points would need to shrink. We also give empirical evidence that the volume expands over time, suggesting that the dynamics in games with fully-mixed equilibrium is chaotic.

Reinforcement Learning Informed Evolutionary Search for Autonomous Systems Testing

Evolutionary search (ES)-based techniques are commonly used for testing autonomous robotic systems. However, these approaches often rely on computationally expensive simulator-based models for test scenario evaluation. To improve the computational efficiency of the search-based testing, we propose augmenting the ES with a reinforcement learning (RL) agent trained using surrogate rewards derived from domain knowledge. In our approach, known as RIGAA (Reinforcement learning Informed Genetic Algorithm for Autonomous systems testing), we first train an RL agent to learn useful constraints of the problem and then use it to produce a certain part of the initial population of the search algorithm. By incorporating an RL agent into the search process, we aim to guide the algorithm towards promising regions of the search space from the start, enabling more efficient exploration of the solution space. We evaluate RIGAA on two case studies: maze generation for an autonomous “Ant” robot and road topology generation for an autonomous vehicle lane-keeping assist system. In both case studies, RIGAA reveals more failures of a high level of diversity than the compared baselines. RIGAA also outperforms the state-of-the-art tools for vehicle lane-keeping assist system testing, such as AmbieGen, CRAG, WOGAN, and Frenetic in terms of the number of revealed failures in a two-hour budget.

Hierarchical Reinforcement Learning: Structuring Decision-Making for Complex AI Tasks

Hierarchical Reinforcement Learning (abbreviated as HRL) represents a progress in the realm of artificial intelligence (AI) focusing on handling the increasing complexity of decision-making processes effectively. By breaking down tasks into smaller sub tasks that are easier to manage and comprehend; HRL plays a key role in enhancing the learning efficiency of AI models. The hierarchical system mirrors decision making approaches by breaking down complex goals into smaller achievable steps; this allows reinforcement learning agents to effectively learn and adjust to dynamic environments, with limited rewards and long-term objectives. In contrast to the limitations faced by flat reinforcement learning techniques in terms of scalability and efficiency Hierarchical Reinforcement Learning (HRL) introduces a higher-level policy that controls the series and implementation of sub tasks. This study delves into the evolution, application, and influence of reinforcement learning models in artificial intelligence. We explore HRL frameworks, such, as the Options framework and temporal adaptive models that offer a systematic decision-making method. Furthermore, the study analyzes the role HRL plays in fields including robotics, self-governing systems, and game artificial intelligence. HRL is on the brink of transforming how AI tackles challenges by shaping decision making processes in intricate settings that mimic real world scenarios. This piece explores the issue at hand along, with its solutions and applications while discussing the influence and extent of HRL in enhancing AI technologies. Keywords: Hierarchical Reinforcement Learning, AI, Reinforcement Learning, Decision-Making, Sub-tasks, Options Framework, Temporal-Adaptive Models, Robotics, Autonomous Systems, Game AI, Artificial Intelligence.

Learning in practice: reinforcement learning-based traffic signal control augmented with actuated control

ABSTRACT Most Reinforcement Learning (RL) based Traffic Signal Control (TSC) models are trained in simulation platforms, which inevitably suffer from the performance degradation after deployment due to the mismatch between the traffic simulator and the real-world traffic system. Toward the real-world training of TSC agents, this study integrates the actuated control into the RL-based TSC framework as a defense mechanism, enabling the traffic system to remain resilient to suboptimal actions of agent during the real-world training process. The introduction of actuated control can significantly reduce the green time wastes due to unreasonable timing plans. The numerical results demonstrate that the proposed approach can reduce interferences to the traffic system operation during the real-world training of TSC agents, particularly at the early stage of agent learning, thereby promoting the practical deployment of RL-based TSC models. Meanwhile, agents can converge to better signal policies with the help of defense mechanism.

Optimizing 5G network slicing with DRL: Balancing eMBB, URLLC, and mMTC with OMA, NOMA, and RSMA

The advent of 5th Generation (5G) networks has introduced the strategy of network slicing as a paradigm shift, enabling the provision of services with distinct Quality of Service (QoS) requirements. The 5th Generation New Radio (5G NR) standard complies with the use cases Enhanced Mobile Broadband (eMBB), Ultra-Reliable Low Latency Communications (URLLC), and Massive Machine Type Communications (mMTC), which demand a dynamic adaptation of network slicing to meet the diverse traffic needs. This dynamic adaptation presents both a critical challenge and a significant opportunity to improve 5G network efficiency. This paper proposes a Deep Reinforcement Learning (DRL) agent that performs dynamic resource allocation in 5G wireless network slicing according to traffic requirements of the 5G use cases within two scenarios: eMBB with URLLC and eMBB with mMTC. The DRL agent evaluates the performance of different decoding schemes such as Orthogonal Multiple Access (OMA), Non-Orthogonal Multiple Access (NOMA), and Rate Splitting Multiple Access (RSMA) and applies the best decoding scheme in these scenarios under different network conditions. The DRL agent has been tested to maximize the sum rate in scenario eMBB with URLLC and to maximize the number of successfully decoded devices in scenario eMBB with mMTC, both with different combinations of number of devices, power gains and number of allocated frequencies. The results show that the DRL agent dynamically chooses the best decoding scheme and presents an efficiency in maximizing the sum rate and the decoded devices between 84% and 100% for both scenarios evaluated.

A reinforcement learning based software simulator for motor brain-computer interfaces.

Intracortical motor brain-computer interfaces (BCIs) are expensive and time-consuming to design because accurate evaluation traditionally requires real-time experiments. In a BCI system, a user interacts with an imperfect decoder and continuously changes motor commands in response to unexpected decoded movements. This "closed-loop" nature of BCI leads to emergent interactions between the user and decoder that are challenging to model. The gold standard for BCI evaluation is therefore real-time experiments, which significantly limits the speed and community of BCI research. We present a new BCI simulator that enables researchers to accurately and quickly design BCIs for cursor control entirely in software. Our simulator replaces the BCI user with a deep reinforcement learning (RL) agent that interacts with a simulated BCI system and learns to optimally control it. We demonstrate that our simulator is accurate and versatile, reproducing the published results of three distinct types of BCI decoders: (1) a state-of-the-art linear decoder (FIT-KF), (2) a "two-stage" BCI decoder requiring closed-loop decoder adaptation (ReFIT-KF), and (3) a nonlinear recurrent neural network decoder (FORCE).

Reinforcement learning for rotation sensing with ultracold atoms in an optical lattice

In this paper, we investigate a design approach of reinforcement learning to engineer a gyroscope in an optical lattice for the inertial sensing of rotations. Our methodology is not based on traditional atom interferometry, that is, splitting, reflecting, and recombining wavefunction components. Instead, the learning agent is assigned the task of generating lattice shaking sequences that optimize the sensitivity of the gyroscope to rotational signals in an end-to-end design philosophy. What results is an interference device that is completely distinct from the familiar Mach-Zehnder-type interferometer. For the same total interrogation time, the end-to-end design leads to a twentyfold improvement in sensitivity over traditional Bragg interferometry. Published by the American Physical Society 2024

GSADDQN: Combining GraphSAGE and reinforcement learning for routing optimization in software-defined optical transport network

As a result of rapid development of network communication technology, optical transport network (OTN) traffic has also experienced rapid growth in terms of information volume scale, traffic complexity, and spatiotemporal distribution dynamics. Because the OTN traffic demand has complex spatiotemporal fluctuations, the traditional deep reinforcement learning (DRL) algorithm has been applied to the routing optimization of software-defined OTNs. However, the traditional DRL algorithm has problems such as slow convergence, weak generalization ability, and load imbalance when performing routine tasks. To address these issues, we propose a graph sampling and aggregation (GraphSAGE)-based dueling deep Q-network (GSADDQN) algorithm for software-defined OTN routing optimization. First, we design a DRL-based routing decision model to find the best routing strategy for each optical network’s source–destination traffic demand. Second, considering the sparse connection characteristics of optical network nodes, we use sampling neighbors and a deep aggregation mechanism as the neural network model of the Dueling Deep Q-Network (Dueling DQN) algorithm so that the reinforcement learning agent can consciously aggregate important network information and improve the model’s convergence performance and generalization ability. Finally, we design simulation routing experiments based on Gym and evaluate the algorithm’s load balancing and generalization capabilities for different network topologies. The experimental results show that the GSADDQN algorithm has good convergence performance and load balancing ability in routing optimization of optical transmission networks and can generalize new network structures, maintaining good decision-making ability even during network node failures.

Transformação sustentável

The objective of the research was to apply composting techniques and social technology to treat food waste in a low -income condominium in southern Brazil, with the joint participation of university researchers and residents. The purpose was not only to reduce the environmental impact caused by waste, but also generate organic compounds of environmental and agronomic value. The project started with an observational visit to the condominium to identify appropriate spaces for the execution of the work. Then the researchers promoted the project through posters and direct conversations with the residents. After this interaction, training workshops were held for participating residents, totaling 40 families and 122 people. The workshops addressed topics such as the impacts of food waste, segregation at source, the composting process and social technology. Educational materials were distributed to assist in the learning process, and visual resources, such as workshops and educational videos, were used. Finally, the composting process was implemented. Social technology played a significant role throughout the process. During workshops and interactions with residents, participants became active learning agents, sharing traditional knowledge and integrating them with scientific knowledge. Educational materials have represented an important tool for social technology, providing information in an accessible and motivational way. The active participation of researchers and residents was also a positive aspect of this investigation, as it strengthened the condominium community, created social bonds and promoted awareness of environmental responsibility.

Bayesian Mechanics of Synaptic Learning Under the Free-Energy Principle.

The brain is a biological system comprising nerve cells and orchestrates its embodied agent's perception, behavior, and learning in dynamic environments. The free-energy principle (FEP) advocated by Karl Friston explicates the local, recurrent, and self-supervised cognitive dynamics of the brain's higher-order functions. In this study, we continue to refine the FEP through a physics-guided formulation; specifically, we apply our theory to synaptic learning by considering it an inference problem under the FEP and derive the governing equations, called Bayesian mechanics. Our study uncovers how the brain infers weight changes and postsynaptic activity, conditioned on the presynaptic input, by deploying generative models of the likelihood and prior belief. Consequently, we exemplify the synaptic efficacy in the brain with a simple model; in particular, we illustrate that the brain organizes an optimal trajectory in neural phase space during synaptic learning in continuous time, which variationally minimizes synaptic surprisal.

A critical review of 50 years of initial teacher education in Jamaica: challenges and solutions for improving its effectiveness

ABSTRACT This review of initial teacher education (ITE) in Jamaica highlights the policies, pivotal events, educational initiatives and reforms that have shaped the landscape of teacher education in Jamaica over the last 50 years. It offers a critical reflection on the provision of ITE and the findings of a recent report on education in Jamaica, which suggests that its ITE is weak and ineffective. The review highlights what has taken place over the last 50 years and shows how more emphasis has been placed on concerns over the supply of teachers than on the provision of the type of ITE that produces quality teachers. Drawing on recognised principles for best practice in ITE and a critical theoretical perspective around the importance of teacher and learner agency, the paper suggests more attention must be given to building a globally informed, but also locally crafted knowledge base for teacher preparation in Jamaica. This involves preparing teacher educators who are educated, critically conscious and certified to assume responsibility for developing high-quality ITE in Jamaica.

Algorithmic trading of real-time electricity with machine learning

Algorithmic trading is becoming the dominant approach in many electricity spot and futures markets. This paper focuses on the emerging interest in the less documented real-time imbalance markets, by developing reinforcement learning agents to find profit-making opportunities algorithmically. We develop a repeatable experimental setting to compare different market participants and explore the applications of Q-learning with neural networks for three types of market participants: a non-physical trader, a gas generator, and a battery electricity storage system. We backtest all three agents using British data across summer and winter months to compare their profits, risks and various experimental design considerations.