Execution Paradigm Research Articles

Efficient predefined time adaptive neural network for motor execution EEG signal classification based brain-computer interaction

Nowadays, Electroencephalogram (EEG) devices that do not require invasive procedures get more attraction. Brain-Computer Interface (BCI) systems use EEG analysis to identify users' mental states, cognitive shifts, and stimuli-induced reactions. The Motor Execution (ME) paradigm is a vital control paradigm that holds great significance in this framework. In this manuscript, an Efficient Predefined Time Adaptive Neural Network for EEG-Based Brain-Computer Interaction in Motor Execution Classification (EPTNN-BCI-EEG) is proposed. Initially, the input signals are collected from EEG Dataset. The input signals are preprocessed using Robust M-Type Error-State Kalman Filter (RMESKF) to remove the artifacts and baseline signals. Then, the pre-processed signals are given to Signed Cumulative Distribution Transform (SCDT) for feature extraction. SCDT is used to extract Spatial and temporal features. Afterward, Efficient Predefined Time Adaptive Neural Network (EPTNN) is used to classify the EEG gender and signal, such as Audio/Video and Male/Female. In General, EPTNN does not expose any adoption of optimization systems to determine optimal parameters to classify the EEG gender and signal. Hence, Bitwise Arithmetic Optimization Algorithm (BAOA) is proposed to optimize the EPTNN. The proposed EPTNN-BCI-EEG approach is implemented in MATLAB and the performance metrics, such as Recall, Accuracy, Root Mean Square Error (RMSE), F1-score, Specificity, ROC, Computation time are assessed. The performance of EPTNN-BCI-EEG approach provides 17.82 %, 28.95 %, and 19.2 % higher accuracy, 24.38 %, 34.53 %, and 18.78 % higher recall, 26.7 %, 24.2 %, and 32.7 % higher specificity when analysed with the existing methods.

A Local-and-Global Attention Reinforcement Learning Algorithm for Multiagent Cooperative Navigation.

The cooperative navigation algorithm is the crucial technology for multirobot systems to accomplish autonomous collaborative operations, and it is still a challenge for researchers. In this work, we propose a new multiagent reinforcement learning algorithm called multiagent local-and-global attention actor-critic (MLGA2C) for multiagent cooperative navigation. Inspired by the attention mechanism, we design the local-and-global attention module to dynamically extract and encode critical environmental features. Meanwhile, based on the centralized training and decentralized execution (CTDE) paradigm, we extend a new actor-critic method to handle feature encoding and make navigation decisions. We also evaluate the proposed algorithm in two cooperative navigation scenarios: static target navigation and dynamic pedestrian target tracking. The multiple experimental results show that our algorithm performs well in cooperative navigation tasks with increasing agents.

Multi-agent cooperative strategy with explicit teammate modeling and targeted informative communication

The mainstream Multi-Agent Reinforcement Learning (MARL) methods introduce the teammate modeling or the communication mechanism into Centralized Training Decentralized Execution (CTDE) paradigm, which can improve coordination performance. However, the existing teammate modeling methods predict either actions or local observations, limiting their applicability. In addition, the traditional communication mechanism only considers the quantity of the communication links while ignoring the quality of retained communication links, leading to inefficient and redundant communication. To solve the above problems, this paper proposes a novel Multi-Agent Cooperative Strategy with Explicit Teammate Modeling and Targeted Informative Communication (MACS), which can generate and send the more informative message with the higher communication efficiency, further improving the coordination performance. Specifically, the Variational Auto-Encoder (VAE) is leveraged to allow each agent to simultaneously predict the observations and actions of teammates, thus generating more comprehensive communication message. Then, we propose a new Mutual Information (MI) between the communication message and teammate Q-value, which can obtain the informative message, ensuring the exploration and stability of the method. In addition, a targeted dynamic informative communication graph is established by the Graph Neural Network (GNN) which can reduce the redundant communication link through hypothetical analysis, further improving the overall communication efficiency. Eventually, we conduct experiments in StarCraft II, Collaborative Navigation, and Multi-Target Multi-Sensor Coverage environments. Experimental results show that the proposed approach is superior to the state-of-the-art in terms of coordination performance and communication efficiency.

Towards Efficient Coordination in Multi-Agent Reinforcement Learning through Hybrid Information-Driven Approaches

Abstract: This paper introduces a novel framework to enhance coordination in multi-agent reinforcement learning (MARL) systems by integrating mutual information (MMI) principles with information-driven strategies. Firstly, we propose a variational approach leveraging MMI to promote coordinated behaviors among agents by regulating the cumulative return alongside the simultaneous mutual information between multi-agent actions. Through the introduction of a latent variable inducing nonzero mutual information and the application of a variational bound, a tractable lower bound is derived for the MMI-regularized objective function. This bound combines maximum entropy reinforcement learning with reducing uncertainty in other agents' actions. Subsequently, we present a practical algorithm, Variational Maximum Mutual Information Multi-Agent Actor-Critic (VM3-AC), utilizing policy iteration to maximize the derived lower bound, following a centralized learning with decentralized execution (CTDE) paradigm. Secondly, we explore the challenges of large state spaces and limited computational resources in distributed multi-agent systems, proposing a hybrid information-driven MARL approach. This approach integrates informationtheoretic models as heuristics to aid navigation in sparse state spaces, complemented by information-based rewards within an RL framework to learn higher-level policies efficiently. Our preliminary findings suggest that this hybrid approach could enhance exploration efficiency significantly, demonstrating approximately three orders of magnitude improvement over naive baseline metrics. Although still in its early stages, this work presents a promising direction for future research in achieving efficient coordination in MARL systems

Intrinsic Action Tendency Consistency for Cooperative Multi-Agent Reinforcement Learning

Efficient collaboration in the centralized training with decentralized execution (CTDE) paradigm remains a challenge in cooperative multi-agent systems. We identify divergent action tendencies among agents as a significant obstacle to CTDE's training efficiency, requiring a large number of training samples to achieve a unified consensus on agents' policies. This divergence stems from the lack of adequate team consensus-related guidance signals during credit assignment in CTDE. To address this, we propose Intrinsic Action Tendency Consistency, a novel approach for cooperative multi-agent reinforcement learning. It integrates intrinsic rewards, obtained through an action model, into a reward-additive CTDE (RA-CTDE) framework. We formulate an action model that enables surrounding agents to predict the central agent's action tendency. Leveraging these predictions, we compute a cooperative intrinsic reward that encourages agents to align their actions with their neighbors' predictions. We establish the equivalence between RA-CTDE and CTDE through theoretical analyses, demonstrating that CTDE's training process can be achieved using N individual targets. Building on this insight, we introduce a novel method to combine intrinsic rewards and RA-CTDE. Extensive experiments on challenging tasks in SMAC, MPE, and GRF benchmarks showcase the improved performance of our method.

Multi-Agent Deep Reinforcement Learning for Multi-Lane Freeways Differential Variable Speed Limit Control in Mixed Traffic Environment

In advanced freeway traffic management systems, variable speed limit control (VSLC) is frequently discussed as one of the control measures. However, in a mixed traffic environment where connected and automated vehicles (CAVs) and human-driven vehicles coexist, the existing VSLC strategies for multi-lane freeways have two major shortcomings: the lack of precise control at the individual vehicle level, and the implementation of uniform VSLC across all lanes. This paper proposes a novel differential variable speed limit control (DVSLC) strategy based on multi-agent reinforcement learning (MARL) in a mixed traffic environment (abbreviated as MARL-DVSLC). The proposed MARL-DVSLC approach utilized a centralized training with decentralized execution paradigm to learn the joint actions of variable speed limit controllers across all lanes, thereby setting different speed limits for each lane. The reward function takes into account the total time spent (TTS) on freeways to improve traffic mobility. Note that MARL-DVSLC disseminates speed limit information to CAVs via infrastructure-to-vehicle (I2V) communication. The effectiveness of MARL-DVSLC is verified under different simulation scenarios. Moreover, its performance is compared with the feedback-based VSLC method, the DVSLC method based on deep deterministic policy gradient (DDPG) (abbreviated as DDPG-DVSLC), and the no-control case in relation to performance. The results indicate that the proposed strategy can effectively improve traffic efficiency and reduce the spatiotemporal range of traffic congestion at a 30% penetration rate of CAVs. Compared with the suboptimal DDPG-DVSLC method, the proposed strategy can improve TTS by 12.88% with stable traffic demand and 10.24% with fluctuating traffic demand.

Coordination as inference in multi-agent reinforcement learning

The Centralized Training and Decentralized Execution (CTDE) paradigm, where a centralized critic is allowed to access global information during the training phase while maintaining the learned policies executed with only local information in a decentralized way, has achieved great progress in recent years. Despite the progress, CTDE may suffer from the issue of Centralized–Decentralized Mismatch (CDM): the suboptimality of one agent’s policy can exacerbate policy learning of other agents through the centralized joint critic. In contrast to centralized learning, the cooperative model that most closely resembles the way humans cooperate in nature is fully decentralized, i.e. Independent Learning (IL). However, there are still two issues that need to be addressed before agents coordinate through IL: (1) how agents are aware of the presence of other agents, and (2) how to coordinate with other agents to improve joint policy under IL. In this paper, we propose an inference-based coordinated MARL method: Deep Motor System (DMS). DMS first presents the idea of individual intention inference where agents are allowed to disentangle other agents from their environment. Secondly, causal inference was introduced to enhance coordination by reasoning each agent’s effect on others’ behavior. The proposed model was extensively experimented on a series of Multi-Agent MuJoCo and StarCraftII tasks. Results show that the proposed method outperforms independent learning algorithms and the coordination behavior among agents can be learned even without the CTDE paradigm compared to the state-of-the-art baselines including IPPO and HAPPO.

Multiagent Continual Coordination via Progressive Task Contextualization.

Cooperative multiagent reinforcement learning (MARL) has attracted significant attention and has the potential for many real-world applications. Previous arts mainly focus on facilitating the coordination ability from different aspects (e.g., nonstationarity and credit assignment) in single-task or multitask scenarios, ignoring the stream of tasks that appear in a continual manner. This ignorance makes the continual coordination an unexplored territory, neither in problem formulation nor efficient algorithms designed. Toward tackling the mentioned issue, this article proposes an approach, multiagent continual coordination via progressive task contextualization (MACPro). The key point lies in obtaining a factorized policy, using shared feature extraction layers but separated independent task heads, each specializing in a specific class of tasks. The task heads can be progressively expanded based on the learned task contextualization. Moreover, to cater to the popular centralized training with decentralized execution (CTDE) paradigm in MARL, each agent learns to predict and adopt the most relevant policy head based on local information in a decentralized manner. We show in multiple multiagent benchmarks that existing continual learning methods fail, while MACPro is able to achieve close-to-optimal performance. More results also disclose the effectiveness of MACPro from multiple aspects, such as high generalization ability.

Multi-Task Heterogeneous Ensemble Learning-Based Cross-Subject EEG Classification Under Stroke Patients.

Robot-assisted motor training is applied for neurorehabilitation in stroke patients, using motor imagery (MI) as a representative paradigm of brain-computer interfaces to offer real-life assistance to individuals facing movement challenges. However, the effectiveness of training with MI may vary depending on the location of the stroke lesion, which should be considered. This paper introduces a multi-task electroencephalogram-based heterogeneous ensemble learning (MEEG-HEL) specifically designed for cross-subject training. In the proposed framework, common spatial patterns were used for feature extraction, and the features according to stroke lesions are shared and selected through sequential forward floating selection. The heterogeneous ensembles were used as classifiers. Nine patients with chronic ischemic stroke participated, engaging in MI and motor execution (ME) paradigms involving finger tapping. The classification criteria for the multi-task were established in two ways, taking into account the characteristics of stroke patients. In the cross-subject session, the first involved a direction recognition task for two-handed classification, achieving a performance of 0.7419 (±0.0811) in MI and 0.7061 (±0.1270) in ME. The second task focused on motor assessment for lesion location, resulting in a performance of 0.7457 (±0.1317) in MI and 0.6791 (±0.1253) in ME. Comparing the specific-subject session, except for ME on the motor assessment task, performance on both tasks was significantly higher than the cross-subject session. Furthermore, classification performance was similar to or statistically higher in cross-subject sessions compared to baseline models. The proposed MEEG-HEL holds promise in improving the practicality of neurorehabilitation in clinical settings and facilitating the detection of lesions.

Multi-agent graph reinforcement learning for decentralized Volt-VAR control in power distribution systems

Volt/Var control (VVC) is a crucial function in power distribution systems to minimize power loss and maintain voltages within allowable limits. However, incomplete and inaccurate information about the distribution network makes model-based VVC methods difficult to implement in practice. In this paper, we propose a novel multi-agent graph-based deep reinforcement learning (DRL) algorithm named MASAC-HGRN to address the VVC problem under partial observation constraints. Our proposed algorithm divides the power distribution system into several regions, each region treated as an agent. Unlike traditional model-based or global-observation-based DRL methods, our proposed method leverages a practical decentralized training and decentralized execution (DTDE) paradigm to address the partial observation constraints. The well-trained agents gather information only from their interconnected neighbors and realize decentralized local control. Numerical studies with IEEE 33-bus and 123-bus distribution test feeders demonstrate that our proposed MASAC-HGRN algorithm outperforms the state-of-art RL algorithms and traditional model-based approaches in terms of VVC performance. Moreover, the DTDE framework exhibits flexibility and robustness in extensive robustness experiments.

Serverless Vehicular Edge Computing for the Internet of Vehicles

Rapid growth in the popularity of smart vehicles and increasing demand for vehicle autonomy brings new opportunities for vehicular edge computing (VEC). VEC aims at offloading the time-sensitive computational load of connected vehicles to edge devices, e.g., roadside units. However, VEC offloading raises complex resource management challenges and thus remains largely inaccessible to automotive companies. Recently, serverless computing emerged as a convenient approach to the execution of functions without the hassle of infrastructure management. In this work, we propose the idea of serverless VEC as the execution paradigm for the Internet of vehicles applications. Further, we analyze its benefits and drawbacks, and identify technology gaps. We also propose emulation as a design, evaluation, and experimentation methodology for serverless VEC solutions. Using our emulation toolkit, we validate the feasibility of serverless VEC for real-world traffic scenarios.

Credit assignment with predictive contribution measurement in multi-agent reinforcement learning

Credit assignment is a crucial issue in multi-agent tasks employing a centralized training and decentralized execution paradigm. While value decomposition has demonstrated strong performance in Q-learning-based approaches and certain Actor–Critic variants, it remains challenging to achieve efficient credit assignment in multi-agent tasks using policy gradient methods due to decomposable value limitations. This paper introduces Predictive Contribution Measurement, an explicit credit assignment method that compares prediction errors among agents and allocates surrogate rewards based on their relevance to global state transitions, with a theoretical guarantee. With multi-agent proximal policy optimization (MAPPO) as a training backend, we propose Predictive Contribution MAPPO (PC-MAPPO). Our experiments demonstrate that PC-MAPPO, with a 10% warm-up phase, outperforms MAPPO, QMIX, and Weighted QMIX on StarCraft multi-agent challenge tasks, particularly in maps requiring heightened cooperation to defeat enemies, such as the map corridor. Employing a pre-trained predictor, PC-MAPPO achieves significantly improved performance on all tested super-hard maps. In parallel training scenarios, PC-MAPPO exhibits superior data efficiency and achieves state-of-the-art performance compared to other methods.

PROTECTION OF WOMEN'S AND CHILDREN'S RIGHTS BASED ON SYSTEM INTERCONNECTION: A New Paradigm of Execution of Women and Children's Rights after Divorce

The implementation of post-divorce decisions in the religious court demands great attention because the system for implementing decisions in divorce cases still needs to be stronger. The cost of executing the decision is not commensurate with the nominal (the ex-husband must pay “the obligation” to his ex-wife). As a result, court decisions become “useless”, which are only authoritative in writing but weak in implementation. Through this paper, the author wants to explain a new concept related to the fulfillment of the rights of women and children after divorce based on an interconnected system so that post-divorce women and children are guaranteed their fulfillment of rights without going through the process of execution in court. This research is normative research through a statutory approach and a conceptual approach. This research shows that ensuring the women's and children's rights fulfillment requires an interconnected system. Courts must involve non-judicial institutions in an integrated manner following their respective authorities. This new framework makes institutions outside the judiciary as external partners in implementing single identity-based court decisions.

Inhibition, Shifting and Updating: Inter and intra-domain commonalities and differences from an executive functions activation likelihood estimation meta-analysis

Executive functions are higher-order mental processes that support goal-directed behavior. Among these processes, Inhibition, Updating, and Shifting have been considered core executive domains. In this meta-analysis, we comprehensively investigate the neural networks of these executive domains and we synthesize for the first time the neural convergences and divergences among the most frequently used executive paradigms within those domains. A systematic search yielded 1055 published neuroimaging studies (including 26,191 participants in total). Our study revealed that a fronto-parietal network was shared by the three main domains. Furthermore, we executed conjunction analyses among the paradigms of the same domain to extract the core distinctive components of the main executive domains. This approach showed that Inhibition and Shifting are characterized by a strongly lateralized neural activation in the right and left hemisphere, respectively. In addition, both networks overlapped with the Updating network but not with each other. Remarkably, our study detected heterogeneity among the paradigms from the same domain. More specifically, analysis of Inhibition tasks revealed differing activations for Response Inhibition compared to Interference Control paradigms, suggesting that Inhibition encompasses relatively heterogeneous sub-functions. Shifting analyses revealed a bilateral overlap of the Wisconsin Card Sorting Task with the Updating network, but this pattern was absent for Rule Switching and Dual Task paradigms. Moreover, our Updating meta-analyses revealed the neural signatures associated with the specific modules of the Working Memory model from Baddeley and Hitch. To our knowledge, this is the most comprehensive meta-analysis of executive functions to date. Its paradigm-driven analyses provide a unique contribution to a better understanding of the neural convergences and divergences among executive processes that are relevant for clinical applications, such as cognitive enhancement and neurorehabilitation interventions.

Dataflow Driven Partitioning of Machine Learning Applications for Optimal Energy Use in Batteryless Systems

Sensing systems powered by energy harvesting have traditionally been designed to tolerate long periods without energy. As the Internet of Things (IoT) evolves toward a more transient and opportunistic execution paradigm, reducing energy storage costs will be key for its economic and ecologic viability. However, decreasing energy storage in harvesting systems introduces reliability issues. Transducers only produce intermittent energy at low voltage and current levels, making guaranteed task completion a challenge. Existing ad hoc methods overcome this by buffering enough energy either for single tasks, incurring large data-retention overheads, or for one full application cycle, requiring a large energy buffer. We present Julienning : an automated method for optimizing the total energy cost of batteryless applications. Using a custom specification model, developers can describe transient applications as a set of atomically executed kernels with explicit data dependencies. Our optimization flow can partition data- and energy-intensive applications into multiple execution cycles with bounded energy consumption. By leveraging interkernel data dependencies, these energy-bounded execution cycles minimize the number of system activations and nonvolatile data transfers, and thus the total energy overhead. We validate our methodology with two batteryless cameras running energy-intensive machine learning applications. Using a solar testbed, we replay real-world illuminance traces to experimentally demonstrate optimized batteryless execution with a transducer-to-application energy efficiency of 74.5%. Partitioning results demonstrate that compared to ad hoc solutions, our method can reduce the required energy storage by over 94% while only incurring a 0.12% energy overhead.

A leader-following paradigm based deep reinforcement learning method for multi-agent cooperation games

Multi-agent deep reinforcement learning algorithms with centralized training with decentralized execution (CTDE) paradigm has attracted growing attention in both industry and research community. However, the existing CTDE methods follow the action selection paradigm that all agents choose actions at the same time, which ignores the heterogeneous roles of different agents. Motivated by the human wisdom in cooperative behaviors, we present a novel leader-following paradigm based deep multi-agent cooperation method (LFMCO) for multi-agent cooperative games. Specifically, we define a leader as someone who broadcasts a message representing the selected action to all subordinates. After that, the followers choose their individual action based on the received message from the leader. To measure the influence of leader’s action on followers, we introduced a concept of information gain, i.e., the change of followers’ value function entropy, which is positively correlated with the influence of leader’s action. We evaluate the LFMCO on several cooperation scenarios of StarCraft2. Simulation results confirm the significant performance improvements of LFMCO compared with four state-of-the-art benchmarks on the challenging cooperative environment.

Algorithm 1028: VTMOP: Solver for Blackbox Multiobjective Optimization Problems

VTMOP is a Fortran 2008 software package containing two Fortran modules for solving computationally expensive bound-constrained blackbox multiobjective optimization problems. VTMOP implements the algorithm of [ 32 ], which handles two or more objectives, does not require any derivatives, and produces well-distributed points over the Pareto front. The first module contains a general framework for solving multiobjective optimization problems by combining response surface methodology, trust region methodology, and an adaptive weighting scheme. The second module features a driver subroutine that implements this framework when the objective functions can be wrapped as a Fortran subroutine. Support is provided for both serial and parallel execution paradigms, and VTMOP is demonstrated on several test problems as well as one real-world problem in the area of particle accelerator optimization.

Intelligent Network Service Optimization in the Context of 5G/NFV

Our contemporary society has never been more connected and aware of vital information in real time, through the use of innovative technologies. A considerable number of applications have transitioned into the cyber-physical domain, automating and optimizing their routines and processes via the dense network of sensing devices and the immense volumes of data they collect and instantly share. In this paper, we propose an innovative architecture based on the monitoring, analysis, planning, and execution (MAPE) paradigm for network and service performance optimization. Our study confirms distinct evidence that the utilization of learning algorithms, consuming datasets enriched with the users’ empirical opinions as input during the analysis and planning phases, contributes greatly to the optimization of video streaming quality, especially by handling different packet loss rates, paving the way for the achievable provision of a resilient communications platform for calamity assessment and management.

Multiagent Reinforcement-Learning-Aided Service Function Chain Deployment for Internet of Things

Nowadays, the compelling applications of the Internet of Things (IoT) bring unexpected economic benefits to our daily lives. But at the same time, it also poses huge challenges to service providers. Diverse proprietary hardware (i.e., firewall and code conversion) have to be deployed in networks for meeting different applications’ requirements. Recently, network functions virtualization (NFV) is considered a promising technique. In the NFV-enabled architecture, network services can be implemented via a set of orderly virtual network functions (VNFs) on standardized compute nodes, which is termed service function chains (SFCs). However, with the explosion of IoT applications, embedding multiple SFCs in a shared NFV-enabled infrastructure becomes a challenging problem. Centralized schemes suffer from the scalability and private issue, while distributed schemes suffer from the nonconvergence problem. In this article, we propose a hybrid intelligent control architecture, which adopts the centralized training and distributed execution paradigm. A centralized critic is introduced to ease the training process of the distributed network nodes. Besides, considering the competitive behavior of users, we formulate the resource allocation problem as a multiuser competition game model. Based on this, we proposed a multiagent reinforcement learning-based SFCs deployment algorithm.

Efficiently Detecting Non-Stationary Opponents: A Bayesian Policy Reuse Approach under Partial Observability

In multi-agent domains, dealing with non-stationary opponents that change behaviors (policies) consistently over time is still a challenging problem, where an agent usually requires the ability to detect the opponent’s policy accurately and adopt the optimal response policy accordingly. Previous works commonly assume that the opponent’s observations and actions during online interactions are known, which can significantly limit their applications, especially in partially observable environments. This paper focuses on efficient policy detecting and reusing techniques against non-stationary opponents without their local information. We propose an algorithm called Bayesian policy reuse with LocAl oBservations (Bayes-Lab) by incorporating variational autoencoders (VAE) into the Bayesian policy reuse (BPR) framework. Following the centralized training with decentralized execution (CTDE) paradigm, we train VAE as an opponent model during the offline phase to extract the latent relationship between the agent’s local observations and the opponent’s local observations. During online execution, the trained opponent models are used to reconstruct the opponent’s local observations, which can be combined with episodic rewards to update the belief about the opponent’s policy. Finally, the agent reuses the best response policy based on the updated belief to improve online performance. We demonstrate that Bayes-Lab outperforms existing state-of-the-art methods in terms of detection accuracy, accumulative rewards, and episodic rewards in a predator–prey scenario. In this experimental environment, Bayes-Lab can achieve about 80% detection accuracy and the highest accumulative rewards, and its performance is less affected by the opponent policy switching interval. When the switching interval is less than 10, its detection accuracy is at least 10% higher than other algorithms.