Action Selection Strategy Research Articles

Knowledge guided fuzzy deep reinforcement learning

Reinforcement learning (RL) addresses complex sequential decision-making problems through interactive trial-and-error and the handling of delayed rewards. However, reinforcement learning typically starts from scratch, necessitating extensive exploration, which results in low learning efficiency. In contrast, humans often leverage prior knowledge to learn. Inspired by this, this paper proposes a semantic knowledge-guided reinforcement learning method (KFDQN), which fully utilizes knowledge to influence reinforcement learning, thereby improving learning efficiency, training stability, and performance. In terms of knowledge representation, considering the strong fuzziness of semantic knowledge, a fuzzy system is constructed to represent this knowledge. In terms of knowledge integration, a knowledge-guided framework that integrates a hybrid action selection strategy (HYAS), a hybrid learning method (HYL), and knowledge updating is constructed in conjunction with the existing reinforcement learning framework. The HYAS integrates knowledge into action selection, reducing the randomness of traditional exploration methods. The HYL incorporates knowledge into the learning target, thereby reducing uncertainty in the learning objective. Knowledge updating ensures that new data is utilized to update knowledge, avoiding the negative impact of knowledge limitations on the learning process. The algorithm is validated through numerical tasks in OpenAI Gym and real-world mobile robot Goal Reach and obstacle avoidance tasks. The results confirm that the algorithm effectively combines knowledge and reinforcement learning, resulting in a 28.6% improvement in learning efficiency, a 19.56% enhancement in performance, and increased training stability.

An Exploratory Study of Contextual Control Modes in Teamwork.

This study investigated the relationship between human team member contextual control and team performance under time constraints. Contextual control modes, which describe different strategies for action selection in dynamic environments, characterize how humans maintain performance under variable demands. Control modes have not yet been studied in teamwork settings. Modeling of the cross-level interaction between team members' control modes and emerging team behaviors can improve understanding of effective teamwork in dynamic environments. A human-subjects study explored the relationship between individual contextual control and team performance. Questionnaires about contextual control were used to elicit individual control modes. Analysis compared team members' control modes and investigated how control modes changed under varying time pressures. Participant's control modes differed in their look ahead horizon, the extensiveness of prior action evaluation, and their prior experience. Many team members shifted control modes during trials, resulting in both convergence and divergence of paired control modes. No effects on communication rate were found due to changes in team members' control modes, but partially significant findings may suggest that the control mode divergence affects performance. Teams can operate in multiple control mode configurations that change dynamically according to context. Further research with an increased sample size is warranted to analyze how time constraints influence team members' control modes and overall teaming processes and whether divergence of team control mode is favorable under time pressures. Further study of contextual control in teams may help improve team design to better support teams in coping with time constraints in dynamic environments.

Traffic signal phase control at urban isolated intersections: an adaptive strategy utilizing the improved D3QN algorithm

Abstract The intelligent control of traffic signals at urban single intersections has emerged as an effective approach to mitigating urban traffic congestion. However, the existing fixed phase control strategy of traffic signal lights lacks capability to dynamically adjust signal phase switching based on real-time traffic conditions leading to traffic congestion. In this paper, an adaptive real-time control method employed by the traffic signal phase at a single intersection is considered based on the improved Double Dueling Deep Q Network (D3QN) algorithm. Firstly, the traffic signal phase control problem is modeled as a Markov decision process, with its state, action, and reward defined. Subsequently, to enhance the convergence speed and learning performance of the D3QN algorithm, attenuation action selection strategy and priority experience playback technology based on tree summation structure are introduced. Then, traffic flow data from various traffic scenarios are utilized to train the traffic signal control model based on the improved D3QN to obtain the optimal signal phase switch strategy. Finally, the effectiveness and optimal performance of the improved D3QN-based traffic signal control strategy are validated across diverse traffic scenarios. The simulation results show that, compared with the control strategy based on AC, DQN, DDQN, D3QN, and C-D3QN algorithms, the cumulative reward of the proposed I-D3QN strategy is increased by at least 6.57%, and the average queue length and average waiting time are reduced by at least 9.64% and 7.61%, which can effectively reduce the congestion at isolated intersections and significantly improve traffic efficiency.

Deconstructing Deep Active Inference: A Contrarian Information Gatherer.

Active inference is a theory of perception, learning, and decision making that can be applied to neuroscience, robotics, psychology, and machine learning. Recently, intensive research has been taking place to scale up this framework using Monte Carlo tree search and deep learning. The goal of this activity is to solve more complicated tasks using deep active inference. First, we review the existing literature and then progressively build a deep active inference agent as follows: we (1) implement a variational autoencoder (VAE), (2) implement a deep hidden Markov model (HMM), and (3) implement a deep critical hidden Markov model (CHMM). For the CHMM, we implemented two versions, one minimizing expected free energy, CHMM[EFE] and one maximizing rewards, CHMM[reward]. Then we experimented with three different action selection strategies: the ε-greedy algorithm as well as softmax and best action selection. According to our experiments, the models able to solve the dSprites environment are the ones that maximize rewards. On further inspection, we found that the CHMM minimizing expected free energy almost always picks the same action, which makes it unable to solve the dSprites environment. In contrast, the CHMM maximizing reward keeps on selecting all the actions, enabling it to successfully solve the task. The only difference between those two CHMMs is the epistemic value, which aims to make the outputs of the transition and encoder networks as close as possible. Thus, the CHMM minimizing expected free energy repeatedly picks a single action and becomes an expert at predicting the future when selecting this action. This effectively makes the KL divergence between the output of the transition and encoder networks small. Additionally, when selecting the action down the average reward is zero, while for all the other actions, the expected reward will be negative. Therefore, if the CHMM has to stick to a single action to keep the KL divergence small, then the action down is the most rewarding. We also show in simulation that the epistemic value used in deep active inference can behave degenerately and in certain circumstances effectively lose, rather than gain, information. As the agent minimizing EFE is not able to explore its environment, the appropriate formulation of the epistemic value in deep active inference remains an open question.

Two-Stage Optimization Model Based on Neo4j-Dueling Deep Q Network

To alleviate the power flow congestion in active distribution networks (ADNs), this paper proposes a two-stage load transfer optimization model based on Neo4j-Dueling DQN. First, the Neo4j graph model was established as the training environment for Dueling DQN. Meanwhile, the power supply paths from the congestion point to the power source point were obtained using the Cypher language built into Neo4j, forming a load transfer space that served as the action space. Secondly, based on various constraints in the load transfer process, a reward and penalty function was formulated to establish the Dueling DQN training model. Finally, according to the ε−greedy action selection strategy, actions were selected from the action space and interacted with the Neo4j environment, resulting in the optimal load transfer operation sequence. In this paper, Python was used as the programming language, TensorFlow open-source software library was used to form a deep reinforcement network, and Py2neo toolkit was used to complete the linkage between the python platform and Neo4j. We conducted experiments on a real 79-node system, using three power flow congestion scenarios for validation. Under the three power flow congestion scenarios, the time required to obtain the results was 2.87 s, 4.37 s and 3.45 s, respectively. For scenario 1 before and after load transfer, the line loss, voltage deviation and line load rate were reduced by about 56.0%, 76.0% and 55.7%, respectively. For scenario 2 before and after load transfer, the line loss, voltage deviation and line load rate were reduced by 41.7%, 72.9% and 56.7%, respectively. For scenario 3 before and after load transfer, the line loss, voltage deviation and line load rate were reduced by 13.6%, 47.1% and 37.7%, respectively. The experimental results show that the trained model can quickly and accurately derive the optimal load transfer operation sequence under different power flow congestion conditions, thereby validating the effectiveness of the proposed model.

A Bio-Inspired Dopamine Model for Robots with Autonomous Decision-Making.

Decision-making systems allow artificial agents to adapt their behaviours, depending on the information they perceive from the environment and internal processes. Human beings possess unique decision-making capabilities, adapting to current situations and anticipating future challenges. Autonomous robots with adaptive and anticipatory decision-making emulating humans can bring robots with skills that users can understand more easily. Human decisions highly depend on dopamine, a brain substance that regulates motivation and reward, acknowledging positive and negative situations. Considering recent neuroscience studies about the dopamine role in the human brain and its influence on decision-making and motivated behaviour, this paper proposes a model based on how dopamine drives human motivation and decision-making. The model allows robots to behave autonomously in dynamic environments, learning the best action selection strategy and anticipating future rewards. The results show the model's performance in five scenarios, emphasising how dopamine levels vary depending on the robot's situation and stimuli perception. Moreover, we show the model's integration into the Mini social robot to provide insights into how dopamine levels drive motivated autonomous behaviour regulating biologically inspired internal processes emulated in the robot.

A Multi-Area Task Path-Planning Algorithm for Agricultural Drones Based on Improved Double Deep Q-Learning Net

With the global population growth and increasing food demand, the development of precision agriculture has become particularly critical. In precision agriculture, accurately identifying areas of nitrogen stress in crops and planning precise fertilization paths are crucial. However, traditional coverage path-planning (CPP) typically considers only single-area tasks and overlooks the multi-area tasks CPP. To address this problem, this study proposed a Regional Framework for Coverage Path-Planning for Precision Fertilization (RFCPPF) for crop protection UAVs in multi-area tasks. This framework includes three modules: nitrogen stress spatial distribution extraction, multi-area tasks environmental map construction, and coverage path-planning. Firstly, Sentinel-2 remote-sensing images are processed using the Google Earth Engine (GEE) platform, and the Green Normalized Difference Vegetation Index (GNDVI) is calculated to extract the spatial distribution of nitrogen stress. A multi-area tasks environmental map is constructed to guide multiple UAV agents. Subsequently, improvements based on the Double Deep Q Network (DDQN) are introduced, incorporating Long Short-Term Memory (LSTM) and dueling network structures. Additionally, a multi-objective reward function and a state and action selection strategy suitable for stress area plant protection operations are designed. Simulation experiments verify the superiority of the proposed method in reducing redundant paths and improving coverage efficiency. The proposed improved DDQN achieved an overall step count that is 60.71% of MLP-DDQN and 90.55% of Breadth-First Search–Boustrophedon Algorithm (BFS-BA). Additionally, the total repeated coverage rate was reduced by 7.06% compared to MLP-DDQN and by 8.82% compared to BFS-BA.

A hierarchical reinforcement learning-aware hyper-heuristic algorithm with fitness landscape analysis

The automation of meta-heuristic algorithm configuration holds the utmost significance in evolutionary computation. A hierarchical reinforcement learning-aware hyper-heuristic algorithm with fitness landscape analysis (HRLHH) is proposed to flexibly configure the suitable algorithms under various optimization scenarios. Two kinds of fitness landscape analysis techniques improved based on specific problem characteristics construct the state spaces for hierarchical reinforcement learning. Among them, an adaptive classification based on dynamic ruggedness of information entropy is designed to discern the complexity of problems, which serves as the basis for decision-making actions in upper-layer space. Additionally, an online dispersion metric based on knowledge is further presented to distinguish the precise landscape features in lower-layer space. In light of the characteristics of the state spaces, the hierarchical action spaces composed of meta-heuristics with disparate exploration and exploitation are designed, and various action selection strategies are introduced. Taking into account the real-time environment and algorithm evolution behavior, dynamic reward mechanisms based on evolutionary success rate and population convergence rate are utilized to enhance search efficiency. The experimental results on the IEEE Congress on Evolutionary Computation (CEC) 2017, CEC 2014, and large-scale CEC 2013 test suites demonstrate that the proposed HRLHH exhibits superiority in terms of accuracy, stability, and convergence speed, and possesses strong generalization.

Intelligent decision-making for a “Three-Variable” frequency-hopping pattern based on OC-CDRL

The frequency hopping pattern of the existing frequency hopping communication system is not designed according to the electromagnetic interference environment, resulting in blind anti-jamming. Therefore, to address this problem, a “three-variable” frequency-hopping pattern is proposed, where the frequency, hopping rate, and instantaneous bandwidth of the frequency-hopping signal vary randomly based on the background electromagnetic interference. The decision-making problem of the “three-variable” frequency-hopping pattern is modeled as a Markov decision process (MDP) by constructing the state-action-reward tuple. The designed frequency varies continuously within a small frequency band selected from a pseudo-random sequence to alleviate the problem of dimension explosion in decision-making. At the same time, discrete values for the hopping rate and instantaneous bandwidth are designed. To solve this MDP problem efficiently, a combined deep reinforcement learning algorithm (OC-CDRL) based on optimistic exploration and conservative estimation is proposed, which combines the features of TD3 and D3QN algorithms and designs the corresponding states, actions, and rewards to deal with continuous and discrete action spaces, respectively. To address the problem that the D3QN algorithm tends to fall into local optimal solutions, an optimistic exploration strategy (OES) for action selection is proposed to improve the degree of exploration. Moreover, the loss function is improved by conservatively estimating state–action pairs outside the experience replay buffer, reducing the overestimation of the optimistic action-value function and increasing the stability and convergence of the algorithm. Comparative simulation results of the algorithms in different electromagnetic interference environments show that the OC-CDRL algorithm effectively avoids most regions with higher interference and has better adaptability and anti-jamming capability.

AMARL: An Attention-Based Multiagent Reinforcement Learning Approach to the Min-Max Multiple Traveling Salesmen Problem.

In recent years, the multiple traveling salesmen problem (MTSP or multiple TSP) has received increasing research interest and one of its main applications is coordinated multirobot mission planning, such as cooperative search and rescue tasks. However, it is still challenging to solve MTSP with improved inference efficiency as well as solution quality in varying situations, e.g., different city positions, different numbers of cities, or agents. In this article, we propose an attention-based multiagent reinforcement learning (AMARL) approach, which is based on the gated transformer feature representations for min-max multiple TSPs. The state feature extraction network in our proposed approach adopts the gated transformer architecture with reordering layer normalization (LN) and a new gate mechanism. It aggregates fixed-dimensional attention-based state features irrespective of the number of agents and cities. The action space of our proposed approach is designed to decouple the interaction of agents' simultaneous decision-making. At each time step, only one agent is assigned to a non-zero action so that the action selection strategy can be transferred across tasks with different numbers of agents and cities. Extensive experiments on min-max multiple TSPs were conducted to illustrate the effectiveness and advantages of the proposed approach. Compared with six representative algorithms, our proposed approach achieves state-of-the-art performance in solution quality and inference efficiency. In particular, the proposed approach is suitable for tasks with different numbers of agents or cities without extra learning, and experimental results demonstrate that the proposed approach realizes powerful transfer capability across tasks.

Automated Test Case Generation for Path Coverage by Using Multi-Objective Particle Swarm Optimization Algorithm with Reinforcement Learning and Relationship Matrix Strategies

Software testing is an important phase in software life cycle, and test case generation occupies the main part of the testing workload. Path coverage is the most stringent and difficult coverage criteria, so automatic test case generation for path coverage (ATCG-PC) is an important task in software testing. ATCG-PC is usually abstracted as an optimization search problem, and the commonly used solution algorithm is swarm intelligence algorithm. This paper proposes a path coverage test case generation algorithm based on improved multi-objective particle swarm optimization (IMOPSO) algorithm. The algorithm uses the multi-objective strategy to generate test cases for a group of similar paths at a time, and is supplemented by the particle action selection strategy based on reinforcement learning and the secondary search algorithm based on relationship matrix, which makes use of the similarity information between paths to speed up the early convergence and help to quickly reach the optimal solution around the local optimal position. By grouping the paths according to the similarity, the algorithm reduces the test case consumption, and improves the efficiency of test case generation. Experimental results show that, compared with the existing methods, the proposed algorithm can achieve higher path coverage with less test case consumption and is also suitable for large-scale programs.

Action Selection in Everyday Activities: The Opportunistic Planning Model.

While action selection strategies in well-defined domains have received considerable attention, little is yet known about how people choose what to do next in ill-defined tasks. In this contribution, we shed light on this issue by considering everyday tasks, which in many cases have a multitude of possible solutions (e.g., it does not matter in which order the items are brought to the table when setting a table) and are thus categorized as ill-defined problems. Even if there are no hard constraints on the ordering of subtasks in everyday activities, our research shows that people exhibit specific preferences. We propose that these preferences arise from bounded rationality, that is, people only have limited knowledge and processing power available, which results in a preference to minimize the overall physical and cognitive effort. In the context of everyday activities, this can be achieved by (a) taking properties of the spatial environment into account to use them to one's advantage, and (b) employing a stepwise-optimal action selection strategy. We present the Opportunistic Planning Model as an explanatory cognitive model, which instantiates these assumptions, and show that the model is able to generalize to new everyday tasks, outperforming machine learning models such as neural networks duringgeneralization.

ConcaveQ: Non-monotonic Value Function Factorization via Concave Representations in Deep Multi-Agent Reinforcement Learning

Value function factorization has achieved great success in multi-agent reinforcement learning by optimizing joint action-value functions through the maximization of factorized per-agent utilities. To ensure Individual-Global-Maximum property, existing works often focus on value factorization using monotonic functions, which are known to result in restricted representation expressiveness. In this paper, we analyze the limitations of monotonic factorization and present ConcaveQ, a novel non-monotonic value function factorization approach that goes beyond monotonic mixing functions and employs neural network representations of concave mixing functions. Leveraging the concave property in factorization, an iterative action selection scheme is developed to obtain optimal joint actions during training. It is used to update agents’ local policy networks, enabling fully decentralized execution. The effectiveness of the proposed ConcaveQ is validated across scenarios involving multi-agent predator-prey environment and StarCraft II micromanagement tasks. Empirical results exhibit significant improvement of ConcaveQ over state-of-the-art multi-agent reinforcement learning approaches.

Real-Time Adjustment Method for Metro Systems with Train Delays Based on Improved Q-Learning

This paper presents a solution to address the challenges of unexpected events in the operation of metro trains, which can lead to increased delays and safety risks. An improved Q-learning algorithm is proposed to reschedule train timetables via incorporating train detention and different section running times as actions. To enhance computational efficiency and convergence rate, a simulated annealing dynamic factor is introduced to improve action selection strategies. Additionally, importance sampling is employed to evaluate different policies effectively. A case study of Shenzhen Metro is conducted to demonstrate the effectiveness of the proposed method. The results show that the method achieves convergence, fast computation speed, and real-time adjustment capabilities. Compared to traditional methods such as no adjustment, manual adjustment, and FIFO (First-In-First-Out), the proposed method significantly reduces the average total train delay by 54% and leads to more uniform train headways. The proposed method utilizes a limited number of variables for practical state descriptions, making it well suited for real-world applications. It also exhibits good scalability and transferability to other metro systems.

COLREGs-Based Path Planning for USVs Using the Deep Reinforcement Learning Strategy

This research introduces a two-stage deep reinforcement learning approach for the cooperative path planning of unmanned surface vehicles (USVs). The method is designed to address cooperative collision-avoidance path planning while adhering to the International Regulations for Preventing Collisions at Sea (COLREGs) and considering the collision-avoidance problem within the USV fleet and between USVs and target ships (TSs). To achieve this, the study presents a dual COLREGs-compliant action-selection strategy to effectively manage the vessel-avoidance problem. Firstly, we construct a COLREGs-compliant action-evaluation network that utilizes a deep learning network trained on pre-recorded TS avoidance trajectories by USVs in compliance with COLREGs. Then, the COLREGs-compliant reward-function-based action-selection network is proposed by considering various TS encountering scenarios. Consequently, the results of the two networks are fused to select actions for cooperative path-planning processes. The path-planning model is established using the multi-agent proximal policy optimization (MAPPO) method. The action space, observation space, and reward function are tailored for the policy network. Additionally, a TS detection method is introduced to detect the motion intentions of TSs. The study conducted Monte Carlo simulations to demonstrate the strong performance of the planning method. Furthermore, experiments focusing on COLREGs-based TS avoidance were carried out to validate the feasibility of the approach. The proposed TS detection model exhibited robust performance within the defined task.

Model-based optimal action selection for Dyna-Q reverberation suppression cognitive sonar

The Doppler shift of low-speed targets is frequently disturbed by the reverberation Doppler spread clutter under the shallow sea. The clutter is generated by underwater scatterers, which increases the difficulty of Doppler estimation. To solve this problem, a reverberation target resolution function based on the Doppler spread clutter statistical model is proposed in this paper. Through the width of reverberation Doppler clutter, this function adjusts the waveform parameters by determining whether the target is discriminable. In addition, the reverberation Doppler spread clutter is time-spatial varying and affected by grazing angle, waves, wind speed, fish and other effects. Thus, the sonar waveform parameters need to be adjusted constantly. Therefore, this paper combines the cognitive sonar based on reinforcement learning with the reverberation target resolution function to evaluate different waveforms in different environments. Consequently, the sonar can adjust the waveform parameters in real-time and obtain the optimal waveform in different environments. Meanwhile, in this paper, the action selection strategy of Dyna-Q reinforcement learning is optimized, and the model-based maximum action selection Dyna-Q algorithm (Dyna-Q-Max-Action) is proposed. Compared with the traditional Dyna-Q and Q-learning algorithms, the proposed algorithm needs fewer episodes. Finally, numerical simulation verified the effectiveness of the proposed algorithm.

Reinforcement learning based CPG-controlled method with high adaptability and robustness: An experimental study on a robotic fishtail

An adaptive and robust control is imperative for robots operating in complex environments. The artificial intelligence approach presents a promising solution, while training and practical application in an actual robot remain a significant challenge. This paper proposes a reinforcement learning based control strategy with lightweight computation, aiming to optimize thrust in complex flows and varying self-structural characteristics of robotic fishtails. Improved Q-learning algorithm and CPG control are utilized to enable the robotic fishtail to make autonomous and accurate decisions in response to unknown changes. The control strategy is trained in actual physical flow fields, with an action selection strategy and a reward system proposed to accelerate convergence and enhance training stability. An integrated system of online learning, response measuring, and real-time monitoring is developed for the training and testing processes. Variable environmental tests are performed under different turbulent flows. Distinct caudal fins are utilized to conduct tests on self-structural variation. The experimental results confirm the robustness and adaptability of the control strategy, as well as its superiority over the PID approach in terms of accuracy, stability, and response speed. The control strategy architecture and physical experimental method could offer experience and application reference for the intelligent control of actual robots.

Dynamic Batch Learning in High-Dimensional Sparse Linear Contextual Bandits

We study the problem of dynamic batch learning in high-dimensional sparse linear contextual bandits, where a decision maker, under a given maximum-number-of-batch constraint and only able to observe rewards at the end of each batch, can dynamically decide how many individuals to include in the next batch (at the end of the current batch) and what personalized action-selection scheme to adopt within each batch. Such batch constraints are ubiquitous in a variety of practical contexts, including personalized product offerings in marketing and medical treatment selection in clinical trials. We characterize the fundamental learning limit in this problem via a regret lower bound and provide a matching upper bound (up to log factors), thus prescribing an optimal scheme for this problem. To the best of our knowledge, our work provides the first inroad into a theoretical understanding of dynamic batch learning in high-dimensional sparse linear contextual bandits. Notably, even a special case of our result—when no batch constraint is present—yields that the simple exploration-free algorithm using the LASSO estimator already achieves the minimax optimal [Formula: see text] regret bound (s0 is the sparsity parameter or an upper bound thereof and T is the learning horizon) for standard online learning in high-dimensional linear contextual bandits (for the no-margin case), a result that appears unknown in the emerging literature of high-dimensional contextual bandits. This paper was accepted by Baris Ata, stochastic models and simulation. Funding: This work is supported by the National Science Foundation [Grant CCF-2106508]. Z. Zhou gratefully acknowledges the Digital Twin research grant from Bain & Company and the New York University’s 2022-2023 Center for Global Economy and Business faculty research grant for support on this work. Z. Ren was supported by the National Science Foundation [Grant OAC 1934578] and by the Discovery Innovation Fund for Biomedical Data Sciences.

Dyna-Validator: A Model-based Reinforcement Learning Method with Validated Simulated Experiences

Dyna is a planning paradigm that naturally weaves learning and planning together through environmental models. Dyna-style reinforcement learning improves the sample efficiency using the simulation experience generated by the environment model to update the value function. However, the existing Dyna-style planning methods are usually based on tabular methods, only suitable for tasks with low-dimensional and small-scale space. In addition, the quality of the simulation experience generated by the existing methods cannot be guaranteed, which significantly limits its application in tasks such as continuous control of high-dimensional robots and autonomous driving. To this end, we propose a model-based approach that controls planning through a validator. The validator filters high-quality experiences for policy learning and decides whether to stop planning. To deal with the exploration and exploitation dilemma in reinforcement learning, a combination of ϵ-greedy strategy and simulated annealing (SA) cooling schedule control is designed as an action selection strategy. The excellent performance of the proposed method is demonstrated in a set of classical Atari games. Experimental results show that learning dynamic models in some games can improve sample efficiency. This benefit is maximized by choosing the proper planning steps. In the optimization planning phase, our method maintains a smaller gap with the current state-of-the-art model-based reinforcement learning (MuZero). In order to achieve a good compromise between model accuracy and optimal programming step size, it is necessary to control the programming reasonably. The practical application of this method in a physical robot system helps reduce the influence of an imprecise depth prediction model on the task. Without human supervision, it is easier to collect training data and learn complex skills (such as grabbing and carrying items) while being more effective at scaling tasks that have never been seen before.

Q-functionals for Value-Based Continuous Control

We present Q-functionals, an alternative architecture for continuous control deep reinforcement learning. Instead of returning a single value for a state-action pair, our network transforms a state into a function that can be rapidly evaluated in parallel for many actions, allowing us to efficiently choose high-value actions through sampling. This contrasts with the typical architecture of off-policy continuous control, where a policy network is trained for the sole purpose of selecting actions from the Q-function. We represent our action-dependent Q-function as a weighted sum of basis functions (Fourier, Polynomial, etc) over the action space, where the weights are state-dependent and output by the Q-functional network. Fast sampling makes practical a variety of techniques that require Monte-Carlo integration over Q-functions, and enables action-selection strategies besides simple value-maximization. We characterize our framework, describe various implementations of Q-functionals, and demonstrate strong performance on a suite of continuous control tasks.