Action Selection Research Articles

Optimizing Camera Motion with MCTS and Target Motion Modeling in Multi-Target Active Object Tracking

In this work, we are dedicated to multi-target active object tracking (AOT), where the goal is to achieve continuous tracking of targets through real-time control of camera. This form of active camera control can be applied to unmanned aerial vehicles (UAV), intelligent robots, and sports events. Our work is conducted in an environment featuring multiple cameras and targets, where our goal is to maximize target coverage. Contrasting with previous research, our work introduces additional degrees of freedom for the cameras, allowing them not only to rotate but also to move along boundary lines. In addition, we model the motion of target to predict the future position of the target in environment. With target’s future position, we use Monte Carlo Tree Search (MCTS) method to find the optimal action of camera. Since the action space is large, we propose to leverage the action selection from multi-agent reinforcement learning (MARL) network to prune the search tree of Monte Carlo Tree Search method, so as to find the optimal action more efficiently. We establish a multi-target 2D environment to simulate several sports games, and experimental results demonstrate that our method can effectively improve the target coverage. The code is available at: http://github.com/HopeChanger/ActiveObjectTracking .

To lie or to tell the truth? The influence of processing the opponent's feedback on the forthcoming choice.

The brain mechanisms of deceptive behavior are relatively well studied, and the key brain regions involved in its processing were established. At the same time, the brain mechanisms underlying the processes of preparation for deception are less known. We studied BOLD-signal changes during the presentation of the opponent's feedback to a previous deceptive or honest action during the computer game. The goal of the game was to mislead the opponent either by means of deception or by means of telling the truth. As a result, it was shown that several brain regions that were previously demonstrated as involved in deception execution, such as the left anterior cingulate cortex and anterior insula, also underlie processes related to deception preparation. The results obtained also allowed us to suggest that brain regions responsible for performance monitoring, intention assessment, suppression of non-selected solutions, and reward processing could be involved in shaping future action selection and preparation for deception. By shedding light on the brain mechanisms underlying deception, our study contributes to a deeper understanding of this complex cognitive process. Furthermore, it emphasizes the significance of exploring brain mechanisms governing the choice between deception and truth at various stages of decision-making.

An algorithm for multi-armed bandit based on variance change sensitivity

The Multi-Arm Bandit problem is becoming increasingly popular as it enables real-world sequential decision making across application domains, including clinical trials, recommender systems, and online decision making. The Multi-Arm Bandit problem is a classical problem of exploration and exploitation dilemma in reinforcement learning and it needs to decide on the optimal strategy based on the reward situation of each rocker arm. However, the existing Multi-Arm Bandit algorithms have many shortcomings, such as blind exploration, weak generalization ability and endless exploration. Aiming at the shortcomings of the existing Multi-Armed Bandit algorithms, this paper proposes a Multi-Armed Bandit algorithm based on variance change sensitivity. The algorithm takes the variance change of reward as a clue, and adjusts the exploration probability by the average variance change of all actions, and selects the action with the largest variance change when exploring. At the same time, in order to reduce the waste of action selection times and maximize the cumulative reward, A parameter Ncon was introduced to record the number of consecutive selections of the same action. The exploration stops when Ncon reaches a certain value. Experiments show that the exploration algorithm can obtain higher reward value and lower regret value in the end.

Time and Distance Perception in Highly Dynamic Environments

Abstract Space- and time-based tasks, like flying airplanes, driving cars, or the manual controlled rendezvous and docking of spacecrafts can be very demanding for pilots, drivers, or astronauts, depending on selected trajectories, approaching speeds, or distracting information. One characteristic of highly dynamic environments is that persons have to consider their spatial position in relation to surrounding elements based on passing time, detected speed, and trajectories to avoid collisions with other objects. An inadequate perception of the own position in relation to other objects and passing time can lead to impaired decision-making processes, resulting in errors or collisions. A framework for the explanation of cognitive processes involved in perception, anticipation, and decision-making in dynamic environments is provided by the concept of situation awareness. Based on this framework persons must perceive relevant elements, integrate them in a coherent mental representation and anticipate the development of the current situation in the near future, to plan safe upcoming maneuvers. Thus, for safe maneuvers anticipation processes are playing a major role in decision-making and action selection. In this paper it is proposed that time and distance perceptions are central features of anticipation processes in dynamic situations. It is suggested to combine theories like the attentional-gate model and time-to-contact with the framework of situation awareness to explain anticipation processes in dynamic environments. The major aim is to develop precise cognitive models of persons acting in dynamic situations. Such cognitive models can be the basis for the development and parameterization of assistance systems, supporting anticipation in dynamic environments.

Robust Multi-Agent Communication With Graph Information Bottleneck Optimization.

Recent research on multi-agent reinforcement learning (MARL) has shown that action coordination of multi-agents can be significantly enhanced by introducing communication learning mechanisms. Meanwhile, graph neural network (GNN) provides a promising paradigm for communication learning of MARL. Under this paradigm, agents and communication channels can be regarded as nodes and edges in the graph, and agents can aggregate information from neighboring agents through GNN. However, this GNN-based communication paradigm is susceptible to adversarial attacks and noise perturbations, and how to achieve robust communication learning under perturbations has been largely neglected. To this end, this paper explores this problem and introduces a robust communication learning mechanism with graph information bottleneck optimization, which can optimally realize the robustness and effectiveness of communication learning. We introduce two information-theoretic regularizers to learn the minimal sufficient message representation for multi-agent communication. The regularizers aim at maximizing the mutual information (MI) between the message representation and action selection while minimizing the MI between the agent feature and message representation. Besides, we present a MARL framework that can integrate the proposed communication mechanism with existing value decomposition methods. Experimental results demonstrate that the proposed method is more robust and efficient than state-of-the-art GNN-based MARL methods.

Q-learning guided algorithms for bi-criteria minimization of total flow time and makespan in no-wait permutation flowshops

Combining Deep Reinforcement Learning and meta-heuristic techniques represents a new research direction for enhancing the search capabilities of meta-heuristic methods in the context of production scheduling. Q-learning is a prominent reinforcement learning in which its utilization aims to direct the selection of actions, thus preventing the necessity for a random exploration in the iterative process of the metaheuristics. In this study, we provide Q-learning guided algorithms for the Bi-Criteria No-Wait Flowshop Scheduling Problem (NWFSP). The problem is treated as a bi-criteria combinatorial optimization problem where total flow time and makespan are optimized simultaneously. Firstly, a deterministic mixed-integer linear programming (MILP) model is provided. Then, Q-learning guided algorithms are developed: Bi-Criteria Iterated Greedy Algorithm with Q-Learning (BC-IGQL). Bi-Criteria Block Insertion Heuristic Algorithm with Q-Learning (BC-BIHQL). Moreover, the performance of the proposed Q-learning guided algorithms is compared over a collection of Bi-Criteria Genetic Local Search Algorithms (BC-GLS), Bi-Criteria Iterated Greedy Algorithm (BC-IG), Bi-Criteria Iterated Greedy Algorithm with a Local Search (BC-IGALL) and Bi-Criteria Variable Block Insertion Heuristic Algorithm (BC-VBIH). The complete computational experiment, performed on 480 problem instances of Vallada et al. (2015), which is known as the VRF benchmark set, indicates that the BC-BIHQL and the BC-IGQL algorithms outperform the BC-GLS, BC-IG, BC-IGALL, and BC-VBIH algorithms in comparative performance metrics. More specifically, the proposed BC-BIHQL and BC-IGQL algorithms can yield more non-dominated bi-criteria solutions with the most substantial competitiveness than the remaining algorithms. At the same time, both are competitive with each other on the benchmark problems. Moreover, the BC-IGQL algorithm dominates almost 97% and 99% of the solutions reached by the BC-IG, BC-IGALL, and BC-VBIH algorithms in small and large datasets. Similarly, The BC-BIHQL algorithm dominates almost 98% and 99% of the solutions reached by the BC-IG, BC-IGALL, and BC-VBIH algorithms in small and large datasets, respectively. This means that, among all the features that have been compared, the Q-learning-guided algorithms demonstrate the highest level of competitiveness. The outcomes of this study encourage us to discover many more bi-criteria NWFSPs to reveal the trade-off between other conflicting objectives, such as makespan & the number of early jobs, to overcome various industries' problems.

Differential effects of acute and prolonged morphine withdrawal on motivational and goal-directed control over reward-seeking behaviour.

Opioid addiction is a relapsing disorder marked by uncontrolled drug use and reduced interest in normally rewarding activities. The current study investigated the impact of spontaneous withdrawal from chronic morphine exposure on emotional, motivational and cognitive processes involved in regulating the pursuit and consumption of food rewards in male rats. In Experiment 1, rats experiencing acute morphine withdrawal lost weight and displayed somatic signs of drug dependence. However, hedonically driven sucrose consumption was significantly elevated, suggesting intact and potentially heightened reward processing. In Experiment 2, rats undergoing acute morphine withdrawal displayed reduced motivation when performing an effortful response for palatable food reward. Subsequent reward devaluation testing revealed that acute withdrawal disrupted their ability to exert flexible goal-directed control over reward seeking. Specifically, morphine-withdrawn rats were impaired in using current reward value to select actions both when relying on prior action-outcome learning and when given direct feedback about the consequences of their actions. In Experiment 3, rats tested after prolonged morphine withdrawal displayed heightened rather than diminished motivation for food rewards and retained their ability to engage in flexible goal-directed action selection. However, brief re-exposure to morphine was sufficient to impair motivation and disrupt goal-directed action selection, though in this case, rats were only impaired in using reward value to select actions in the presence of morphine-paired context cues and in the absence of response-contingent feedback. We suggest that these opioid-withdrawal induced deficits in motivation and goal-directed control may contribute to addiction by interfering with the pursuit of adaptive alternatives to drug use.

Exploring Dogs' Capacity to Innovate Actions on Cue

Creative animals can flexibly respond to novel circumstances, a capacity that is closely tied to cognitive complexity. We examined the capacity of four dogs to learn to innovate actions when instructed to do so using a “create” cue. An action performed in response to this cue was considered novel if the action had not yet been offered within the current training session and was not already associated with a specific cue. The dogs were able to self-select many novel actions, similarly to what has been observed in trained dolphins. Training animals to innovate actions on cue can help reveal aspects of the cognitive processes underlying action selection, cognitive control, and behavioral flexibility. Successful learning of the “create” cue may indicate that the dogs were able to monitor their recent actions and form an abstract concept of avoiding recently performed actions. Reinforcing dogs for spontaneity may increase the range of actions that they perform within the training context. We conclude that the dogs in this study were capable of learning to innovate actions in ways that are comparable to what has been shown in dolphins. Though it is unclear how other dogs would perform at this task, our results demonstrate it is possible for dogs to learn to innovate actions on cue.

Abstract deliberation by visuomotor neurons in prefrontal cortex

During visually guided behavior, the prefrontal cortex plays a pivotal role in mapping sensory inputs onto appropriate motor plans. When the sensory input is ambiguous, this involves deliberation. It is not known whether the deliberation is implemented as a competition between possible stimulus interpretations or between possible motor plans. Here we study neural population activity in the prefrontal cortex of macaque monkeys trained to flexibly report perceptual judgments of ambiguous visual stimuli. We find that the population activity initially represents the formation of a perceptual choice before transitioning into the representation of the motor plan. Stimulus strength and prior expectations both bear on the formation of the perceptual choice, but not on the formation of the action plan. These results suggest that prefrontal circuits involved in action selection are also used for the deliberation of abstract propositions divorced from a specific motor plan, thus providing a crucial mechanism for abstract reasoning.

Research on Intelligent Identification and Analysis Algorithm of Tunnel Engineering Geological Information

When tunneling, detecting abrupt changes in geological circumstances can be difficult. In recent years, the proliferation of tunneling characteristics has been strongly related to the surrounding geology. These parameters offer significant potential for using data-driven artificial intelligence (AI) approaches to infer patterns from information without reference to known consequences. This research introduces the Simulated Fire Hawk Optimizer-based Deep Action Selection Network (SFHO-DASN) model, which uses a Simulated Fire Hawk Optimizer to anticipate the geological conditions needed for tunneling. The optimized hybrid neural network technique can anticipate geological conditions effectively, as demonstrated by a case study of the constructed model. It is especially significant for rock types, water intrusion, karst caverns, and surface subsidence, for which the predicted accuracy is greater than 95%. These findings imply that the geological circumstances behind the tunnel face could be reliably and correctly predicted by a DASN that has received the proper training. This procedure's most significant advantage is its ability to adjust every scored parameter's weighting in response to variations in geological circumstances. The accuracy performance of the proposed neural network outperforms the conventional neural network, as indicated by the area under the curve (AUC) and performance analysis. A proposed model for geology prediction can attain predictive accuracy with a small number of tunneling parameters, according to an analysis of the feature relevance of each tunneling parameter. The suggested approach ought to be more practical for proposals about tunnel support architecture in the East Asian geological region and for future tunnel building.

Excitatory/inhibitory motor balance reflects individual differences during joint action coordination.

Joint action (JA) is a continuous process of motor co-regulation based on the integration of contextual (top-down) and kinematic (bottom-up) cues from partners. The fine equilibrium between excitation and inhibition in sensorimotor circuits is, thus, central to such a dynamic process of action selection and execution. In a bimanual task adapted to become a unimanual JA task, the participant held a bottle (JA), while a confederate had to reach and unscrew either that bottle or another stabilized by a mechanical clamp (No_JA). Prior knowledge was manipulated in each trial such that the participant knew (K) or not (No_K) the target bottle in advance. Online transcranial magnetic stimulation (TMS) was administered at action-relevant landmarks to explore corticospinal excitability (CSE) and inhibition (cortical silent period [cSP]). CSE was modulated early on before the action started if prior information was available. In contrast, cSP modulation emerged later during the reaching action, regardless of prior information. These two indexes could thus reflect the concurrent elaboration of contextual priors (top-down) and the online sampling of partner's kinematic cues (bottom-up). Furthermore, participants selected either one of two possible behavioural strategies, preferring early or late force exertion on the bottle. One translates into a reduced risk of motor coordination failure and the other into reduced metabolic expenditure. Each strategy was characterised by a specific excitatory/inhibitory profile. In conclusion, the study of excitatory/inhibitory balance paves the way for the neurophysiological determination of individual differences in the combination of top-down and bottom-up processing during JA coordination.

Salience Interest Option: Temporal abstraction with salience interest functions

Reinforcement Learning (RL) is a significant machine learning subfield that emphasizes learning actions based on environment to obtain optimal behavior policy. RL agents can make decisions at variable time scales in the form of temporal abstractions, also known as options. The issue of discovering options has seen a considerable research effort. Most notably, the Interest Option Critic (IOC) algorithm first extends the initial set to the interest function, providing a method for learning options specialized to certain state space regions. This approach offers a specific attention mechanism for action selection. Unfortunately, this method still suffers from the classic issues of poor data efficiency and lack of flexibility in RL when learning options end-to-end through backpropagation. This paper proposes a new approach called Salience Interest Option Critic (SIOC), which chooses subsets of existing initiation sets for RL. Specifically, these subsets are not learned by backpropagation, which is slow and tends to overfit, but through particle filters. This approach enables the rapid and flexible identification of critical subsets using only reward feedback. We conducted experiments in discrete and continuous domains, and our proposed method demonstrate higher efficiency and flexibility than other methods. The generated options are more valuable within a single task and exhibited greater interpretability and reusability in multi-task learning scenarios.

Rethinking the external globus pallidus and information flow in cortico-basal ganglia-thalamic circuits.

For decades, the external globus pallidus (GPe) has been viewed as a passive way-station in the indirect pathway of the cortico-basal ganglia-thalamic (CBGT) circuit, sandwiched between striatal inputs and basal ganglia outputs. According to this model, one-way descending striatal signals in the indirect pathway amplify the suppression of downstream thalamic nuclei by inhibiting GPe activity. Here, we revisit this assumption, in light of new and emerging work on the cellular complexity, connectivity and functional role of the GPe in behaviour. We show how, according to this new circuit-level logic, the GPe is ideally positioned for relaying ascending and descending control signals within the basal ganglia. Focusing on the problem of inhibitory control, we illustrate how this bidirectional flow of information allows for the integration of reactive and proactive control mechanisms during action selection. Taken together, this new evidence points to the GPe as being a central hub in the CBGT circuit, participating in bidirectional information flow and linking multifaceted control signals to regulate behaviour.

Autistic traits in neurotypical adults are related to impaired perceptual–motor coordination

Individuals with autism spectrum disorder (ASD) are more prone to injury due to falls or collisions with objects. This may be caused by impaired perceptual–motor coordination, including inaccurate body-related spatial perception and insufficient anticipatory motor planning due to a detail-focused processing style. To investigate this hypothesis, an action-selection task was developed to create conditions likely to induce collisions with obstacles and to assess perceptual accuracy and predictive attentional properties for the action selection of subsequent movements in a sequence of actions. Sixteen participants completed an autism-spectrum quotient (AQ) survey and the task. Results indicated that AQ correlated significantly with body-related spatial perception, and it was a significant predictor of the number of collisions. For the association between collisions and the eye-tracking data, results of a generalized linear mixed model showed that collisions were more likely to happen when the total fixation time to exit width was short and when the time to first fixation was late. This suggests that participants with higher autistic traits were not able to properly plan subsequent movements. Overall, these findings suggested that impaired perceptual–motor coordination, which could occur due to difficulties in perceiving body-related spatial relationships and anticipatory motor planning, may underlie the high susceptibility to injury observed in individuals with ASD.

Music training is related to late ERP modulation and enhanced performance during Simon task but not Stroop task.

Increasing evidence suggests that music training correlates with better performance in tasks measuring executive function components including inhibitory control, working memory and selective attention. The Stroop and Simon tasks measure responses to congruent and incongruent information reflecting cognitive conflict resolution. However, there are more reports of a music-training advantage in the Simon than the Stroop task. Reports indicate that these tasks may differ in the timing of conflict resolution: the Stroop task might involve early sensory stage conflict resolution, while the Simon task may do so at a later motor output planning stage. We hypothesize that musical experience relates to conflict resolution at the late motor output stage rather than the early sensory stage. Behavioral responses, and event-related potentials (ERP) were measured in participants with varying musical experience during these tasks. It was hypothesized that musical experience correlates with better performance in the Simon but not the Stroop task, reflected in ERP components in the later stage of motor output processing in the Simon task. Participants were classified into high- and low-music training groups based on the Goldsmith Musical Sophistication Index. Electrical brain activity was recorded while they completed visual Stroop and Simon tasks. The high-music training group outperformed the low-music training group on the Simon, but not the Stroop task. Mean amplitude difference (incongruent-congruent trials) was greater for the high-music training group at N100 for midline central (Cz) and posterior (Pz) sites in the Simon task and midline central (Cz) and frontal (Fz) sites in the Stroop task, and at N450 at Cz and Pz in the Simon task. N450 difference peaks occurred earlier in the high-music training group at Pz. Differences between the groups at N100 indicate that music training may be related to better sensory discrimination. These differences were not related to better behavioral performance. Differences in N450 responses between the groups, particularly in regions encompassing the motor and parietal cortices, suggest a role of music training in action selection during response conflict situations. Overall, this supports the hypothesis that music training selectively enhances cognitive conflict resolution during late motor output planning stages.

Neural Correlates of Online Action Preparation.

When performing movements in rapid succession, the brain needs to coordinate ongoing execution with the preparation of an upcoming action. Here we identify the processes and brain areas involved in this ability of online preparation. Human participants (both male and female) performed pairs of single-finger presses or three-finger chords in rapid succession, while 7T fMRI was recorded. In the overlap condition, they could prepare the second movement during the first response and in the nonoverlap condition only after the first response was completed. Despite matched perceptual and movement requirements, fMRI revealed increased brain activity in the overlap condition in regions along the intraparietal sulcus and ventral visual stream. Multivariate analyses suggested that these areas are involved in stimulus identification and action selection. In contrast, the dorsal premotor cortex, known to be involved in planning upcoming movements, showed no discernible signs of heightened activity. This observation suggests that the bottleneck during simultaneous action execution and preparation arises at the level of stimulus identification and action selection, whereas movement planning in the premotor cortex can unfold concurrently with the execution of a current action without requiring additional neural activity.

Bayesian Reinforcement Learning With Limited Cognitive Load.

All biological and artificial agents must act given limits on their ability to acquire and process information. As such, a general theory of adaptive behavior should be able to account for the complex interactions between an agent's learning history, decisions, and capacity constraints. Recent work in computer science has begun to clarify the principles that shape these dynamics by bridging ideas from reinforcement learning, Bayesian decision-making, and rate-distortion theory. This body of work provides an account of capacity-limited Bayesian reinforcement learning, a unifying normative framework for modeling the effect of processing constraints on learning and action selection. Here, we provide an accessible review of recent algorithms and theoretical results in this setting, paying special attention to how these ideas can be applied to studying questions in the cognitive and behavioral sciences.

Demand response for residential building heating: Effective Monte Carlo Tree Search control based on physics-informed neural networks

To reduce global carbon emissions and limit climate change, controlling energy consumption in buildings is an important piece of the puzzle. Here, we specifically focus on using a demand response (DR) algorithm to limit the energy consumption of a residential building's heating system while respecting user's thermal comfort. In that domain, Reinforcement learning (RL) methods have been shown to be quite effective. One such RL method is Monte Carlo Tree Search (MCTS), which has achieved impressive success in playing board games (go, chess). A particular advantage of MCTS is that its decision tree structure naturally allows to integrate exogenous constraints (e.g., by trimming branches that violate them), while conventional RL solutions need more elaborate techniques (e.g., indirectly by adding penalties in the cost/reward function, or through a backup controller that corrects constraint-violating actions). The main aim of this paper is to study the adoption of MCTS for building control, since this (to the best of our knowledge) has remained largely unexplored. A specific property of MCTS is that it needs a simulator component that can predict subsequent system states, based on actions taken. A straightforward data-driven solution is to use black-box neural networks (NNs). We will however extend a Physics-informed Neural Network (PiNN) model to deliver multi-timestep predictions, and show the benefit it offers in terms of lower prediction errors (−32% MAE) as well as better MCTS performance (−4% energy cost, +7% thermal comfort) compared to a black-box NN. A second contribution will be to extend a vanilla MCTS version to adopt the ideas applied in AlphaZero (i.e., using learned prior and value functions and an action selection heuristic) to obtain lower computational costs while maintaining control performance.

A hippocampus-accumbens code guides goal-directed appetitive behavior

The dorsal hippocampus (dHPC) is a key brain region for the expression of spatial memories, such as navigating towards a learned reward location. The nucleus accumbens (NAc) is a prominent projection target of dHPC and implicated in value-based action selection. Yet, the contents of the dHPC→NAc information stream and their acute role in behavior remain largely unknown. Here, we found that optogenetic stimulation of the dHPC→NAc pathway while mice navigated towards a learned reward location was both necessary and sufficient for spatial memory-related appetitive behaviors. To understand the task-relevant coding properties of individual NAc-projecting hippocampal neurons (dHPC→NAc), we used in vivo dual-color two-photon imaging. In contrast to other dHPC neurons, the dHPC→NAc subpopulation contained more place cells, with enriched spatial tuning properties. This subpopulation also showed enhanced coding of non-spatial task-relevant behaviors such as deceleration and appetitive licking. A generalized linear model revealed enhanced conjunctive coding in dHPC→NAc neurons which improved the identification of the reward zone. We propose that dHPC routes specific reward-related spatial and behavioral state information to guide NAc action selection.

AoI minimization of ambient backscatter-assisted EH-CRN with cooperative spectrum sensing

To evaluate the timeliness and freshness of information from energy-constrained devices, we minimize the weighted average age of information (AoI) of the ambient backscatter-assisted energy harvesting cognitive radio network (AB-EH-CRN), where secondary transmitters (STs) follow the non-orthogonal multiple access (NOMA) strategy to transmit status updates. The secondary receiver (SR) schedules the selected actions for STs to perform spectrum access, and receives status updates from them. We formulate the AoI minimization problem where the action selection and transmission power of STs are considered as variables. To exploit spectrum resources in the AB-EH-CRN with unknown signal-to-noise ratios (SNRs) at the STs, we propose the deep neural network (DNN)-based cooperative spectrum sensing (CSS), and adopt an energy threshold approach for STs to determine the energy allocation between spectrum sensing (SS) and packet transmission. Moreover, we address the AoI minimization problem by the proposed hybrid action and energy penalty deep deterministic policy gradient (HAEP-DDPG) algorithm, which adapts to the hybrid continuous and discrete action spaces. Simulation results validate the advantage of the proposed HAEP-DDPG algorithm compared with comparison schemes in terms of AoI.