Complex Action Sequences Research Articles

Introduction to latent variable energy-based models: a path toward autonomous machine intelligence

Abstract Current automated systems have crucial limitations that need to be addressed before artificial intelligence can reach human-like levels and bring new technological revolutions. Among others, our societies still lack level-5 self-driving cars, domestic robots, and virtual assistants that learn reliable world models, reason, and plan complex action sequences. In these notes, we summarize the main ideas behind the architecture of autonomous intelligence of the future proposed by Yann LeCun. In particular, we introduce energy-based and latent variable models and combine their advantages in the building block of LeCun’s proposal, that is, in the hierarchical joint-embedding predictive architecture.

Long sequence Hopfield memory*

Abstract Sequence memory is an essential attribute of natural and artificial intelligence that enables agents to encode, store, and retrieve complex sequences of stimuli and actions. Computational models of sequence memory have been proposed where recurrent Hopfield-like neural networks are trained with temporally asymmetric Hebbian rules. However, these networks suffer from limited sequence capacity (maximal length of the stored sequence) due to interference between the memories. Inspired by recent work on Dense Associative Memories, we expand the sequence capacity of these models by introducing a nonlinear interaction term, enhancing separation between the patterns. We derive novel scaling laws for sequence capacity with respect to network size, significantly outperforming existing scaling laws for models based on traditional Hopfield networks, and verify these theoretical results with numerical simulation. Moreover, we introduce a generalized pseudoinverse rule to recall sequences of highly correlated patterns. Finally, we extend this model to store sequences with variable timing between states’ transitions and describe a biologically-plausible implementation, with connections to motor neuroscience.

Updating predictions in a complex repertoire of actions and its neural representation

Even though actions we observe in everyday life seem to unfold in a continuous manner, they are automatically divided into meaningful chunks, that are single actions or segments, which provide information for the formation and updating of internal predictive models. Specifically, boundaries between actions constitute a hub for predictive processing since the prediction of the current action comes to an end and calls for updating of predictions for the next action. In the current study, we investigated neural processes which characterize such boundaries using a repertoire of complex action sequences with a predefined probabilistic structure. Action sequences consisted of actions that started with the hand touching an object (T) and ended with the hand releasing the object (U). These action boundaries were determined using an automatic computer vision algorithm. Participants trained all action sequences by imitating demo videos. Subsequently, they returned for an fMRI session during which the original action sequences were presented in addition to slightly modified versions thereof. Participants completed a post-fMRI memory test to assess the retention of original action sequences. The exchange of individual actions, and thus a violation of action prediction, resulted in increased activation of the action observation network and the anterior insula. At U events, marking the end of an action, increased brain activation in supplementary motor area, striatum, and lingual gyrus was indicative of the retrieval of the previously encoded action repertoire. As expected, brain activation at U events also reflected the predefined probabilistic branching structure of the action repertoire. At T events, marking the beginning of the next action, midline and hippocampal regions were recruited, reflecting the selected prediction of the unfolding action segment. In conclusion, our findings contribute to a better understanding of the various cerebral processes characterizing prediction during the observation of complex action repertoires.

A brain machine interface framework for exploring proactive control of smart environments

Brain machine interfaces (BMIs) can substantially improve the quality of life of elderly or disabled people. However, performing complex action sequences with a BMI system is onerous because it requires issuing commands sequentially. Fundamentally different from this, we have designed a BMI system that reads out mental planning activity and issues commands in a proactive manner. To demonstrate this, we recorded brain activity from freely-moving monkeys performing an instructed task and decoded it with an energy-efficient, small and mobile field-programmable gate array hardware decoder triggering real-time action execution on smart devices. Core of this is an adaptive decoding algorithm that can compensate for the day-by-day neuronal signal fluctuations with minimal re-calibration effort. We show that open-loop planning-ahead control is possible using signals from primary and pre-motor areas leading to significant time-gain in the execution of action sequences. This novel approach provides, thus, a stepping stone towards improved and more humane control of different smart environments with mobile brain machine interfaces.

Mutual information oriented deep skill chaining for multi‐agent reinforcement learning

AbstractMulti‐agent reinforcement learning relies on reward signals to guide the policy networks of individual agents. However, in high‐dimensional continuous spaces, the non‐stationary environment can provide outdated experiences that hinder convergence, resulting in ineffective training performance for multi‐agent systems. To tackle this issue, a novel reinforcement learning scheme, Mutual Information Oriented Deep Skill Chaining (MioDSC), is proposed that generates an optimised cooperative policy by incorporating intrinsic rewards based on mutual information to improve exploration efficiency. These rewards encourage agents to diversify their learning process by engaging in actions that increase the mutual information between their actions and the environment state. In addition, MioDSC can generate cooperative policies using the options framework, allowing agents to learn and reuse complex action sequences and accelerating the convergence speed of multi‐agent learning. MioDSC was evaluated in the multi‐agent particle environment and the StarCraft multi‐agent challenge at varying difficulty levels. The experimental results demonstrate that MioDSC outperforms state‐of‐the‐art methods and is robust across various multi‐agent system tasks with high stability.

Enhancing Robustness of Viewpoint Changes in 3D Skeleton-Based Human Action Recognition

Previous research on 3D skeleton-based human action recognition has frequently relied on a sequence-wise viewpoint normalization process, which adjusts the view directions of all segmented action sequences. This type of approach typically demonstrates robustness against variations in viewpoint found in short-term videos, a characteristic commonly encountered in public datasets. However, our preliminary investigation of complex action sequences, such as discussions or smoking, reveals its limitations in capturing the intricacies of such actions. To address these view-dependency issues, we propose a straightforward, yet effective, sequence-wise augmentation technique. This strategy enhances the robustness of action recognition models, particularly against changes in viewing direction that mainly occur within the horizontal plane (azimuth) by rotating human key points around either the z-axis or the spine vector, effectively creating variations in viewing directions. We scrutinize the robustness of this approach against real-world viewpoint variations through extensive empirical studies on multiple public datasets, including an additional set of custom action sequences. Despite the simplicity of our approach, our experimental results consistently yield improved action recognition accuracies. Compared to the sequence-wise viewpoint normalization method used with advanced deep learning models like Conv1D, LSTM, and Transformer, our approach showed a relative increase in accuracy of 34.42% for the z-axis and 10.86% for the spine vector.

Sex peptide regulates female receptivity through serotoninergic neurons in Drosophila

The courtship ritual is a dynamic interplay between males and females. Courtship successfully leading to copulation is determined by the intention of both parties which is conveyed by complex action sequences. In Drosophila, the neural mechanisms controlling the female's willingness to mate, or sexual receptivity, have only recently become the focus of investigations. Here, we report that pre-mating sexual receptivity in females requires activity within a subset of serotonergic projection neurons (SPNs), which positively regulate courtship success. Of interest, a male-derived sex peptide, SP, which was transferred to females during copulation acted to inhibit the activity of SPN and suppressed receptivity. Downstream of 5-HT, subsets of 5-HT7 receptor neurons played critical roles in SP-induced suppression of sexual receptivity. Together, our study reveals a complex serotonin signaling system in the central brain of Drosophila which manages the female's desire to mate.

Reinforcement Learning with Stochastic Reward Machines

Reward machines are an established tool for dealing with reinforcement learning problems in which rewards are sparse and depend on complex sequences of actions. However, existing algorithms for learning reward machines assume an overly idealized setting where rewards have to be free of noise. To overcome this practical limitation, we introduce a novel type of reward machines, called stochastic reward machines, and an algorithm for learning them. Our algorithm, based on constraint solving, learns minimal stochastic reward machines from the explorations of a reinforcement learning agent. This algorithm can easily be paired with existing reinforcement learning algorithms for reward machines and guarantees to converge to an optimal policy in the limit. We demonstrate the effectiveness of our algorithm in two case studies and show that it outperforms both existing methods and a naive approach for handling noisy reward functions.

Segmentation and Recognition Model for Complex Action Sequences

The researches on human behavior recognition have always on the trend, amongst which the sensor-based recognition is one of the most important fields.While the performance of individual action recognition methods can be described as satisfactory, however, it is difficult to obtain the location and category information from continuous and complex action sequences. In this paper we proposed a novel segmentation and recognition model for continuous action sequence. This model segments the sequence then categorize them into three types: dynamic actions, static actions and transition actions. By analyzing the characteristics of gait signals in dynamic actions, we proposed a method of biometric-based gait cycle detection to identify all the dynamic actions in the sequence. Then, based on the difference of the signal conversion frequency between static actions and transition actions, we analyzed the fluctuation point distribution in this two kinds of actions. The result shows that introducing the threshold detection algorithm can be of great benefits when distinguishing static actions from transition actions. This model was tested with public data set and the experiments have proved that the model achieves excellent results on the complex continuous action data set disclosed by UCI. We reached the accuracy over 98%, which showed our model is better than any known method.

Illuminating the prefrontal neural correlates of action sequence disassembling in response–response binding

Execution of two independent actions in quick succession results in transient binding of these two actions. Subsequent repetition of any of these actions automatically retrieves the other. This process is probably fundamental for developing complex action sequences. However, rigid bindings between two actions are not always adaptive. Sometimes, it is necessary to repeat only one of the two previously executed actions. In such situations, stored action sequences must be disassembled, for the sake of flexibility. Exact mechanisms that allow for such an active unbinding of actions remain largely unknown, but it stands to reason, that some form of prefrontal executive control is necessary. Building on prior neuronal research that explored other forms of binding (e.g. between distractors and responses and abstract representations and responses), we explored middle and superior frontal correlates of -response binding in a sequential classification task with functional near-infrared spectroscopy. We found that anterior dorsolateral prefrontal cortex activity varied as a function of response–repetition condition. Activity in the right anterior dorsolateral prefrontal cortex correlated with changes in reaction times due to response–response binding. Our results indicate that the right anterior dorsolateral prefrontal cortex dismantles bindings between consecutive actions, whenever such bindings interfere with current action goals.

DASZL: Dynamic Action Signatures for Zero-shot Learning

There are many realistic applications of activity recognition where the set of potential activity descriptions is combinatorially large. This makes end-to-end supervised training of a recognition system impractical as no training set is practically able to encompass the entire label set. In this paper, we present an approach to fine-grained recognition that models activities as compositions of dynamic action signatures. This compositional approach allows us to reframe fine-grained recognition as zero-shot activity recognition, where a detector is composed "on the fly" from simple first-principles state machines supported by deep-learned components. We evaluate our method on the Olympic Sports and UCF101 datasets, where our model establishes a new state of the art under multiple experimental paradigms. We also extend this method to form a unique framework for zero-shot joint segmentation and classification of activities in video and demonstrate the first results in zero- shot decoding of complex action sequences on a widely-used surgical dataset. Lastly, we show that we can use off-the-shelf object detectors to recognize activities in completely de-novo settings with no additional training.

Health Care Providers' Performance, Mindset, and Attitudes Toward a Neonatal Resuscitation Computer-Based Simulator: Empirical Study.

BackgroundNeonatal resuscitation involves a complex sequence of actions to establish an infant’s cardiorespiratory function at birth. Many of these responses, which identify the best action sequence in each situation, are taught as part of the recurrent Neonatal Resuscitation Program training, but they have a low incidence in practice, which leaves health care providers (HCPs) less prepared to respond appropriately and efficiently when they do occur. Computer-based simulators are increasingly used to complement traditional training in medical education, especially in the COVID-19 pandemic era of mass transition to digital education. However, it is not known how learners’ attitudes toward computer-based learning and assessment environments influence their performance.ObjectiveThis study explores the relation between HCPs’ attitudes toward a computer-based simulator and their performance in the computer-based simulator, RETAIN (REsuscitation TrAINing), to uncover the predictors of performance in computer-based simulation environments for neonatal resuscitation.MethodsParticipants were 50 neonatal HCPs (45 females, 4 males, 1 not reported; 16 respiratory therapists, 33 registered nurses and nurse practitioners, and 1 physician) affiliated with a large university hospital. Participants completed a demographic presurvey before playing the game and an attitudinal postsurvey after completing the RETAIN game. Participants’ survey responses were collected to measure attitudes toward the computer-based simulator, among other factors. Knowledge on neonatal resuscitation was assessed in each round of the game through increasingly difficult neonatal resuscitation scenarios. This study investigated the moderating role of mindset on the association between the perceived benefits of understanding the terminology used in the computer-based simulator, RETAIN, and their performance on the neonatal resuscitation tasks covered by RETAIN.ResultsThe results revealed that mindset moderated the relation between participants’ perceived terminology used in RETAIN and their actual performance in the game (F3,44=4.56, R2=0.24, adjusted R2=0.19; P=.007; estimate=–1.19, SE=0.38, t44=–3.12, 95% CI –1.96 to –0.42; P=.003). Specifically, participants who perceived the terminology useful also performed better but only when endorsing more of a growth mindset; they also performed worse when endorsing more of a fixed mindset. Most participants reported that they enjoyed playing the game. The more the HCPs agreed that the terminology in the tutorial and in the game was accessible, the better they performed in the game, but only when they reported endorsing a growth mindset exceeding the average mindset of all the participants (F3,44=6.31, R2=0.30, adjusted R2=0.25; P=.001; estimate=–1.21, SE=0.38, t44=−3.16, 95% CI –1.99 to –0.44; P=.003).ConclusionsMindset moderates the strength of the relationship between HCPs’ perception of the role that the terminology employed in a game simulator has on their performance and their actual performance in a computer-based simulator designed for neonatal resuscitation training. Implications of this research include the design and development of interactive learning environments that can support HCPs in performing better on neonatal resuscitation tasks.

Considerations for Comparing Video Game AI Agents with Humans

Video games are sometimes used as environments to evaluate AI agents’ ability to develop and execute complex action sequences to maximize a defined reward. However, humans cannot match the fine precision of the timed actions of AI agents; in games such as StarCraft, build orders take the place of chess opening gambits. However, unlike strategy games, such as chess and Go, video games also rely heavily on sensorimotor precision. If the “finding” was merely that AI agents have superhuman reaction times and precision, none would be surprised. The goal is rather to look at adaptive reasoning and strategies produced by AI agents that may replicate human approaches or even result in strategies not previously produced by humans. Here, I will provide: (1) an overview of observations where AI agents are perhaps not being fairly evaluated relative to humans, (2) a potential approach for making this comparison more appropriate, and (3) highlight some important recent advances in video game play provided by AI agents.

High-Level Motor Planning Assessment During Performance of Complex Action Sequences in Humans and a Humanoid Robot

Examining complex cognitive-motor performance in humanoid robots and humans can inform their interactions in a social context of team dynamics. Namely, the understanding of human cognitive-motor control and learning mechanisms can inform human motor behavior and also the development of intelligent controllers for robots when interacting with people. While prior humans and humanoid robot studies mainly examined motion planning, only a few have investigated high-level motor planning underlying action sequences for complex task execution. This sparse work has largely considered well-constrained problems using fairly simple performance assessment methods without detailed action sequence analyses. Here we qualitatively and quantitatively assess action sequences generated by humans and a humanoid robot during execution of two tasks providing various challenge levels and learning paradigms while offering flexible success criteria. The Levenshtein distance and its operators are adapted to the motor domain to provide a detailed performance assessment of action sequences by comparing them to a reference sequence (perfect sequence having a minimal number of actions). The results reveal that (i) humans produced a large variety of action sequences combining perfect and imperfect sequences while still reaching the task goal, whereas the robot generated perfect/near-perfect successful action sequences; (ii) the Levenshtein distance and the number of insertions provide reliable performance markers capable of differentiating perfect and imperfect sequences; (iii) the deletion operator is the most sensitive marker of action sequence failure. This work complements prior efforts for complex task performance assessment in humans and humanoid robots and has the potential to inform human–machine interactions.

A Novel Application of Levenshtein Distance for Assessment of High-Level Motor Planning Underlying Performance During Learning of Complex Motor Sequences

Few studies have examined high-level motor plans underlying cognitive-motor performance during practice of complex action sequences. These investigations have assessed performance through fairly simple metrics without examining how practice affects the structures of action sequences. By adapting the Levenshtein distance (LD) method to the motor domain, we propose a computational approach to accurately capture performance dynamics during practice of action sequences. Practice performance dynamics were assessed by computing the LD based on the number of insertions, deletions, and substitutions of actions needed to transform any sequence into a reference sequence (having a minimal number of actions to complete the task). Also, combining LD-based performance with mental workload metrics allowed assessment of cognitive-motor efficiency dynamics. This approach was tested on the Tower of Hanoi task. The findings revealed that throughout practice this method could capture: i) action sequence performance improvements as indexed by a reduced LD (decrease of insertions and substitutions), ii) structural modifications of the high-level plans, iii) an attenuation of mental workload, and iv) enhanced cognitive-motor efficiency. This effort complements prior work examining the practice of complex action sequences in healthy adults and has potential for probing cognitive-motor impairment in clinical populations as well as the development/assessment of cognitive robotic controllers.

A Mobile Phone App for the Generation and Characterization of Motor Habits.

Habits are a powerful route to efficiency; the ability to constantly shift between goal-directed and habitual strategies, as well as integrate them into behavioral output, is key to optimal performance in everyday life. When such ability is impaired, it may lead to loss of control and to compulsive behavior. Habits have successfully been induced and investigated in rats using methods such as overtraining stimulus-response associations and outcome devaluation, respectively. However, such methods have ineffectively measured habits in humans because (1) human habits usually involve more complex sequences of actions than in rats and (2) of pragmatic impediments posed by the extensive time (weeks or even months), it may take for routine habits to develop. We present here a novel behavioral paradigm—a mobile-phone app methodology—for inducing and measuring habits in humans during their everyday schedule and environment. It assumes that practice is key to achieve automaticity and proficiency and that the use of a hierarchical sequence of actions is the best strategy for capturing the cognitive mechanisms involved in habit formation (including “chunking”) and consolidation. The task is a gamified self-instructed and self-paced app on a mobile phone that enables subjects to learn and practice two sequences of finger movements, composed of chords and single presses. It involves a step-wise learning procedure in which subjects begin responding to a visual and auditory cued sequence by generating responses on the screen using four fingers. Such cues progressively disappear throughout 1 month of training, enabling the subject ultimately to master the motor skill involved. We present preliminary data for the acquisition of motor sequence learning in 29 healthy individuals, each trained over a month period. We demonstrate an asymptotic improvement in performance, as well as its automatic nature. We also report how people integrate the task into their daily routine, the development of motor precision throughout training, and the effect of intermittent reinforcement and reward extinction in habit preservation. The findings help to validate this “real world” app for measuring human habits.

Deficits in Planning Sequential Goal-Directed Action Impact Motor Execution in Children With Hemiplegic Cerebral Palsy: A Kinematic Analysis

There is mounting experimental evidence that suggests children with hemiplegic cerebral palsy (HCP) lack the ability to properly plan their motor actions. The purpose of this investigation was to determine how these planning deficits might affect the ability of children with HCP to execute a complex sequence of actions that have different final end states in the hand/arm position. Thirteen children with HCP (age = 6.8 ± 2.9 yrs, Manual Ability Classification levels = II–IV) and 15 typically developing (TD) children (age = 5.8 ± 1.1 yrs) completed a sequential motor task that involved reaching to grasp an object and place it into six-different target positions. Each target had a biomechanically different final hand/arm position for placing the object. A three-dimensional motion capture system was used to calculate the resultant spatiotemporal kinematics of a marker that was placed on the hand. In addition, video analysis was used to quantify the child’s reaction time, end-state comfort effect, and task failures. Compared with the TD children, the children with HCP had notable deviations in their spatiotemporal kinematics when reaching for the object and placing it in the respective target positions. The children with HCP also exhibited slower reaction time, reduced end-state comfort, and increased number of task failures when placing the objects in the respective target potions. Interestingly, the children with HCP showed impaired planning on the affected as well as the less affected arm. This suggests that the inability of children with HCP to properly plan a motor action might partly arise from central deficits in their cognitive processing, and not exclusively from the peripheral deficits seen in the musculoskeletal system. Altogether our results imply that children with HCP lack the ability to plan and execute sequential motor actions.

Decoding hierarchical control of sequential behavior in oscillatory EEG activity.

Despite strong theoretical reasons for assuming that abstract representations organize complex action sequences in terms of subplans (chunks) and sequential positions, we lack methods to directly track such content-independent, hierarchical representations in humans. We applied time-resolved, multivariate decoding analysis to the pattern of rhythmic EEG activity that was registered while participants planned and executed individual elements from pre-learned, structured sequences. Across three experiments, the theta and alpha-band activity coded basic elements and abstract control representations, in particular, the ordinal position of basic elements, but also the identity and position of chunks. Further, a robust representation of higher level, chunk identity information was only found in individuals with above-median working memory capacity, potentially providing a neural-level explanation for working-memory differences in sequential performance. Our results suggest that by decoding oscillatory activity we can track how the cognitive system traverses through the states of a hierarchical control structure.

Evolving Neural Arrays A new mechanism for learning complex action sequences


 
 
 Incremental evolution has proved to be an extremely useful mechanism in complex actions sequence learning. Its performance is based on the decomposition of the original problem into increasingly complex stages whose learning is carried out sequentially, starting from the simplest stage and thus increasing its generality and difficulty.
 The present work proposes neural array applications as a novel mechanism for complex actions sequence learning. Each array is composed by several neural nets obtained by means of an evolving process allowing them to acquire various degrees of specialization. Neural nets constituting the same array are organized so that, in each assessment, there is only one in charge of its response.
 The proposed strategy is applied to problems presented by obstacle evasion and target reaching as a means to show the capability of this proposal to solve complex problems. The measurements carried out show the superiority of evolving neural arrays over traditional neuroevolving methods that handle neural network populations – SANE is being particularly used as a comparative reference due to its high performance.
 Neural array capability to recover from previous defective evolving stages has been tested, evincing highly plausible final successful outcomes – even in those adverse cases.
 Finally, conclusions are presented as well as some future lines of work.
 
 

A Modular Dynamic Sensorimotor Model for Affordances Learning, Sequences Planning, and Tool-Use

This paper proposes a computational model for learning robot control and sequence planning based on the ideomotor principle. This model encodes covariation laws between sensors and motors in a modular fashion and exploits these primitive skills to build complex action sequences, potentially involving tool-use. Implemented for a robotic arm, the model starts with raw unlabeled sensor and motor vectors and autonomously assigns functions to neutral objects in the environment. Our experimental evaluation highlights the emergent properties of such a modular system and we discuss their consequences from ideomotor and sensorimotor-theoretic perspectives.