Action Prediction Task Research Articles

Contextual expectations shape the motor coding of movement kinematics during the prediction of observed actions: A TMS study

Contextual information may shape motor resonance and support intention understanding during observation of incomplete, ambiguous actions. It is unclear, however, whether this effect is contingent upon kinematics ambiguity or contextual information is continuously integrated with kinematics to predict the overarching action intention. Moreover, a differentiation between the motor mapping of the intention suggested by context or kinematics has not been clearly demonstrated. In a first action execution phase, 29 participants were asked to perform reaching-to-grasp movements towards big or small food objects with the intention to eat or to move; electromyography from the First Dorsal Interosseous (FDI) and Abductor Digiti Minimi (ADM) was recorded. Depending on object size, the intentions to eat or to move were differently implemented by a whole-hand or a precision grip kinematics, thus qualifying an action-muscle dissociation. Then, in a following action prediction task, the same participants were asked to observe an actor performing the same actions and to predict his/her intention while motor resonance was assessed for the same muscles. Of note, videos were interrupted at early or late action phases, and actions were embedded in contexts pointing toward an eating or a moving intention, congruently or incongruently with kinematics. We found greater involvement of the FDI or ADM in the execution of precision or whole-hand grips, respectively. Crucially, this pattern of activation was mirrored during observation of the same actions in congruent contexts, but it was cancelled out or reversed in the incongruent ones, either when videos were interrupted at either early or long phases of action deployment. Our results extend previous evidence by showing that contextual information shapes motor resonance not only under conditions of perceptual uncertainty but also when more informative kinematics is available.

Feasibility and Efficacy of a Virtual Reality Social Prediction Training in Children and Young Adults with Congenital Cerebellar Malformations.

This study tested the feasibility and efficacy of a Virtual Reality (VR) social prediction training (VR-Spirit) specifically designed for patients with congenital cerebellar malformation. The study is a randomised controlled trial in which 28 cerebellar patients aged 7-25 yo were randomly allocated to the VR-Spirit or to a control intervention in VR. The VR-Spirit required participants to compete with different avatars in scenarios that prompted them to form predictions about avatars' intentions. The control intervention consisted of games currently adopted for motor rehabilitation. Social prediction as well as secondary neuropsychological and behavioural outcomes were assessed at the beginning (T0), at the end (T2) and after 2months (T3). The experimental group showed a significant increase, compared to the control participants, in social prediction assessed through a VR task. Moreover, at least at T3, the VR-Spirit enhanced the use of contextual predictions in a computer-based action prediction task. Importantly, these effects were generalized to secondary neuropsychological outcomes, specifically theory of mind and, only at T2, inhibition. No differences between the interventions were detected on emotional-behavioural problems. Lastly, both interventions showed high feasibility and acceptability. These findings confirm that it is possible to develop condition-specific rehabilitative training on the basis of neurocognitive functions impaired in case of congenital malformation. The VR-Spirit demonstrated to generalize its effects to theory of mind abilities, and it might be thus extended to other neurodevelopmental disorders that present social perception deficits and alterations of predictive processing.Trial registration: ISRCTN, ID: ISRCTN22332873. Retrospectively registered on 12 March 2018.

Excitatory cerebellar transcranial direct current stimulation boosts the leverage of prior knowledge for predicting actions.

The cerebellum causally supports social processing by generating internal models of social events based on statistical learning of behavioral regularities. However, whether the cerebellum is only involved in forming or also in using internal models for the prediction of forthcoming actions is still unclear. We used cerebellar transcranial Direct Current Stimulation (ctDCS) to modulate the performance of healthy adults in using previously learned expectations in an action prediction task. In a first learning phase of this task, participants were exposed to different levels of associations between specific actions and contextual elements, to induce the formation of either strongly or moderately informative expectations. In a following testing phase, which assessed the use of these expectations for predicting ambiguous (i.e. temporally occluded) actions, we delivered ctDCS. Results showed that anodic, compared to sham, ctDCS boosted the prediction of actions embedded in moderately, but not strongly, informative contexts. Since ctDCS was delivered during the testing phase, that is after expectations were established, our findings suggest that the cerebellum is causally involved in using internal models (and not just in generating them). This encourages the exploration of the clinical effects of ctDCS to compensate poor use of predictive internal models for social perception.

Selective Action Prediction in Infancy Depending on Linguistic Cues: An EEG and Eyetracker Study.

Humans' capacity to predict actions and to socially categorize individuals is at the basis of social cognition. Such capacities emerge in early infancy. By 6 months of age, infants predict others' reaching actions considering others' epistemic state. At a similar age, infants are biased to attend to and interact with more familiar individuals, considering adult-like social categories such as the language people speak. We report that these two core processes are interrelated early on in infancy. In a belief-based action prediction task, 6-month-old infants (males and females) presented with a native speaker generated online predictions about the agent's actions, as revealed by the activation of participants' sensorimotor areas before the agent's movement. However, infants who were presented with a foreign speaker did not recruit their motor system before the agent's action. The eyetracker analysis provided further evidence that linguistic group familiarity influences how infants predict others' actions, as only infants presented with a native speaker modified their attention to the stimuli as a function of the agent's forthcoming behavior. The current findings suggest that infants' emerging capacity to predict others' actions is modulated by social cues, such as others' linguistic group. A facilitation to predict and encode the actions of native speakers relative to foreign speakers may explain, in part, why infants preferentially attend to, imitate, and learn from the actions of native speakers.

Asymmetric multi-task learning for interpretable gaze-driven grasping action forecasting.

This work tackles the problem of automatically predicting the grasping intention of humans observing their environment, with eye-tracker glasses and video cameras recording the scene view. Our target application is the assistance to people with motor disabilities and potential cognitive impairments, using assistive robotics. Our proposal leverages the analysis of human attention captured in the form of gaze fixations recorded by an eye-tracker on the first person video, as the anticipation of prehension actions is a well studied and well known phenomenon. We propose a multi-task system that simultaneously addresses the prediction of human attention in the near future, and the anticipation of grasping actions. In our model, visual attention is modeled as a competitive process between a discrete set of states, each one associated to a well-known gaze movement pattern from visual psychology. We additionally consider an asymmetric multitask problem, where attention modeling is an auxiliary task that helps to regularize the learning process of the main action prediction task, and propose a constrained multi-task loss that naturally deals with this asymmetry. Our model shows superior performance than other losses for dynamic multi-task learning, current dominant deep architectures for general action forecasting and particularly-tailored models for predicting grasping intention. In particular, it provides state-of-the-art performance in three datasets for egocentric action anticipation, with an average precision of 0.569 and 0.524 in GITW and Sharon datasets, respectively, and an accuracy of 89.2% and a success rate of 51.7% in Invisible dataset.

When action prediction grows old: An fMRI study.

Predicting the unfolding of others' actions (action prediction) is crucial for successfully navigating the social world and interacting efficiently. Age-related changes in this domain have remained largely unexplored, especially for predictions regarding simple gestures and independent of contextual information or motor expertise. Here, we evaluated whether healthy aging impacts the neurophysiological processes recruited to anticipate, from the observation of implied-motion postures, the correct conclusion of simple grasping and pointing actions. A color-discrimination task served as a control condition to assess the specificity of the age-related effects. Older adults showed reduced efficiency in performance that was yet not specific to the action prediction task. Nevertheless, fMRI results revealed task-specific age-related differences: while both groups showed stronger recruitment of the lateral occipito-temporal cortex bilaterally during the action prediction than the control task, the younger participants additionally showed a higher bilateral engagement of parietal regions. Importantly, in both groups, the recruitment of visuo-motor processes in the right posterior parietal cortex was a predictor of good performance. These results support the hypothesis of decreased involvement of sensorimotor processes in cognitive tasks when processing action- and body-related stimuli in healthy aging. These results have implications for social interaction, which requires the fast reading of others' gestures.

Multi-Head TrajectoryCNN: A New Multi-Task Framework for Action Prediction

Action prediction is an important task in human activity analysis, which has many practical applications, such as human–robot interactions and autonomous driving. Action prediction often comprises two subtasks: action semantic prediction and future human motion prediction. Most of the existing works treat these subtasks separately, ignoring the correlations, leading to unsatisfying performance. By contrast, we jointly model these tasks and improve human motion predictions utilizing their action semantics. In terms of methodology, we propose a novel multi-task framework (Multi-head TrajectoryCNN) to simultaneously predict the action semantics and human motion of future human movements. Specifically, we first extract a general spatiotemporal representation of partial observations via two regression blocks. Then, we propose a regression head and a classification head for predicting future human motion and action semantics of human motion, respectively. For the regression head, another two stacked regression blocks and two convolutional layers are applied to predict future poses from the general representation learning. For the classification head, we propose a classification block and stack two regression blocks to predict action semantics from the general representation. In this way, the regression and classification heads are incorporated into a unified framework. During the backward propagation of the network, the human motion prediction and the semantic prediction may be enhanced by each other. NTU RGB+D is a widely used large-scale dataset for action recognition, which was collected by 40 different subjects from three views. Based on the official protocols, we use the skeletal modality and process action sequences with fixed lengths for the evaluation of our action prediction task. Experiments on NTU RGB+D show our model’s state-of-the-art performance. Furthermore, the experimental results also show that semantic information is of great help in predicting future human motion.

Cerebellar Damage Affects Contextual Priors for Action Prediction in Patients with Childhood Brain Tumor.

Predictive coding accounts of action perception sustain that kinematics information is compared with contextual top-down predictions (i.e., priors) to understand actions in conditions of perceptual ambiguity. It has been previously shown that the cerebellum contributes to motor simulation of observed actions. Here, we tested the hypothesis that a specific contribution of the cerebellum to action perception is to provide contextual priors that guide the sampling of perceptual kinematic information. To this aim, we compared the performance of 42 patients with childhood brain tumor affecting infratentorial (ITT) or supratentorial (STT) areas with that of peers with typical development in an action prediction task. First, participants were exposed to videos depicting a child performing different reaching-to-grasp actions, which were associated with contextual cues in a probabilistic fashion. Then, they were presented with shortened versions of the same videos and asked to infer the action outcome; since kinematics was ambiguous, we expected their responses would be biased toward the previously learned contextual priors. We found that patients with brain tumor were impaired in predicting actions when compared to healthy controls. However, STT patients presented a reliable probabilistic effect, while ITT patients, who had cerebellar damage, did not rely on contextual priors in predicting actions. Furthermore, we found an association between the use of contextual priors and the ability to infer others' mental states as assessed by a standardized test. These results suggest that the cerebellum provides contextual priors to understand others' actions and this predictive function might underlie complex social cognition abilities.

Confidence-Guided Self Refinement for Action Prediction in Untrimmed Videos.

Many existing methods formulate the action prediction task as recognizing early parts of actions in trimmed videos. In this paper, we focus on predicting actions from ongoing untrimmed videos where actions might not happen at the very beginning of videos. It is extremely challenging to predict actions in such untrimmed videos due to ambiguous or even no information of actions in the early parts of videos. To address this problem, we propose a prediction confidence that assesses the decision quality of a prediction model. Guided by the confidence, the model continuously refines the prediction results by itself with the increasing observed video frames. Specifically, we build a Self Prediction Refining Network (SPR-Net) which incrementally learns the confidence for action prediction. SPR-Net consists of three modules: a temporal hybrid network, an incremental confidence learner, and a self-refining Gumbel softmax sampler. The temporal hybrid network generates the action category distributions by integrating static scene and dynamic motion information. The incremental confidence learner calculates the confidence in an incremental manner, judging the extent to which the temporal hybrid network should believe its prediction result. The self-refining Gumbel softmax sampler models the mutual relationship between the prediction confidence and the category distribution, which enables them to be jointly learned in an end-to-end fashion. We also present a sparse self-attention mechanism to encode local spatio-temporal features into the frame-level motion representation to further improve the prediction performance. Extensive experiments on five datasets (i.e., UT-Interaction, BIT-Interaction, UCF101, THUMOS14, and ActivityNet) validate the effectiveness of the proposed method.

Predicting user reactions to Twitter feed content based on personality type and social cues

The events in the past few years clearly indicate that the modern social, political and economical landscapes are heavily influenced by how information flows through social networks. For instance, the recent outcomes of the US presidential elections and the Brexit vote show that misinformation and otherwise influencing content can affect events of great importance. In this paper, we adopt a simplified version of the recently proposed Network Knowledge Base (NKB) model to tackle the problem of predicting basic actions that a user can take given the content of their social media feeds: either take action (by reusing content seen in their feeds or creating new one), or otherwise take no action. We propose processing raw data obtained from social media based on the framework defined by the NKB model, and then formulate an action/no action prediction task that takes as input five features (including the user’s personality type and other social cues), and then go on to show—via an extensive empirical evaluation with real-world Twitter data—that machine learning classification algorithms can be successfully applied in this setting to make predictions about user reactions. The main result obtained is that, out of the features considered, personality type based on the Big-5 (also known as OCEAN) model is the most impactful; furthermore, though the rest of the features taken individually do not have a significant impact, the best results are obtained when they are all taken together. This is a first step in applying the NKB model towards understanding the effect of pathogenic social media phenomena such as fake news, how they spread via cascades, and how to counteract their ill effects.

Adversarial Action Prediction Networks.

Different from after-the-fact action recognition, action prediction task requires action labels to be predicted from partially observed videos containing incomplete action executions. It is challenging because these partial videos have insufficient discriminative information, and their temporal structure is damaged. We study this problem in this paper, and propose an efficient and powerful deep network for learning representative and discriminative features for action prediction. Our approach exploits abundant sequential context information in full videos to enrich the feature representations of partial videos. This information is encoded in latent representations using a variational autoencoder (VAE), which are encouraged to be progress-invariant. Decoding such latent representations using another VAE, we can reconstruct missing information in the features extracted from partial videos. An adversarial learning scheme is adopted to differentiate the reconstructed features from the features directly extracted from full videos in order to well align their distributions. A multi-class classifier is also used to encourage the features to be discriminative. Our network jointly learns features and classifiers, and generates the features particularly optimized for action prediction. Extensive experimental results on UCF101, Sports-1M and BIT datasets demonstrate that our approach remarkably outperforms state-of-the-art methods, and shows significant speedup over these methods. Results also show that actions differ in their prediction characteristics; some actions can be correctly predicted even though only the beginning 10% portion of videos is observed.

Boosting and Decreasing Action Prediction Abilities Through Excitatory and Inhibitory tDCS of Inferior Frontal Cortex.

Influential theories suggest that humans predict others' upcoming actions by using their own motor system as an internal forward model. However, evidence that the motor system is causally essential for predicting others' actions is meager. Using transcranial direct current stimulation (tDCS), we tested the role of the inferior frontal cortex (IFC), in action prediction (AP). We devised a novel AP task where participants observed the initial phases of right-hand reaching-to-grasp actions and had to predict their outcome (i.e., the goal/object to be grasped). We found that suppression by cathodal (inhibitory) tDCS of the left IFC, but not the left superior temporal sulcus or the right IFC, selectively impaired performance on the AP task, but not on a difficulty-matched control task. Remarkably, anodal (excitatory) tDCS of the left IFC brought about a selective improvement in the AP task. These findings indicate that the left IFC is necessary for predicting the outcomes of observed human right-hand actions. Crucially, our study shows for the first time that down- and up-regulating excitability within the motor system can hinder and enhance AP abilities, respectively. These findings support predictive coding theories of action perception and have implications for enhancement of AP abilities.

Changing ideas about others' intentions: updating prior expectations tunes activity in the human motor system.

Predicting intentions from observing another agent’s behaviours is often thought to depend on motor resonance – i.e., the motor system’s response to a perceived movement by the activation of its stored motor counterpart, but observers might also rely on prior expectations, especially when actions take place in perceptually uncertain situations. Here we assessed motor resonance during an action prediction task using transcranial magnetic stimulation to probe corticospinal excitability (CSE) and report that experimentally-induced updates in observers’ prior expectations modulate CSE when predictions are made under situations of perceptual uncertainty. We show that prior expectations are updated on the basis of both biomechanical and probabilistic prior information and that the magnitude of the CSE modulation observed across participants is explained by the magnitude of change in their prior expectations. These findings provide the first evidence that when observers predict others’ intentions, motor resonance mechanisms adapt to changes in their prior expectations. We propose that this adaptive adjustment might reflect a regulatory control mechanism that shares some similarities with that observed during action selection. Such a mechanism could help arbitrate the competition between biomechanical and probabilistic prior information when appropriate for prediction.

The left inferior parietal lobe represents stored hand-postures for object use and action prediction.

Action semantics enables us to plan actions with objects and to predict others' object-directed actions as well. Previous studies have suggested that action semantics are represented in a fronto-parietal action network that has also been implicated to play a role in action observation. In the present fMRI study it was investigated how activity within this network changes as a function of the predictability of an action involving multiple objects and requiring the use of action semantics. Participants performed an action prediction task in which they were required to anticipate the use of a centrally presented object that could be moved to an associated target object (e.g., hammer—nail). The availability of actor information (i.e., presenting a hand grasping the central object) and the number of possible target objects (i.e., 0, 1, or 2 target objects) were independently manipulated, resulting in different levels of predictability. It was found that making an action prediction based on actor information resulted in an increased activation in the extrastriate body area (EBA) and the fronto-parietal action observation network (AON). Predicting actions involving a target object resulted in increased activation in the bilateral IPL and frontal motor areas. Within the AON, activity in the left inferior parietal lobe (IPL) and the left premotor cortex (PMC) increased as a function of the level of action predictability. Together these findings suggest that the left IPL represents stored hand-postures that can be used for planning object-directed actions and for predicting other's actions as well.

Dynamic Simulation and Static Matching for Action Prediction: Evidence From Body Part Priming

Accurately predicting other people's actions may involve two processes: internal real-time simulation (dynamic updating) and matching recently perceived action images (static matching). Using a priming of body parts, this study aimed to differentiate the two processes. Specifically, participants played a motion-controlled video game with either their arms or legs. They then observed arm movements of a point-light actor, which were briefly occluded from view, followed by a static test pose. Participants judged whether this test pose depicted a coherent continuation of the previously seen action (i.e., "action prediction task"). Evidence of dynamic updating was obtained after compatible effector priming (i.e., arms), whereas incompatible effector priming (i.e., legs) indicated static matching. Together, the results support action prediction as engaging two distinct processes, dynamic simulation and static matching, and indicate that their relative contributions depend on contextual factors like compatibility of body parts involved in performed and observed action.

Tracking speakers’ false beliefs: is theory of mind available earlier for word learning?

Happé and Loth (2002) describe word learning as a 'privileged domain' in the development of a theory of mind. We test this claim in a series of experiments based on the Sally-Anne paradigm. Three- and 4-year-old children's ability to represent others' false beliefs was investigated in tasks that required the child either to predict the actions of a protagonist in a story or to learn the meaning of a new word used by the protagonist. Experiment 1 replicated previous findings of better performance in a false belief word-learning task compared to a false belief action-prediction task. However, systematic manipulation of the task parameters in Experiments 2 and 3 revealed that this performance discrepancy disappeared when tasks were equated in their 'referential pull' (Perner, Rendl & Garnham, 2007). We conclude that the notion of a precocious theory of mind for word learning is not required to explain dissociations in performance on false belief tasks.