Learning Motion Patterns Research Articles

Gait Symmetric Adaptation and Aftereffect Through Concurrent Split-Belt Treadmill Walking and Explicit Visual Feedback Distortion.

Gait asymmetry may be improved through various gait training methods. Combining split-belt treadmill walking (SB) with visual feedback distortion (VD) could enhance motor learning, thereby improving gait symmetry adaptation and retention. This study compared step length symmetry adaptation and aftereffects between SB-only and the combined explicit VD with SB, as well as between explicit VD-only and the combined explicit VD with SB. Trials consisted of a 28-minute walking with three phases: a 3-minute baseline, a 10-minute adaptation, and a 15-minute post-adaptation. In the VD trial, two bars representing the right and left step lengths were displayed. The length of the right bar gradually decreased by 3% during the adaptation period, prompting participants to consciously correct their steps to match the heights of the two bars. In the SB trial, the speed of the right treadmill belt was incrementally increased by 5%. The VD+SB trial combined both perturbations. After the removal of these perturbations, the aftereffect of the adapted asymmetric step length was evaluated in the post-adaptation period. During the adaptation period, the step length symmetry ratio shifted negatively in the SB trial, while it increased positively in the VD trial, indicating longer right steps than left. In the VD+SB trial, subjects extended their right step more than their left. Notably, the VD+SB trial demonstrated a longer aftereffect compared to both the SB-only and VD-only trials. The visual distortion paradigm can be explicitly applied and integrated with split-belt treadmill walking to enhance the efficacy of symmetric gait adaptation, resulting in more sustained effects on the retention of newly learned motor patterns.

Designing and Analyzing In-Place Motor Tasks in Virtual Reality With Goal Functions.

Goal functions make virtual goal-oriented motor tasks easier to analyze and manipulate by explicitly linking movement to outcome. However, they have only been used to study constrained (e.g., planar) upper limb movements. We present a design framework for integrating goal functions with unconstrained postural and upper limb movements in a virtual reality (VR) device. VR tasks designed with the framework can mimic unconstrained natural motions and thus train a range of functional movements yet remain analytically tractable. We created three in-place VR motor tasks: a bow-and-arrow, a reach-and-strike, and a punching bag task. Each task was adjusted to subject-specific workspace limits and anthropometrics. We studied the effects of 3 days of practice and 3 reach/lean distances on task performance in 12 healthy adults. Subjects performed all tasks on day 1 with moderate proficiency and improved with practice at all reach/lean distances. Task-specific results showed that performance decreased and movement variability increased near the edge of the reaching workspace; viewing angles and the imperfect depth cues in VR likely led to biases in performance and practice could attenuate the former effect; in reach-and-strike, subjects learned movement patterns similar to those seen in a real-world striking sport. These results show that our framework can deliver tasks useful for analyzing and training motor performance and can guide future in-place motor training. Post-hoc, we demonstrated the feasibility of generalizable methods that adjust required movement speeds and task difficulty for impaired populations.

A Deep Learning Model with a Self-Attention Mechanism for Leg Joint Angle Estimation across Varied Locomotion Modes.

Conventional trajectory planning for lower limb assistive devices usually relies on a finite-state strategy, which pre-defines fixed trajectory types for specific gait events and activities. The advancement of deep learning enables walking assistive devices to better adapt to varied terrains for diverse users by learning movement patterns from gait data. Using a self-attention mechanism, a temporal deep learning model is developed in this study to continuously generate lower limb joint angle trajectories for an ankle and knee across various activities. Additional analyses, including using Fast Fourier Transform and paired t-tests, are conducted to demonstrate the benefits of the proposed attention model architecture over the existing methods. Transfer learning has also been performed to prove the importance of data diversity. Under a 10-fold leave-one-out testing scheme, the observed attention model errors are 11.50% (±2.37%) and 9.31% (±1.56%) NRMSE for ankle and knee angle estimation, respectively, which are small in comparison to other studies. Statistical analysis using the paired t-test reveals that the proposed attention model appears superior to the baseline model in terms of reduced prediction error. The attention model also produces smoother outputs, which is crucial for safety and comfort. Transfer learning has been shown to effectively reduce model errors and noise, showing the importance of including diverse datasets. The suggested joint angle trajectory generator has the potential to seamlessly switch between different locomotion tasks, thereby mitigating the problem of detecting activity transitions encountered by the traditional finite-state strategy. This data-driven trajectory generation method can also reduce the burden on personalization, as traditional devices rely on prosthetists to experimentally tune many parameters for individuals with diverse gait patterns.

Stochastic video normality network for abnormal event detection in surveillance videos

Video Anomaly Detection (VAD) aims to automatically identify unexpected spatial–temporal patterns to detect abnormal events in surveillance videos. Existing unsupervised VAD methods either use a single proxy task to represent the prototypical normal patterns, or split the video into static and dynamic parts and learn spatial and temporal normality separately, which cannot model the inherent uncertainty of motion. In this regard, we propose for the first time to learn video normality via stochastic motion representation. Specifically, the proposed Stochastic Video Normality (SVN) network learns the prototypical local appearance patterns via deterministic multi-task learning and global motion patterns in a non-deterministic manner. The stochastic representation module uses recurrent networks to model historical motion as a conditional Gaussian distribution and infers motion explicitly with the learned prior. Besides, a mask auto-encoder is introduced to explore the relationship between appearance and motion, boosting the model to learn spatial–temporal normality. Experimental results show that our proposed SVN network performs comparably to the state-of-the-art methods. Extensive analysis demonstrates the effectiveness of multi-task learning-based appearance representation and stochastic motion representation for unsupervised video anomaly detection.

Cognitive load causes kinematic changes in both elite and non-elite rowers

The current motor literature suggests that extraneous cognitive load may affect performance and kinematics in a primary motor task. A common response to increased cognitive demand, as observed in past studies, might be to reduce movement complexity and revert to previously learned movement patterns, in line with the progression-regression hypothesis. However, according to several accounts of automaticity, motor experts should be able to cope with dual task demands without detriment to their performance and kinematics. To test this, we conducted an experiment asking elite and non-elite rowers to use a rowing ergometer under conditions of varying task load. We employed single-task conditions with low cognitive load (i.e., rowing only) and dual-task conditions with high cognitive load (i.e., rowing and solving arithmetic problems). The results of the cognitive load manipulations were mostly in line with our hypotheses. Overall, participants reduced movement complexity, for example by reverting towards tighter coupling of kinematic events, in their dual-task performance as compared to single-task performance. The between-group kinematic differences were less clear. In contradiction to our hypotheses, we found no significant interaction between skill level and cognitive load, suggesting that the rowers' kinematics were affected by cognitive load irrespective of skill level. Overall, our findings contradict several past findings and automaticity theories, and suggest that attentional resources are required for optimal sports performance.

Long-Short Term Spatio-Temporal Aggregation for Trajectory Prediction

Pedestrian trajectory prediction in crowd scenes plays a significant role in intelligent transportation systems. The main challenges are manifested in learning motion patterns and addressing future uncertainty. Typically, trajectory prediction is considered in two dimensions, including temporal dynamics modeling and social interactions capturing. For temporal dependencies, although existing models based on recurrent neural networks (RNNs) or convolutional neural networks (CNNs) achieve high performance on short-term prediction, they still suffer from limited scalability for long sequences. For social interactions, previous graph-based methods only consider fixed features but ignore dynamic interactions between pedestrians. Considering that the transformer network has a strong capability of capturing spatial and long-term temporal dynamics, we propose Long-Short Term Spatio-Temporal Aggregation (LSSTA) network for human trajectory prediction. First, a modern variant of graph neural networks, named spatial encoder, is presented to characterize spatial interactions between pedestrians. Second, LSSTA utilizes a transformer network to handle long-term temporal dependencies and aggregates the spatial and temporal features with a temporal convolution network (TCN). Thus, TCN is combined with the transformer to form a long-short term temporal dependency encoder. Additionally, multi-modal prediction is an efficient way to address future uncertainty. Existing auto-encoder modules are extended with static scene information and future ground truth for multi-modal trajectory prediction. Experimental results on complex scenes demonstrate the superior performance of our method in comparison to existing approaches.

A Probabilistic Dynamic Movement Primitives Framework on Human Hand Motion Prediction for an Object Transfer Scenario

Recent advancements in Human-Robot Collaboration (HRC) have opened up promising prospects for revolutionizing the current manufacturing automation. Accurate modeling of human motion patterns is crucial for enabling the robot to understand human intention and predict their future motion based on online observations. As the widely used deterministic methods often lack confidence information about the provided result to account for the possible variability of human motion, in this work, a Probabilistic Dynamic Movement Primitives (PDMP)-based framework is adopted to recognize goal-directed human movements in the reaching phase and making online predictions. The proposed framework employs PDMP based on off-line demonstrations of relevant hand movements. To avoid frame-dependency, a novel DMP formulation with rotation and magnitude scaling is used, allowing for generalization of learned motion patterns to similar tasks. The proposed framework has been validated in experiments regarding an object transfer scenario on the workbench, using a Intel RealSense camera and OpenPose system for motion capturing. Results show that this framework can offer good performance in hand motion prediction in presence of human motion variations, and can be generalized to relevant tasks beyond the limited demonstrated trajectories.

Dark‐Mode Human–Machine Communication Realized by Persistent Luminescence and Deep Learning

Increasing ubiquitous collaborative intelligence between humans and machines requires human–machine communication (HMC) that is more human and less machine‐like to accomplish given tasks. Although speech signals are considered the best modes of communication in HMC, background noise often interferes with these signals. Therefore, research focused on integrating lip‐reading technology into HMC has gained significant attention. However, lip‐reading functions effectively only in well‐lit environments. In contrast, HMC may occur daily in dark environments owing to potential energy shortages, increased exploration in darkness, nighttime emergencies, etc. Herein, a possible method for HMC in the dark mode is presented, which is realized based on deep learning motion patterns of persistent luminescence (PL) of the skin surrounding the lips. An ultrasoft PL–polymer composite patch is used to record the motion pattern of the skin during speech in the dark. It is found that visual geometric group network (VGGNET‐5) and residual neural network (ResNet‐34) could predict spoken words in darkness with test accuracies of 98.5% and 98.75%, respectively. Furthermore, these models could effectively distinguish similar‐sounding words such as “around” and “ground.” Dark‐mode communication can allow a wide range of people, including disabled people with limited dexterity and voice tremors, to communicate with artificial intelligence machines.

Enhanced Medial Prefrontal Cortex and Hippocampal Activity Improves Memory Generalization in APP/PS1 Mice: A Multimodal Animal MRI Study.

Memory generalization allows individuals to extend previously learned movement patterns to similar environments, contributing to cognitive flexibility. In Alzheimer’s disease (AD), the disturbance of generalization is responsible for the deficits of episodic memory, causing patients with AD to forget or misplace things, even lose track of the way home. Cognitive training can effectively improve the cognition of patients with AD through changing thinking mode and memory flexibility. In this study, a T-shaped maze was utilized to simulate cognitive training in APP/PS1 mice to elucidate the potential mechanisms of beneficial effects after cognitive training. We found that cognitive training conducted by a T-shaped maze for 4 weeks can improve the memory generalization ability of APP/PS1 mice. The results of functional magnetic resonance imaging (fMRI) showed that the functional activity of the medial prefrontal cortex (mPFC) and hippocampus was enhanced after cognitive training, and the results of magnetic resonance spectroscopy (MRS) showed that the neurochemical metabolism of N-acetyl aspartate (NAA) and glutamic acid (Glu) in mPFC, hippocampus and reuniens (Re) thalamic nucleus were escalated. Furthermore, the functional activity of mPFC and hippocampus was negatively correlated with the escape latency in memory generalization test. Therefore, these results suggested that cognitive training might improve memory generalization through enhancing the functional activity of mPFC and hippocampus and increasing the metabolism of NAA and Glu in the brain regions of mPFC, hippocampus and Re nucleus.

Vision-Based Traffic Conflict Detection Using Trajectory Learning and Prediction

Although traffic conflict techniques have proven to be effective means for road safety analysis, they still suffer from incomplete conceptualization, observer subjectivity, and high data collection cost. To address these problems, video analysis has been increasingly applied to gain a better understanding of the behaviors of road users based on detailed motion data. However, the motion patterns underlying these data are rarely extracted to study the safety of their interactions. This article presents a vision-based method of traffic conflict detection through learning motion patterns from trajectories, for which an original algorithm was established through clustering and subsequent modeling. Using the extracted path and velocity information, we clustered trajectories hierarchically by applying an improved fuzzy $K$ -means algorithm with a modified Hausdorff distance. Each obtained cluster was taken as a labeled set to determine the structure and train the parameters of a hidden Markov model (HMM) that encoded the spatiotemporal characteristics of the trajectories as motion patterns. Based on the targeted trajectory predictions by the learned HMMs following the conflict development, a probabilistic model was developed to estimate the collision likelihood between vehicles to identify traffic conflicts. The experimental results obtained using actual traffic videos demonstrated the applicability of the algorithms for learning motion patterns and the feasibility of the approach for traffic conflict detection. The predicted trajectories were sufficiently accurate to calculate the collision probability, which was qualified as an indicator for measuring the conflict severity. These findings will have important implications for effective improvements in active road safety.

Generalizing movement patterns following shoulder fixation.

Generalizing newly learned movement patterns beyond the training context is challenging for most motor learning situations. Here we tested whether learning of a new physical property of the arm during self-initiated reaching generalizes to new arm configurations. Human participants performed a single-joint elbow reaching task and/or countered mechanical perturbations that created pure elbow motion with the shoulder joint free to rotate or locked by the manipulandum. With the shoulder free, we found activation of shoulder extensor muscles for pure elbow extension trials, appropriate for countering torques that arise at the shoulder due to forearm rotation. After locking the shoulder joint, we found a partial reduction in shoulder muscle activity, appropriate because locking the shoulder joint cancels the torques that arise at the shoulder due to forearm rotation. In our first three experiments, we tested whether and to what extent this partial reduction in shoulder muscle activity generalizes when reaching in different situations: 1) different initial shoulder orientation, 2) different initial elbow orientation, and 3) different reach distance/speed. We found generalization for the different shoulder orientation and reach distance/speed as measured by a reliable reduction in shoulder activity in these situations but no generalization for the different elbow orientation. In our fourth experiment, we found that generalization is also transferred to feedback control by applying mechanical perturbations and observing reflex responses in a distinct shoulder orientation. These results indicate that partial learning of new intersegmental dynamics is not sufficient for modifying a general internal model of arm dynamics.NEW & NOTEWORTHY Here we show that partially learning to reduce shoulder muscle activity following shoulder fixation generalizes to other movement conditions, but it does not generalize globally. These findings suggest that the partial learning of new intersegmental dynamics is not sufficient for modifying a general internal model of the arm's dynamics.

Efficacy of dance-based paradigms, wearable sensors, and auditory feedback for gait retraining in children: A feasibility study

BackgroundDifferent feedback modes such as auditory, visual and haptic have been used in the past for gait retraining or learning movement patterns. The primary goal of this study was to investigate whether real time auditory feedback would be effective in children learning novel, dance-based movement patterns. For this purpose, a prototype wearable sensor was developed to provide auditory feedback whenever a child touches their heel to the ground. MethodsTo test the effectiveness of the auditory feedback in learning new patterns, typically developing children were taught simple Indian dance protocols consisting of four counts of foot-work which involved alternating heel-toe movements. The effect of wearing the sensor was assessed by the maximum vertical force with which the subjects struck their foot on the plate. ResultsAuditory feedback reduced the learning time and increased the number of correct movement patterns for trial duration of 2 min. The prototype device did not alter the maximum force with which the subject placed the foot on the ground. ConclusionsReal time auditory feedback can be reliably used to learn novel movement patterns.

Change Detection by Training a Triplet Network for Motion Feature Extraction

Change/motion detection is a challenging problem in video analysis and surveillance system. Recently, the state-of-the-art methods using the sample-based background model have demonstrated astonishing results with this problem. However, they are ineffective in the dynamic scenes that contain complex motion patterns. In this paper, we introduce a novel data-driven approach that combines the sample-based background model with a feature extractor obtained by training a triplet network. We construct the network by three identical convolutional neural networks, each of which is called a motion feature network. Our network can automatically learn motion patterns from small image patches and transform input images of any size into feature embeddings for high-level representations. The sample-based background model of each pixel is then employed by using the color information and the extracted feature embeddings. We also propose an approach to generate triplet examples from CDNet 2014 for training our network model from scratch. The offline trained network can be used on the fly without re-training on any video sequence before each execution. Therefore, it is feasible for real-time surveillance systems. In this paper, we show that our method outperforms the other state-of-the-art methods on CDNet 2014 and other benchmarks (BMC and Wallflower).

Enhancing speech adaptation to a palatal perturbation using ultrasound visual biofeedback

Prior clinical studies suggest that visual biofeedback of the tongue (e.g., ultrasound) may enhance the treatment of speech disorders, but outcomes have been mixed, likely due to variability in both the clinical profiles of participants and the way in which treatments have been carried out. Understanding the true potential of this clinical tool requires a clearer sense of how visual biofeedback interacts with speech motor control. Here, we present a novel experimental approach that mimics the conditions of clinical treatment for speech disorders while maintaining a high level of consistency and control over the speech-learning task. The procedure involves altering the vocal tract of typically developing talkers using a palatal prosthesis to perturb /s/-production in combination with the controlled application of ultrasound biofeedback. As participants practice and improve their speech, changes in articulatory movements are examined using acoustic and kinematic measures. The present study compared a control group (n = 10) receiving only auditory feedback during speech practice with a group receiving visual biofeedback (n = 10). Results indicate an effect of biofeedback, in particular in the retention of learned motor patterns, indicating that talkers integrate real-time visual feedback of tongue movement into the sensorimotor processes driving speech adaptation.Prior clinical studies suggest that visual biofeedback of the tongue (e.g., ultrasound) may enhance the treatment of speech disorders, but outcomes have been mixed, likely due to variability in both the clinical profiles of participants and the way in which treatments have been carried out. Understanding the true potential of this clinical tool requires a clearer sense of how visual biofeedback interacts with speech motor control. Here, we present a novel experimental approach that mimics the conditions of clinical treatment for speech disorders while maintaining a high level of consistency and control over the speech-learning task. The procedure involves altering the vocal tract of typically developing talkers using a palatal prosthesis to perturb /s/-production in combination with the controlled application of ultrasound biofeedback. As participants practice and improve their speech, changes in articulatory movements are examined using acoustic and kinematic measures. The present study compared a contro...

Multi-object tracking through learning relational appearance features and motion patterns

Absract Multi-object tracking (MOT) is to simultaneously track multiple targets, e.g., pedestrians in this work, through locating them and maintaining their identities to make their individual trajectories. Despite of recent advances in object detection, MOT based on the tracking-by-detection principle is a still yet challenging and difficult task in complex and crowded conditions. For example, due to occlusion, missed object detection, and frequent entering and leaving of object in a scene, tracking failures such as identity switches and trajectory fragmentation can often occur. To tackle the issues, a new data association approach, namely, the relational appearance features and motion patterns learning (RAFMPL)-based data association, is proposed for facilitating MOT. In RAFMPL-MOT, the proposed relational features-based appearance model is different from conventional approaches in that it generates tracklets based on relational information by selecting one reference object and utilizing the feature differences between the reference object and the other objects. In addition, the motion patterns learning-based motion model enables linear and nonlinear confident motions patterns to be considered in data association. The proposed approach can effectively cover the key difficulties of MOT. In particular, using RAFMPL-MOT, it is possible to assign the same ID for the object that has been disappeared (even for moderately long period) and then is reappeared in the scene more robustly. Further, it also improves its robustness for occlusion problems frequently occurring in real situations. The experimental results show that the RAFMPL-MOT could generally achieve outperformance compared to the existing competitive MOT approaches.

Synchronisation through learning for two self-propelled swimmers

The coordinated motion by multiple swimmers is a fundamental component in fish schooling. The flow field induced by the motion of each self-propelled swimmer implies non-linear hydrodynamic interactions among the members of a group. How do swimmers compensate for such hydrodynamic interactions in coordinated patterns? We provide an answer to this riddle though simulations of two, self-propelled, fish-like bodies that employ a learning algorithm to synchronise their swimming patterns. We distinguish between learned motion patterns and the commonly used a-priori specified movements, that are imposed on the swimmers without feedback from their hydrodynamic interactions. First, we demonstrate that two rigid bodies executing pre-specified motions, with an alternating leader and follower, can result in substantial drag-reduction and intermittent thrust generation. In turn, we study two self-propelled swimmers arranged in a leader-follower configuration, with a-priori specified body-deformations. These two self-propelled swimmers do not sustain their tandem configuration. The follower experiences either an increase or decrease in swimming speed, depending on the initial conditions, while the swimming of the leader remains largely unaffected. This indicates that a-priori specified patterns are not sufficient to sustain synchronised swimming. We then examine a tandem of swimmers where the leader has a steady gait and the follower learns to synchronize its motion, to overcome the forces induced by the leader’s vortex wake. The follower employs reinforcement learning to adapt its swimming-kinematics so as to minimize its lateral deviations from the leader’s path. Swimming in such a sustained synchronised tandem yields up to reduction in energy expenditure for the follower, in addition to a increase in its swimming-efficiency. The present results show that two self-propelled swimmers can be synchronised by adapting their motion patterns to compensate for flow-structure interactions. Moreover, swimmers can exploit the vortical structures of their flow field so that synchronised swimming is energetically beneficial.

IMPLICIT AND EXPLICIT VIDEO FEEDBACK ON LANDING MECHANICS: A RANDOMIZED CONTROLLED TRIAL

BackgroundLower extremity(LE) patterns have been altered using implicit(IF) and explicit(EF) feedback motor learning strategies, however, it remains unclear which strategy may be more effective for injury prevention.ObjectiveTo examine the effects...

Idiosyncratic body motion influences person recognition

ABSTRACTPerson recognition is an important human ability. The main source of information we use to recognize people is the face. However, there is a variety of other information that contributes to person recognition, and the face is almost exclusively perceived in the presence of a moving body. Here, we used recent motion capture and computer animation techniques to quantitatively explore the impact of body motion on person recognition. Participants were familiarized with two animated avatars each performing the same basic sequence of karate actions with slight idiosyncratic differences in the body movements. The body of both avatars was the same, but they differed in their facial identity and body movements. In a subsequent recognition task, participants saw avatars whose facial identity consisted of morphs between the learned individuals. Across trials, each avatar was seen animated with sequences taken from both of the learned movement patterns. Participants were asked to judge the identity of the avatars. The avatars that contained the two original heads were predominantly identified by their facial identity regardless of body motion. More importantly however, participants identified the ambiguous avatar primarily based on its body motion. This clearly shows that body motion can affect the perception of identity. Our results also highlight the importance of taking into account the face in the context of a body rather than solely concentrating on facial information for person recognition.

A system for efficient motor learning using multimodal augmented feedback

Numerous studies have established that using various forms of augmented feedback improves human motor learning. In this paper, we present a system that enables real-time analysis of motion patterns and provides users with objective information on their performance of an executed set of motions. This information can be used to identify individual segments of improper motion early in the learning process, thus preventing improperly learned motion patterns that can be difficult to correct once fully learned. The primary purpose of the proposed system is to serve as a general tool in the research on impact of different feedback modalities on the process of motor learning, for example, in sports or rehabilitation. The key advantages of the system are high-speed and high-accuracy tracking, as well as its flexibility, as it supports various types of feedback (auditory and visual, concurrent or terminal). The practical application of the proposed system is demonstrated through the example of learning a golf swing.

Learning movement patterns of the occupant in smart home environments: an unsupervised learning approach

In recent years, the concept of smart homes and smart buildings have gradually emerged as mainstream with the development of sensor technology and the introduction of several commercial solutions. As one of the most important aspect of this technical-driven smart home system, learning and recognizing the occupant’s routine activities and movement patterns is the foundation of further human behavior understanding and prediction process. In this paper, we propose a decomposition based unsupervised learning approach, which is able to learn the occupant’s moving patterns directly from the sensor data without event annotations. It overcomes the limitation suffered by a lot of other supervised learning approaches that sensor data sets with event annotations are not difficult to collect since manually data labeling is not scalable and extremely time consuming. By applying the proposed method on Aruba motion sensor data set, we show that the decomposition based pattern learning method leverages the well-developed Non-negative Matrix Factorization algorithm that it can learn the movement pattern candidates of the occupant very fast, with outputs that can be easily interpreted.