Locomotor Tasks Research Articles

Multimodal information bottleneck for deep reinforcement learning with multiple sensors

Reinforcement learning has achieved promising results on robotic control tasks but struggles to leverage information effectively from multiple sensory modalities that differ in many characteristics. Recent works construct auxiliary losses based on reconstruction or mutual information to extract joint representations from multiple sensory inputs to improve the sample efficiency and performance of reinforcement learning algorithms. However, the representations learned by these methods could capture information irrelevant to learning a policy and may degrade the performance. We argue that compressing information in the learned joint representations about raw multimodal observations is helpful, and propose a multimodal information bottleneck model to learn task-relevant joint representations from egocentric images and proprioception. Our model compresses and retains the predictive information in multimodal observations for learning a compressed joint representation, which fuses complementary information from visual and proprioceptive feedback and meanwhile filters out task-irrelevant information in raw multimodal observations. We propose to minimize the upper bound of our multimodal information bottleneck objective for computationally tractable optimization. Experimental evaluations on several challenging locomotion tasks with egocentric images and proprioception show that our method achieves better sample efficiency and zero-shot robustness to unseen white noise than leading baselines. We also empirically demonstrate that leveraging information from egocentric images and proprioception is more helpful for learning policies on locomotion tasks than solely using one single modality.

Reinforcement feedback impairs locomotor adaptation and retention.

Locomotor adaptation is a motor learning process used to alter spatiotemporal elements of walking that are driven by prediction errors, a discrepancy between the expected and actual outcomes of our actions. Sensory and reward prediction errors are two different types of prediction errors that can facilitate locomotor adaptation. Reward and punishment feedback generate reward prediction errors but have demonstrated mixed effects on upper extremity motor learning, with punishment enhancing adaptation, and reward supporting motor memory. However, an in-depth behavioral analysis of these distinct forms of feedback is sparse in locomotor tasks. For this study, three groups of healthy young adults were divided into distinct feedback groups [Supervised, Reward, Punishment] and performed a novel locomotor adaptation task where each participant adapted their knee flexion to 30 degrees greater than baseline, guided by visual supervised or reinforcement feedback (Adaptation). Participants were then asked to recall the new walking pattern without feedback (Retention) and after a washout period with feedback restored (Savings). We found that all groups learned the adaptation task with external feedback. However, contrary to our initial hypothesis, enhancing sensory feedback with a visual representation of the knee angle (Supervised) accelerated the rate of learning and short-term retention in comparison to monetary reinforcement feedback. Reward and Punishment displayed similar rates of adaptation, short-term retention, and savings, suggesting both types of reinforcement feedback work similarly in locomotor adaptation. Moreover, all feedback enhanced the aftereffect of locomotor task indicating changes to implicit learning. These results demonstrate the multi-faceted nature of reinforcement feedback on locomotor adaptation and demonstrate the possible different neural substrates that underly reward and sensory prediction errors during different motor tasks.

Lower Extremity Joint Kinetics during Walk-to-Run and Run-to-Walk Transitions

Lower extremity joint kinetic factors are thought to modulate walk-to-run transition (WRT) and run-to-walk transition (RWT). This study aimed to investigate joint stiffness and energetic patterns during the WRT and RWT processes and identify whether gait transitions occur within a single step or not. Ten healthy subjects participated in treadmill WRT (1.8–2.4 m/s) and RWT (2.4–1.8 m/s) tests. We investigated two steps before transition (S–2, S–1), two steps after transition (S1, S2) and the transition step (S0). We found significant differences between S2 and S–2 for ankle joint stiffness during WRT and RWT (p < 0.001); for hip joint stiffness, we found significant differences between S1 and S–1 during WRT and RWT (p ≤ 0.001). Additionally, stance phase mechanical energy generation was observed to transfer from proximal to distal joints at S0 during WRT, and from distal to proximal at S0 during RWT. Transition step ankle kinematic and kinetic patterns were similar to the target locomotion task gait format in both WRT and RWT. Moreover, RWT required longer adaptation time compared with WRT. These findings indicate that WRT and RWT were modulated before and after the actual transitions, not within a single step. Redistribution of joint mechanical work generation was related to gait transition triggers, which modulate the WRT and RWT processes.

Using mobile eye tracking to measure cognitive load through gaze behavior during walking in lower limb prosthesis users: A preliminary assessment

BackgroundLower limb amputation does not affect only physical and psychological functioning but the use of a prosthetic device can also lead to increased cognitive demands. Measuring cognitive load objectively is challenging, and therefore, most studies use questionnaires that are easy to apply but can suffer from subjective bias. Motivated by this, the present study investigated whether a mobile eye tracker can be used to objectively measure cognitive load by monitoring gaze behavior during a set of motor tasks. MethodsFive prosthetic users and eight able-bodied controls participated in this study. Eye tracking data and kinematics were recorded during a set of motor tasks (level ground walking, walking on uneven terrain, obstacle avoidance, stairs up and ramp down, as well as ramp up and stairs down) while the participants were asked to focus their gaze on a visual target for as long as possible. Target fixation times and increase in pupil diameters were determined and correlated to subjective ratings of cognitive load. FindingsOverall, target fixation time and pupil diameter showed strong negative and positive correlations, respectively, to the subjective rating of cognitive load in the able-bodied controls (−0.75 and 0.80, respectively). However, the individual correlation strength, and in some cases, even the sign, was different across participants. A similar trend could be observed in prosthetic users. InterpretationThe results of this study showed that a mobile eye tracker may be used to estimate cognitive load in prosthesis users during locomotor tasks. This paves the way to establish a new approach to assessing cognitive load, which is objective and yet practical and simple to administer. Nevertheless, future studies should corroborate these results by comparing them to other objective measures as well as focus on translating the proposed approach outside of a laboratory.

Postural control before and after transitional locomotor tasks in children on the autism spectrum: A case-control study

BackgroundInstrumented measurements of postural control provide a more accurate insight into the motor development of children with autism. This study aimed to identify postural control deficits in autistic children during quiet standing before and after transient locomotor task. It was hypothesized that the parameters that characterize the trajectory of center of foot pressure (COP) displacement would be higher in autistic children compared to typically developing children. MethodsSixteen autistic children aged 6–10 but without a comorbidity diagnosis, were enrolled in the study group. The control group comprised 16 typically developing peers.The assessment of the transitional task comprised four different conditions: unperturbed and perturbed transition, stepping up, and stepping down tasks. Analysis of the COP signal was carried out for three distinct phases, i.e., phase 1 – quiet standing before step initiation, phase 2 – transit, and phase 3 - quiet standing until measurement completion. FindingsThe two-way ANOVA with a 2 × 4 factorial design (group × testing condition) revealed a group effect on all posturographic variables in the antero-posterior and medio-lateral directions of phase 1 and in the antero-posterior direction of phase 3. The Bonferroni post-hoc test showed the means of all those variables were significantly higher for the autistic than for typically developing children. Group allocation also had an effect on the time of transit and step length, which turned out to be significantly longer in autistic children compared to healthy peers. InterpretationAutistic children show increased postural sway before and after transitional locomotor tasks compared to typically developing children.The trial was prospectively registered in the Australian and New Zealand Clinical Trials Registry (no. ACTRN12621001113842; date registered: 23.08.2021).

MorphVAE: Advancing Morphological Design of Voxel-Based Soft Robots with Variational Autoencoders

Soft robot design is an intricate field with unique challenges due to its complex and vast search space. In the past literature, evolutionary computation algorithms, including novel probabilistic generative models (PGMs), have shown potential in this realm. However, these methods are sample inefficient and predominantly focus on rigid robots in locomotion tasks, which limit their performance and application in robot design automation. In this work, we propose MorphVAE, an innovative PGM that incorporates a multi-task training scheme and a meticulously crafted sampling technique termed ``continuous natural selection'', aimed at bolstering sample efficiency. This method empowers us to gain insights from assessed samples across diverse tasks and temporal evolutionary stages, while simultaneously maintaining a delicate balance between optimization efficiency and biodiversity. Through extensive experiments in various locomotion and manipulation tasks, we substantiate the efficiency of MorphVAE in generating high-performing and diverse designs, surpassing the performance of competitive baselines.

Task-Oriented Self-Imitation Learning for Robotic Autonomous Skill Acquisition

The inferior sample efficiency of reinforcement learning (RL) and the requirement for high-quality demonstrations in imitation learning (IL) will hinder their application in real-world robots. To address this challenge, a novel self-evolution framework, named task-oriented self-imitation learning (TOSIL), is proposed. To circumvent external demonstrations, the top-K self-generated trajectories are chosen as expert data from both per-episode exploration and long-term return perspectives. Each transition is assigned a guide reward, which is formulated by these trajectories. The guide rewards update as the agent evolves, encouraging good exploration behaviors. This methodology guarantees that the agent explores in the direction relevant to the task, improving sample efficiency and asymptotic performance. The experimental results on locomotion and manipulation tasks indicate that the proposed framework outperforms other state-of-the-art RL methods. Furthermore, the integration of suboptimal trajectories has the potential to improve the sample efficiency while maintaining performance. This is a significant advancement in autonomous skill acquisition for robots.

Reinforcement Learning during Locomotion.

When learning a new motor skill, people often must use trial and error to discover which movement is best. In the reinforcement learning framework, this concept is known as exploration and has been linked to increased movement variability in motor tasks. For locomotor tasks, however, increased variability decreases upright stability. As such, exploration during gait may jeopardize balance and safety, making reinforcement learning less effective. Therefore, we set out to determine if humans could acquire and retain a novel locomotor pattern using reinforcement learning alone. Young healthy male and female participants walked on a treadmill and were provided with binary reward feedback (indicated by a green checkmark on the screen) that was tied to a fixed monetary bonus, to learn a novel stepping pattern. We also recruited a comparison group who walked with the same novel stepping pattern but did so by correcting for target error, induced by providing real-time veridical visual feedback of steps and a target. In two experiments, we compared learning, motor variability, and two forms of motor memories between the groups. We found that individuals in the binary reward group did, in fact, acquire the new walking pattern by exploring (increasing motor variability). Additionally, while reinforcement learning did not increase implicit motor memories, it resulted in more accurate explicit motor memories compared with the target error group. Overall, these results demonstrate that humans can acquire new walking patterns with reinforcement learning and retain much of the learning over 24 h.

The effects of combined transcranial brain stimulation and a 4-week visuomotor stepping training on voluntary step initiation in persons with chronic stroke-a pilot study.

Evidence suggests that transcranial direct current stimulation (tDCS) can enhance motor performance and learning of hand tasks in persons with chronic stroke (PCS). However, the effects of tDCS on the locomotor tasks in PCS are unclear. This pilot study aimed to: (1) determine aggregate effects of anodal tDCS combined with step training on improvements of the neural and biomechanical attributes of stepping initiation in a small cohort of persons with chronic stroke (PCS) over a 4-week training program; and (2) assess the feasibility and efficacy of this novel approach for improving voluntary stepping initiation in PCS. A total of 10 PCS were randomly assigned to one of two training groups, consisting of either 12 sessions of VST paired with a-tDCS (n = 6) or sham tDCS (s-tDCS, n = 4) over 4 weeks, with step initiation (SI) tests at pre-training, post-training, 1-week and 1-month follow-ups. Primary outcomes were: baseline vertical ground reaction force (B-vGRF), response time (RT) to initiate anticipatory postural adjustment (APA), and the retention of B-VGRF and RT. a-tDCS paired with a 4-week VST program results in a significant increase in paretic weight loading at 1-week follow up. Furthermore, a-tDCS in combination with VST led to significantly greater retention of paretic BWB compared with the sham group at 1 week post-training. The preliminary findings suggest a 4-week VST results in improved paretic limb weight bearing (WB) during SI in PCS. Furthermore, VST combined with a-tDCS may lead to better retention of gait improvements (NCT04437251) (https://classic.clinicaltrials.gov/ct2/show/NCT04437251).

Continuous Locomotion Mode and Task Identification for an Assistive Exoskeleton Based on Neuromuscular-Mechanical Fusion.

Human walking parameters exhibit significant variability depending on the terrain, speed, and load. Assistive exoskeletons currently focus on the recognition of locomotion terrain, ignoring the identification of locomotion tasks, which are also essential for control strategies. The aim of this study was to develop an interface for locomotion mode and task identification based on a neuromuscular-mechanical fusion algorithm. The modes of level and incline and tasks of speed and load were explored, and seven able-bodied participants were recruited. A continuous stream of assistive decisions supporting timely exoskeleton control was achieved according to the classification of locomotion. We investigated the optimal algorithm, feature set, window increment, window length, and robustness for precise identification and synchronization between exoskeleton assistive force and human limb movements (human-machine collaboration). The best recognition results were obtained when using a support vector machine, a root mean square/waveform length/acceleration feature set, a window length of 170, and a window increment of 20. The average identification accuracy reached 98.7% ± 1.3%. These results suggest that the surface electromyography-acceleration can be effectively used for locomotion mode and task identification. This study contributes to the development of locomotion mode and task recognition as well as exoskeleton control for seamless transitions.

Abstract WP298: Intracisternal Bolus Administration of Stanniocalcin-2 Promotes Post-Stroke Recovery in a Translational Aged Rodent Model

Stanniocalcin-2 (STC-2) is a glycoprotein implicated in multiple physiological pathways, including calcium and phosphate homeostasis, cell proliferation, and anti-inflammatory responses. The molecular mechanisms by which STC-2 functions in the brain and its specific roles in various neurological processes are not well-defined. We have previously shown that intraventricular administration of STC-2 post-stroke increases endogenous stem cell production and improves functional recovery in an adult rodent model of distal middle cerebral artery occlusion (dMCAO). To further investigate the translatability and preclinical efficacy of STC-2 as a protein therapeutic, we intracisternally administered recombinant STC-2 one week post-dMCAO in aged rats (18-24 months) and comprehensively assessed recovery over a 6 week period via sensorimotor, locomotor, and cognitive tasks. Our results indicate that just two bolus intracisternal injections of STC-2 improved stroke recovery in a battery of behavioral assessments in comparison to a control group (n=9 per group, unpaired t-tests with Welch's correction, see Figure 1A-D). Improved sensorimotor recovery (as assessed by vibrissae-forepaw and hind-rearing tests), locomotor recovery (as assessed by an open field test), and cognitive recovery (as assessed by a novel object recognition test) were observed. Further research is required to delve into the underlying molecular mechanisms responsible. Nevertheless, we are optimistic about the substantial translational prospects of this study. Our findings have the potential to pave the way for a novel protein therapeutic for the treatment of stroke deficits.

Self‐Sustained and Coordinated Rhythmic Deformations with SMA for Controller‐Free Locomotion

This study presents a modular, electronics‐free, and fully onboard control and actuation approach for shape memory alloy (SMA)‐based soft robots to achieve locomotion tasks. This approach exploits the nonlinear mechanics of compliant curved beams and carefully designed mechanical control circuits to create and synchronize rhythmic deformation cycles, mimicking the central pattern generators prevalent in animal locomotions. More specifically, the study elucidates a new strategy to amplify the actuation performance of the shape memory coil actuator by coupling it to a carefully designed, monostable curve beam with a snap‐through buckling behavior. Such SMA‐curved beam assembly is integrated with an entirely mechanical circuit featuring a slider mechanism. This circuit can automatically cut off and supply current to the SMA according to its deformation status, generating a self‐sustained rhythmic deformation cycle using a simple DC power supply. Finally, this study presents a new strategy to coordinate (synchronize) two rhythmic deformation cycles from two robotic modules to achieve efficient crawling locomotion but still use a single DC power. This work represents a significant step toward fully autonomous, electronics‐free SMA‐based locomotion robots with fully onboard actuation and control.

Diverse Imitation Learning via Self-Organizing Generative Models.

Imitation learning (IL) is a well-known problem in the field of Markov decision process (MDP), where one is given multiple demonstration trajectories generated by expert(s), and the goal is to replicate the hidden expert-policies so that when the MDP is run independently, it generates trajectories close to the demonstrated ones. IL is one of the most useful tools used in building versatile robots that can learn from examples. This task becomes particularly challenging when the expert exhibits a mixture of behavior modes. Prior work has introduced latent variables to model variations of the expert policy. However, our experiments show that the existing works do not exhibit appropriate imitation of individual modes. To tackle this problem, we first draw inspiration from the well-known classical technique of self-organizing maps (SOMs) and introduce an encoder-free generative model-referred to as the self-organizing generative (SOG) model-for learning multimodal data distributions from samples. We then apply SOG for behavior cloning (BC)-a framework that learns deterministic policies without considering the environment-to accurately distinguish and imitate different modes. Then, we integrate it with generative adversarial IL (GAIL)-a framework that learns policies while considering the environment-to make the learning robust toward compounding errors at unseen states. We show that our method significantly outperforms the state of the art across multiple experiments within the MuJoCo simulator, including locomotion and robotic manipulation tasks.

Goal-Conditioned Hierarchical Reinforcement Learning With High-Level Model Approximation.

Hierarchical reinforcement learning (HRL) exhibits remarkable potential in addressing large-scale and long-horizon complex tasks. However, a fundamental challenge, which arises from the inherently entangled nature of hierarchical policies, has not been understood well, consequently compromising the training stability and exploration efficiency of HRL. In this article, we propose a novel HRL algorithm, high-level model approximation (HLMA), presenting both theoretical foundations and practical implementations. In HLMA, a Planner constructs an innovative high-level dynamic model to predict the k -step transition of the Controller in a subtask. This allows for the estimation of the evolving performance of the Controller. At low level, we leverage the initial state of each subtask, transforming absolute states into relative deviations by a designed operator as Controller input. This approach facilitates the reuse of subtask domain knowledge, enhancing data efficiency. With this designed structure, we establish the local convergence of each component within HLMA and subsequently derive regret bounds to ensure global convergence. Abundant experiments conducted on complex locomotion and navigation tasks demonstrate that HLMA surpasses other state-of-the-art single-level RL and HRL algorithms in terms of sample efficiency and asymptotic performance. In addition, thorough ablation studies validate the effectiveness of each component of HLMA.

An Underactuated Active Transfemoral Prosthesis With Series Elastic Actuators Enables Multiple Locomotion Tasks

An Underactuated Active Transfemoral Prosthesis With Series Elastic Actuators Enables Multiple Locomotion Tasks

A Deep Learning Model with a Self-Attention Mechanism for Leg Joint Angle Estimation across Varied Locomotion Modes.

Conventional trajectory planning for lower limb assistive devices usually relies on a finite-state strategy, which pre-defines fixed trajectory types for specific gait events and activities. The advancement of deep learning enables walking assistive devices to better adapt to varied terrains for diverse users by learning movement patterns from gait data. Using a self-attention mechanism, a temporal deep learning model is developed in this study to continuously generate lower limb joint angle trajectories for an ankle and knee across various activities. Additional analyses, including using Fast Fourier Transform and paired t-tests, are conducted to demonstrate the benefits of the proposed attention model architecture over the existing methods. Transfer learning has also been performed to prove the importance of data diversity. Under a 10-fold leave-one-out testing scheme, the observed attention model errors are 11.50% (±2.37%) and 9.31% (±1.56%) NRMSE for ankle and knee angle estimation, respectively, which are small in comparison to other studies. Statistical analysis using the paired t-test reveals that the proposed attention model appears superior to the baseline model in terms of reduced prediction error. The attention model also produces smoother outputs, which is crucial for safety and comfort. Transfer learning has been shown to effectively reduce model errors and noise, showing the importance of including diverse datasets. The suggested joint angle trajectory generator has the potential to seamlessly switch between different locomotion tasks, thereby mitigating the problem of detecting activity transitions encountered by the traditional finite-state strategy. This data-driven trajectory generation method can also reduce the burden on personalization, as traditional devices rely on prosthetists to experimentally tune many parameters for individuals with diverse gait patterns.

Development and implementation of a new approach for posture control of a hexapod robot to walk in irregular terrains

AbstractThe adaptability of hexapods for various locomotion tasks, especially in rescue and exploration missions, drives their application. Unlike controlled environments, these robots need to navigate ever-changing terrains, where ground irregularities impact foothold positions and origin shifts in contact forces. This dynamic interaction leads to varying hexapod postures, affecting overall system stability. This study introduces a posture control approach that adjusts the hexapod’s main body orientation and height based on terrain topology. The strategy estimates ground slope using limb positions, thereby calculating novel limb trajectories to modify the hexapod’s angular position. Adjusting the hexapod’s height, based on the calculated slope, further enhances main body stability. The proposed methodology is implemented and evaluated on the ATHENA hexapod (All-Terrain Hexapod for Environment Adaptability). Control feasibility is assessed through dynamic analysis of the hexapod’s multibody model on irregular surfaces, using computational simulations in Gazebo software. Environmental complexity’s impact on hexapod stability is tested on both a ramp and uneven terrain. Independent analyses for each scenario evaluate the controller’s effect on roll and pitch angular velocities, as well as height variations. Results demonstrate the strategy’s suitability for both environments, significantly enhancing posture stability.

Background Material Identification Using a Soft Robot

Soft robotics is an emerging technology that provides robots with the ability to adapt to the environment and safely interact with it. Here, the ability of these robots to identify the surface of interaction is critical for grasping and locomotion tasks. This paper describes the capability of a four-limb soft robot that can identify background materials through the collection of reflection coefficients using an embedded antenna and machine learning techniques. The material of a soft-limb robot was characterized in terms of the relative permittivity and the loss tangent for the design of an antenna to collect reflection coefficients. A slot antenna was designed and embedded into a soft limb in order to extract five features in reflection coefficients including the resonant frequency, −3 dB bandwidth taken from the lowest S11, the value of the lowest S11, −3 dB bandwidth taken from the highest S11, and the number of resonant frequencies. A soft robot with the embedded antenna was tested on nine different background materials in an attempt to identify surrounding terrain information and a better robotic operation. The tested background materials included concrete, fabric, grass, gravel, metal, mulch, soil, water, and wood. The results showed that the robot was capable of distinguishing among the nine different materials with an average accuracy of 93.3% for the nine background materials using a bagged decision-tree-based ensemble method algorithm.

ToRoS: A Topology Optimization Approach for Designing Robotic Skins

Soft robotics offers unique advantages in manipulating fragile or deformable objects, human-robot interaction, and exploring inaccessible terrain. However, designing soft robots that produce large, targeted deformations is challenging. In this paper, we propose a new methodology for designing soft robots that combines optimization-based design with a simple and cost-efficient manufacturing process. Our approach is centered around the concept of robotic skins---thin fabrics with 3D-printed reinforcement patterns that augment and control plain silicone actuators. By decoupling shape control and actuation, our approach enables a simpler and cost-efficient manufacturing process. Unlike previous methods that rely on empirical design heuristics for generating desired deformations, our approach automatically discovers complex reinforcement patterns without any need for domain knowledge or human intervention. This is achieved by casting reinforcement design as a nonlinear constrained optimization problem and using a novel, three-field topology optimization approach tailored to fabrics with 3D-printed reinforcements. We demonstrate the potential of our approach by designing soft robotic actuators capable of various motions such as bending, contraction, twist, and combinations thereof. We also demonstrate applications of our robotic skins to robotic grasping with a soft three-finger gripper and locomotion tasks for a soft quadrupedal robot.

Offline Model-Based Adaptable Policy Learning for Decision-Making in Out-of-Support Regions.

In reinforcement learning, a promising direction to avoid online trial-and-error costs is learning from an offline dataset. Current offline reinforcement learning methods commonly learn in the policy space constrained to in-support regions by the offline dataset, in order to ensure the robustness of the outcome policies. Such constraints, however, also limit the potential of the outcome policies. In this paper, to release the potential of offline policy learning, we investigate the decision-making problems in out-of-support regions directly and propose offline Model-based Adaptable Policy LEarning (MAPLE). By this approach, instead of learning in in-support regions, we learn an adaptable policy that can adapt its behavior in out-of-support regions when deployed. We give a practical implementation of MAPLE via meta-learning techniques and ensemble model learning techniques. We conduct experiments on MuJoCo locomotion tasks with offline datasets. The results show that the proposed method can make robust decisions in out-of-support regions and achieve better performance than SOTA algorithms.