Assistive Robotics Applications Research Articles

A Multi-Modal Foundation Model to Assist People with Blindness and Low Vision in Environmental Interaction.

People with blindness and low vision (pBLV) encounter substantial challenges when it comes to comprehensive scene recognition and precise object identification in unfamiliar environments. Additionally, due to the vision loss, pBLV have difficulty in accessing and identifying potential tripping hazards independently. Previous assistive technologies for the visually impaired often struggle in real-world scenarios due to the need for constant training and lack of robustness, which limits their effectiveness, especially in dynamic and unfamiliar environments, where accurate and efficient perception is crucial. Therefore, we frame our research question in this paper as: How can we assist pBLV in recognizing scenes, identifying objects, and detecting potential tripping hazards in unfamiliar environments, where existing assistive technologies often falter due to their lack of robustness? We hypothesize that by leveraging large pretrained foundation models and prompt engineering, we can create a system that effectively addresses the challenges faced by pBLV in unfamiliar environments. Motivated by the prevalence of large pretrained foundation models, particularly in assistive robotics applications, due to their accurate perception and robust contextual understanding in real-world scenarios induced by extensive pretraining, we present a pioneering approach that leverages foundation models to enhance visual perception for pBLV, offering detailed and comprehensive descriptions of the surrounding environment and providing warnings about potential risks. Specifically, our method begins by leveraging a large-image tagging model (i.e., Recognize Anything Model (RAM)) to identify all common objects present in the captured images. The recognition results and user query are then integrated into a prompt, tailored specifically for pBLV, using prompt engineering. By combining the prompt and input image, a vision-language foundation model (i.e., InstructBLIP) generates detailed and comprehensive descriptions of the environment and identifies potential risks in the environment by analyzing environmental objects and scenic landmarks, relevant to the prompt. We evaluate our approach through experiments conducted on both indoor and outdoor datasets. Our results demonstrate that our method can recognize objects accurately and provide insightful descriptions and analysis of the environment for pBLV.

LLMT: A Transformer-Based Multi-Modal Lower Limb Human Motion Prediction Model for Assistive Robotics Applications

LLMT: A Transformer-Based Multi-Modal Lower Limb Human Motion Prediction Model for Assistive Robotics Applications

Development of a Wearable Ultrasound Transducer for Sensing Muscle Activities in Assistive Robotics Applications

Robotic prostheses and powered exoskeletons are novel assistive robotic devices for modern medicine. Muscle activity sensing plays an important role in controlling assistive robotics devices. Most devices measure the surface electromyography (sEMG) signal for myoelectric control. However, sEMG is an integrated signal from muscle activities. It is difficult to sense muscle movements in specific small regions, particularly at different depths. Alternatively, traditional ultrasound imaging has recently been proposed to monitor muscle activity due to its ability to directly visualize superficial and at-depth muscles. Despite their advantages, traditional ultrasound probes lack wearability. In this paper, a wearable ultrasound (US) transducer, based on lead zirconate titanate (PZT) and a polyimide substrate, was developed for a muscle activity sensing demonstration. The fabricated PZT-5A elements were arranged into a 4 × 4 array and then packaged in polydimethylsiloxane (PDMS). In vitro porcine tissue experiments were carried out by generating the muscle activities artificially, and the muscle movements were detected by the proposed wearable US transducer via muscle movement imaging. Experimental results showed that all 16 elements had very similar acoustic behaviors: the averaged central frequency, -6 dB bandwidth, and electrical impedance in water were 10.59 MHz, 37.69%, and 78.41 Ω, respectively. The in vitro study successfully demonstrated the capability of monitoring local muscle activity using the prototyped wearable transducer. The findings indicate that ultrasonic sensing may be an alternative to standardize myoelectric control for assistive robotics applications.

Dynamic gripping force estimation and reconstruction in EMG-based human-machine interaction

Electromyography (EMG) can reveal the state of muscle activity in advance, therefore, it has been widely used in human–machine interaction (HMI) to predict human intention. Force estimation from EMG signals is acknowledged as an important research topic in HMI. In order to develop a simple and smooth HMI system, it is necessary to estimate the dynamic force effectively and smoothly from a small number of EMG electrodes. In this paper, we have proposed an EMG-based dynamic force reconstruction scheme applied in HMI system. A deep neural prediction network using one-dimensional convolutional structure has been proposed to learn the complex EMG features automatically from three-channel EMG signals. This model was applied in our interactive system to estimate dynamic force and reconstruct it on a robotic gripper for precise EMG-based robot control. Our proposed model outperformed the two-dimensional convolutional neural network (CNN) method and feature-based linear regression. And it can meet the requirement of online interaction. The offline and online tests have shown good estimation performance with R2 of 0.99 and 0.83, respectively. The average prediction speed has reached 115.5 μs per sample. The system has avoided tedious feature extraction process and has demonstrated dynamic recognition in real time which can further advance various prosthesis and assistive robotic applications in the future.

Personalized home-care support for the elderly: a field experience with a social robot at home

Socially assistive robotics (SAR) is getting a lot of attention for its potential in assisting elderly users. However, for robotic assistive applications to be effective, they need to satisfy the particular needs of each user and be well perceived. For this purpose, a personalization based on user’s characteristics such as personality and cognitive profile, and their dynamic changes is a crucial factor. Moreover, most of the existing solutions rely on the availability of specific technological infrastructures, generally requiring high economic investment, and that cannot be easily placed in different environments. Personalization and adaptation of assistive robotics applications to different user’s characteristics and needs, and even to different technological environments, are still not fully addressed in real environments. In the present work, the results of the UPA4SAR project are presented. The project aimed at providing a social robotic system to deliver assistive tasks for home care of patients with mild cognitive impairment in a personalized and adaptive way. We introduce the general architecture of the system and the developed robotic behaviors. Personalization and dynamic adaptation of assistive tasks are realized using a service-oriented approach by taking into account both user’s characteristics and environmental dynamic conditions. Field experimentation of the project was carried out with 7 patients, using the robotic system autonomously running in their homes for a total of 118 days. Results showed a reliable functioning of the proposed robotic system, a generally positive reaction, and a good acceptability rate from patients.

Hybridized neural network inspired behavioural modelling of pneumatic artificial muscles for assistive robotic applications

PurposeThis paper aims to devise a first-of-its-kind methodology to determine the design, operating conditions and actuation strategy of pneumatic artificial muscles (PAMs) for assistive robotic applications. This requires extensive characterization, data set generation and meaningful modelling between PAM characteristics and design variables. Such a characterization should cover a wide range of design and operation parameters. This is a stepping stone towards generating a design guide for this highly popular compliant actuator, just like any conventional element of a mechanism.Design/methodology/approachCharacterization of a large pool of custom fabricated PAMs of varying designs is performed to determine their static and dynamic behaviours. Metaheuristic optimizer-based artificial neural network (ANN) structures are used to determine eight different models representing PAM behaviour. The assistance of knee flexion during level walking is targeted for evaluating the applicability of the developed actuator by attaching a PAM across the joint. Accordingly, the PAM design and the actuation strategy are optimized through a tabletop emulator.FindingsThe dependence of passive length, static contraction, dynamic step response for inflation and deflation of the PAMs on their design dimensions and operating parameters is successfully modelled by the ANNs. The efficacy of these models is investigated to successfully optimize the PAM design, operation parameters and actuation strategy for using a PAM in assisting knee flexion in human gait.Originality/valueCharacterization of static and the dynamic behaviour of a large pool of PAMs with varying designs over a wide range of operating conditions is the novel feature in this article. A lucid customizable fabrication technique is discussed to obtain a wide variety of PAM designs. Metaheuristic-based ANNs are used for tackling high non-linearity in data while modelling the PAM behaviour. An innovative tabletop emulator is used for investigating the utility of the models in the possible application of PAMs in assistive robotics.

A Systematic Study on Electromyography-Based Hand Gesture Recognition for Assistive Robots Using Deep Learning and Machine Learning Models.

Upper limb amputation severely affects the quality of life and the activities of daily living of a person. In the last decade, many robotic hand prostheses have been developed which are controlled by using various sensing technologies such as artificial vision and tactile and surface electromyography (sEMG). If controlled properly, these prostheses can significantly improve the daily life of hand amputees by providing them with more autonomy in physical activities. However, despite the advancements in sensing technologies, as well as excellent mechanical capabilities of the prosthetic devices, their control is often limited and usually requires a long time for training and adaptation of the users. The myoelectric prostheses use signals from residual stump muscles to restore the function of the lost limbs seamlessly. However, the use of the sEMG signals in robotic as a user control signal is very complicated due to the presence of noise, and the need for heavy computational power. In this article, we developed motion intention classifiers for transradial (TR) amputees based on EMG data by implementing various machine learning and deep learning models. We benchmarked the performance of these classifiers based on overall generalization across various classes and we presented a systematic study on the impact of time domain features and pre-processing parameters on the performance of the classification models. Our results showed that Ensemble learning and deep learning algorithms outperformed other classical machine learning algorithms. Investigating the trend of varying sliding window on feature-based and non-feature-based classification model revealed interesting correlation with the level of amputation. The study also covered the analysis of performance of classifiers on amputation conditions since the history of amputation and conditions are different to each amputee. These results are vital for understanding the development of machine learning-based classifiers for assistive robotic applications.

Safety Assessment Review of a Dressing Assistance Robot.

Hazard analysis methods such as HAZOP and STPA have proven to be effective methods for assurance of system safety for years. However, the dimensionality and human factors uncertainty of many assistive robotic applications challenges the capability of these methods to provide comprehensive coverage of safety issues from interdisciplinary perspectives in a timely and cost-effective manner. Physically assistive tasks in which a range of dynamic contexts require continuous human–robot physical interaction such as e.g., robot-assisted dressing or sit-to-stand pose a new paradigm for safe design and safety analysis methodology. For these types of tasks, considerations have to be made for a range of dynamic contexts where the robot-assistance requires close and continuous physical contact with users. Current regulations mainly cover industrial collaborative robotics regarding physical human–robot interaction (pHRI) but largely neglects direct and continuous physical human contact. In this paper, we explore limitations of commonly used safety analysis techniques when applied to robot-assisted dressing scenarios. We provide a detailed analysis of the system requirements from the user perspective and consider user-bounded hazards that can compromise safety of this complex pHRI.

A Deployment Framework for Formally Verified Human-Robot Interactions

In the future, assistive robots will spread to everyday settings and regularly interact with humans. This paper introduces a deployment approach for assistive robotic applications where human-robot interaction is the main element. The deployment infrastructure hinges on a model-to-code transformation technique and a ROS-based middleware layer and enables deployment in real life or simulation in a virtual environment. The approach fits into a model-driven framework for the formal verification of interactive scenarios. At design-time, the application analyst estimates the most likely outcome of the robotic mission through Statistical Model Checking of a Stochastic Hybrid Automata network modeling the scenario. We introduce an innovative approach to convert a specific subset of Stochastic Hybrid Automata into executable code to control the robot and respond to human actions. Deploying or simulating the application allows analysts to validate the results obtained at design time or to refine the formal model based on runs in the real or the virtual scene. The methodology’s effectiveness is tested via simulation of use cases from the healthcare setting, which can significantly benefit from this kind of approach thanks to its innovative features related to human physiology and autonomous behavior.

I2E: A Cognitive Architecture Based on Emotions for Assistive Robotics Applications

Emotions and personality play an essential role in human behavior, their considerations, and decision-making. Humans infer emotions from several modals, merging them all. They are an interface between a subject’s internal and external means. This paper presents the design, implementation, and tests of the Inference of an Emotional State (I2E): a cognitive architecture based on emotions for assistive robotics applications, which uses, as inputs, emotions recognized previously by four affective modals who inferred the emotional state to an assistive robot. Unlike solutions that classify emotions, with a single sign, the architecture proposed in this article will merge four sources of information about emotions into one. For this inference to be closer to a human being, a Fuzzy System Mamdani was used to infer the user’s personalities, and a MultiLayer Perceptron (MLP) was used to infer the robot’s personality. The hypothesis tested in this work was based on the Mehrabian studies and in addition to three experts in psychologists. The I2E architecture proved to be quite efficient for identifying an emotion with various types of input.

A sEMG-Driven Soft ExoSuit Based on Twisted String Actuators for Elbow Assistive Applications

The scope of this letter is to show the applicability of the Twisted String Actuators (TSAs) for lightweight, wearable and assistive robotic applications. To this aim, we have developed a novel surface electromyography (sEMG)-driven soft ExoSuit using the TSAs to perform both single and dual-arm elbow assistive applications. The proposed ExoSuit, with an overall weight of 1650 g, uses a pair of TSAs mounted in the back of the user, connected via tendons to the user's forearms to actuate each arm independently for supporting external loads. We confirm this new light-weight and customizable wearable solution via multiple user studies based on the biceps and tricep' sEMG measurements. We demonstrate that user's muscles can automatically activate and regulate the TSAs and compensate for the user's effort: by using our controller based on a Double Threshold Strategy (DTS) with a standard PID regulator, we report that the system was able to limit the biceps' sEMG activity under an arbitrary target threshold, compensating a muscular activity equal to 220% (related to a single arm 3 kg load) and 110% (related to a dual arm 4 kg load) of the threshold value itself. Moreover, the triceps' sEMG signal detects the external load and, depending on the threshold, returns the system to the initial state where it requires no assistance from the ExoSuit. The experimental results show the proposed ExoSuit's capabilities in both single and dual-arm load compensation tasks. Therefore, the applicability of the TSAs is experimentally demonstrated for a real-case assistive device, fostering future studies and developments of this kind of actuation strategy for wearable robotic systems.

Trajectory-based gait pattern shift detection for assistive robotics applications

In the medical field, rehabilitation of the lower limbs is a slow and continuous process, where healthcare professionals have to follow each patient over long periods of time. In conventional rehabilitation, the progression of a patient is assessed by a professional, who analyzes visual tracking data. This assessment is dependent on each professional’s own experience. In this paper, we propose an approach to analyze tracking data, captured by our robotic walker’s gait tracking system, to detect shifts in the gait pattern over time automatically. For this purpose, the system takes in gait tracking data and segments it into gait cycles (heel strike to heel strike). Then, our approach handles each gait cycle considering it as a group of gait parameters that define trajectories in that time frame. From each gait parameter within the cycle, spatiotemporal features are extracted and similarity rates are computed using autoencoders. These spatiotemporal features and similarity rates are fused in a feature space which is fed to a one-class support vector machine that constructs a model of the observable gait cycle. Each posterior observed gait cycle is checked for shifts, using a set of proposed novelty detection techniques. Experimental tests using a dataset captured using our robotic walker platform revealed a promising performance when detecting if a gait cycle is ‘reference’ or ‘novel’ when compared to a previously trained model of a unique ‘reference’ gait pattern.

Disturbance observer based dynamic load torque compensator for assistive exoskeletons

In assistive robotics applications, the human limb is attached intimately to the robotic exoskeleton. The coupled dynamics of the human-exoskeleton system are highly nonlinear and uncertain, and effectively appear as uncertain load-torques at the joint actuators of the exoskeleton. This uncertainty makes the application of standard computed torque techniques quite challenging. Furthermore, the need for safe human interaction severely limits the gear ratio of the actuators. With small gear ratios, the uncertain joint load-torques cannot be ignored and need to be effectively compensated. A novel disturbance observer based dynamic load-torque compensator is hereby proposed and analysed for the current controlled DC-drive actuators of the exoskeleton, to effectively compensate the said uncertain load-torques at the joint level. The feedforward dynamic load-torque compensator is proposed based on the higher order dynamic model of the current controlled DC-drive. The dynamic load-torque compensator based current controlled DC-drive is then combined with a tailored feedback disturbance observer to further improve the compensation performance in the presence of drive parametric uncertainty. The proposed compensator structure is shown both theoretically and practically to give significantly improved performance w.r.t disturbance observer compensator alone and classical static load-torque compensator, for rated load-torque frequencies up to 1.6 Hz, which is a typical joint frequency bound for normal daily activities for elderly. It is also shown theoretically that the proposed compensator achieves the improved performance with comparable reference current requirement for the current controlled DC-drive.

A Robust Visual Human Detection Approach With UKF-Based Motion Tracking for a Mobile Robot

Robust tracking of a human in a video sequence is an essential prerequisite to an increasing number of applications, where a robot needs to interact with a human user or operates in a human-inhabited environment. This paper presents a robust approach that enables a mobile robot to detect and track a human using an onboard RGB-D sensor. Such robots could be used for security, surveillance, and assistive robotics applications. The proposed approach has real-time computation power through a unique combination of new ideas and well-established techniques. In the proposed method, background subtraction is combined with depth segmentation detector and template matching method to initialize the human tracking automatically. A novel concept of head and hand creation based on depth of interest is introduced in this paper to track the human silhouette in a dynamic environment, when the robot is moving. To make the algorithm robust, a series of detectors (e.g., height, size, and shape) is utilized to distinguish target human from other objects. Because of the relatively high computation time of the silhouette-matching-based method, a confidence level is defined, which allows using the matching-based method only where it is imperative. An unscented Kalman filter is used to predict the human location in the image frame to maintain the continuity of the robot motion. The efficacy of the approach is demonstrated through a real experiment on a mobile robot navigating in an indoor environment.

Spring-based tactile array for assistive robotic surgical applications

Tactile sensing is highly desirable for robotic surgery, more specifically minimally invasive surgery, during tissue/organ manipulation. For safe handling and safe grasping of tissues, the two most important aspects are the monitoring and controlling of force and the stiffness information that exists at the sensor-tissue interface. In this study, a spring-based tactile sensing system is used to measure both the interacting forces and stiffness in surgical assistive applications. A differential analysis is performed to obtain the stiffness of a compliant object whilst individual elements of the proposed sensing array experience different deflections when they come in contact with the object. A lumped model formulation for tactile array performance shows that larger differences in stiffness between the force sensing elements provide higher sensitivity in evaluating the object stiffness. For benchmarking of our sensing principle, a macro tactile-sensor array is designed and tested. Different stiffness combinations of the sensing elements show an inverse linear relationship between the stiffness of the object and the output signal magnitude of the stiffness sensing unit. The proposed tactile array based on spring-based stiffness sensing has high potential for safe grasping/handling of the tissues when integrated into the jaws of a surgical grasper during the minimally invasive robotic procedures.

Floor Covering and Surface Identification for Assistive Mobile Robotic Real-Time Room Localization Application

Assistive robotic applications require systems capable of interaction in the human world, a workspace which is highly dynamic and not always predictable. Mobile assistive devices face the additional and complex problem of when and if intervention should occur; therefore before any trajectory assistance is given, the robotic device must know where it is in real-time, without unnecessary disruption or delay to the user requirements. In this paper, we demonstrate a novel robust method for determining room identification from floor features in a real-time computational frame for autonomous and assistive robotics in the human environment. We utilize two inexpensive sensors: an optical mouse sensor for straightforward and rapid, texture or pattern sampling, and a four color photodiode light sensor for fast color determination. We show how data relating floor texture and color obtained from typical dynamic human environments, using these two sensors, compares favorably with data obtained from a standard webcam. We show that suitable data can be extracted from these two sensors at a rate 16 times faster than a standard webcam, and that these data are in a form which can be rapidly processed using readily available classification techniques, suitable for real-time system application. We achieved a 95% correct classification accuracy identifying 133 rooms' flooring from 35 classes, suitable for fast coarse global room localization application, boundary crossing detection, and additionally some degree of surface type identification.

End-user programming architecture facilitates the uptake of robots in social therapies

This paper proposes an architecture that makes programming of robot behavior of an arbitrary complexity possible for end-users and shows the technical solutions in a way that is easy to understand and generalize to different situations. It aims to facilitate the uptake and actual use of robot technologies in therapies for training social skills to autistic children. However, the framework is easy to generalize for an arbitrary human–robot interaction application, where users with no technical background need to program robots, i.e. in various assistive robotics applications. We identified the main needs of end-user programming of robots as a basic prerequisite for the uptake of robots in assistive applications. These are reusability, modularity, affordances for natural interaction and the ease of use. After reviewing the shortcomings of the existing architectures, we developed an initial architecture according to these principles and embedded it in a robot platform. Further, we used a co-creation process to develop and concretize the architecture to facilitate solutions and create affordances for robot specialists and therapists. Several pilot tests showed that different user groups, including therapists with general computer skills and adolescents with autism could make simple training or general behavioral scenarios within 1 h, by connecting existing behavioral blocks and by typing textual robot commands for fine-tuning the behaviors. In addition, this paper explains the basic concepts behind the TiViPE based robot control platform, and gives guidelines for choosing the robot programming tool and designing end-user platforms for robots.