Assistive Robotic Arm Research Articles

A shared robot control system combining augmented reality and motor imagery brain–computer interfaces with eye tracking

Objective: Brain-computer interface (BCI) control systems monitor neural activity to detect the user's intentions, enabling device control through mental imagery. Despite their potential, decoding neural activity in real-world conditions poses significant challenges, making BCIs currently impractical compared to traditional interaction methods. This study introduces a novel motor imagery (MI) BCI control strategy for operating a physically assistive robotic arm, addressing the difficulties of MI decoding from electroencephalogram (EEG) signals, which are inherently non-stationary and vary across individuals.
Approach: A proof-of-concept BCI control system was developed using commercially available hardware, integrating MI with eye tracking in an augmented reality (AR) user interface to facilitate a shared control approach. This system proposes actions based on the user's gaze, enabling selection through imagined movements. A user study was conducted to evaluate the system's usability, focusing on its effectiveness and efficiency.
Main results:Participants performed tasks that simulated everyday activities with the robotic arm, demonstrating the shared control system's feasibility and practicality in real-world scenarios. Despite low online decoding performance (mean accuracy: 0.52 9, F1: 0.29, Cohen's Kappa: 0.12), participants achieved a mean success rate of 0.83 in the final phase of the user study when given 15 minutes to complete the evaluation tasks. The success rate dropped below 0.5 when a 5-minute cutoff time was selected.
Significance: These results indicate that integrating AR and eye tracking can significantly enhance the usability of BCI systems, despite the complexities of MI-EEG decoding. While efficiency is still low, the effectiveness of our approach was verified. This suggests that BCI systems have the potential to become a viable interaction modality for everyday applications
in the future.

User expectations and experiences of an assistive robotic arm in amyotrophic lateral sclerosis: a multicenter observational study

ObjectiveRobotic arms are innovative assistive devices for ALS patients with progressive motor deficits of arms and hands. The objective was to explore the patients´ expectations towards a robotic arm system and to assess the actual experiences after the provision of the device.MethodsA prospective observational study was conducted at 9 ALS centers in Germany. ALS-related functional deficits were assessed using the ALS-Functional Rating Scale-revised (ALSFRS-R). Motor deficit of the upper limbs was determined using a subscore of three arm-related items of the ALSFRS-R (items 4–6; range 0–12 points). User expectations before provision (expectation group, n = 85) and user experiences after provision (experience group, n = 14) with the device (JACO Assistive Robotic Device, Kinova, Boisbriand, QC, Canada) were assessed.ResultsIn the total cohort, mean ALSFRS-R subscore for arm function was 1.7 (SD: 2.0, 0–9) demonstrating a severe functional deficit of the upper limbs. In the expectation group (n = 85), the following use cases of the robotic arm have been prioritized: handling objects (89%), close-body movements (88%), pressing buttons (87%), serving drinks (86%), and opening cabinets and doors (85%). In the experience group (n = 14), handling objects (79%), serving drinks (79%), near-body movements (71%), pushing buttons (71%), serving food (64%), and opening doors (64%) were the most frequent used cases. Most patients used the device daily (71.4%, n = 10), and 28.6% (n = 4) several times a week. All patients of the experience group found the device helpful, felt safe while using the device, and were satisfied with its reliability. NPS of the assistive robotic arm revealed 64% "promoters" (strong recommendation), 29% "indifferents" (uncertain recommendation) and 7% "detractors" (no recommendation). Total NPS was + 57 demonstrating strong patient satisfaction.ConclusionsInitiation of procurement with a robotic assistive arm was confined to patients with severe functional deficit of the upper limbs. User experience underlined the wide spectrum of use cases of assistive robotic arms in ALS. The positive user experience together with high satisfaction underscore that robotic arm systems serve as a valuable treatment option in ALS patients with severe motor deficits of the arms.

Acceptance of a robotic system for nursing care: a cross-sectional survey with professional nurses, care recipients and relatives

BackgroundThe end-users’ acceptance is a core concept in the development, implementation and evaluation of new systems like robotic systems in daily nursing practice. So far, studies have shown various findings concerning the acceptance of systems that are intended to assist people with support or care needs. Not much has been reported on the acceptance of robots that provide direct physical assistance to nurses in bedside care. Therefore, this study aimed to investigate the acceptance along with ethical implications of the prototype of an assistive robotic arm aiming to support nurses in bedside care, from the perspective of nurses, care recipients and their relatives.MethodsA cross-sectional survey design was applied at an early stage in the technological development of the system. Professional nurses, care recipients and relatives were recruited from a university hospital and a nursing home in Germany. The questionnaire was handed out following either a video or a live demonstration of the lab prototype and a subsequent one-to-one follow-up discussion. Data analysis was performed descriptively.ResultsA total of 67 participants took part in the study. The rejection of specified ethical concerns across all the respondents was 77%. For items related to both perceived usefulness and intention to use, 75% of ratings across all the respondents were positive. In the follow-up discussions, the participants showed interest and openness toward the prototype, although there were varying opinions on aspects such as size, appearance, velocity, and potential impact on workload.ConclusionsRegarding the current state of development, the acceptance among the participants was high, and ethical concerns were relatively minor. Moving forward, it would be beneficial to explore the acceptance in further developmental stages of the system, particularly when the usability is tested.

Robotic Arm-Assisted Total Knee Arthroplasty: Anatomical Alignment and Mid-Term Outcomes from the First Cohort Originating in Greece.

Robotic arm-assisted total knee arthroplasty (RATKA) represents a haptic assistive robotic arm used for bone preparation. The purpose of this study was to present implant survivorship, complications and evaluate patients' satisfaction, clinical and functional outcome of RATKA with a minimum of 1-year follow-up. The Oxford Knee Score was recorded preoperatively and at last follow-up. Patients' satisfaction rates, as well as complications and re-operations were studied. Anatomical alignment including varus, valgus deformities and flexion, extension, pre-and postoperatively were evaluated. A total of 156 patients with mean age = 71.9 years were included in the study. The mean follow-up was 35.7 months, while one revision was performed due to infection. Statistically significant improvement of the Oxford Knee Score, as well as of the knee alignment deformities were recorded, while 99.4% of patients reported to be "very satisfied" or "satisfied" with the procedure. RATKA seems to be a safe, as well as reproductible procedure at short-and mid-term follow-up, while the accurate implant positioning may lead to favorable long-term outcomes.

Adaptive Gravity Compensation Framework Based on Human Upper Limb Model for Assistive Robotic Arm Extender.

The Assistive Robotic Arm Extender (ARAE) is an upper limb assistive and rehabilitation robot that belongs to the end-effector type, enabling it to assist patients with upper limb movement disorders in three-dimensional space. However, the problem of gravity compensation for the human upper limb with this type of robot is crucial, which directly affects the deployment of the robot in the assistive or rehabilitation field. This paper presents an adaptive gravity compensation framework that calculates the compensated force based on the estimated human posture in 3D space. First, we estimated the human arm joint angles in real-time without any wearable sensors, such as inertial measurement unit (IMU) or magnetic sensors, only through the kinematic data of the robot and established human model. The performance of the estimation method was evaluated through a motion capture system, which validated the accuracy of joint angle estimation. Second, the estimated human joint angles were input to the rigid link model to demonstrate the support force profile generated by the robot. The force profile showed that the support force provided by the developed ARAE robot could adaptively change with human arm postures in 3D space. The adaptive gravity compensation framework can improve the usability and feasibility of the 3D end-effector rehabilitation or assistive robot.

A Non-Linear Body Machine Interface for Controlling Assistive Robotic Arms.

Body machine interfaces (BoMIs) enable individuals with paralysis to achieve a greater measure of independence in daily activities by assisting the control of devices such as robotic manipulators. The first BoMIs relied on Principal Component Analysis (PCA) to extract a lower dimensional control space from information in voluntary movement signals. Despite its widespread use, PCA might not be suited for controlling devices with a large number of degrees of freedom, as because of PCs' orthonormality the variance explained by successive components drops sharply after the first. Here, we propose an alternative BoMI based on non-linear autoencoder (AE) networks that mapped arm kinematic signals into joint angles of a 4D virtual robotic manipulator. First, we performed a validation procedure that aimed at selecting an AE structure that would allow to distribute the input variance uniformly across the dimensions of the control space. Then, we assessed the users' proficiency practicing a 3D reaching task by operating the robot with the validated AE. All participants managed to acquire an adequate level of skill when operating the 4D robot. Moreover, they retained the performance across two non-consecutive days of training. While providing users with a fully continuous control of the robot, the entirely unsupervised nature of our approach makes it ideal for applications in a clinical context since it can be tailored to each user's residual movements. We consider these findings as supporting a future implementation of our interface as an assistive tool for people with motor impairments.

A fast and accurate compound collision detector for RRT motion planning

The application of artificial intelligence tools has led to newly developed collision detectors which have better computational efficiency than the kinematics-and-geometry based collision detectors (KCD) to improve robot motion planning strategies. However, new detectors are not very accurate in some cases. To improve the accuracy, a trade-off between efficiency and accuracy is required. We propose a novel compound collision detector (CCD) for collision queries that modifies the planners of rapidly-exploring random tree (RRT) to improve the classical probabilistic collision detector (PCD). It is composed of an exact collision detector (ECD), an inference collision detector (ICD) and a strategy to determine ECD or ICD based on some conditions. In our CCD, we use a sphere-ellipsoidal pseudo distance (SEPD) in the determination strategy to alleviate the problem of highly-frequent outputs of false-positive in narrow passages of PCD, and a node based bounding method (NBB) to increase the speed of data storage and loading for the sub-algorithm ICD. Experiments on a Kinova Jaco assistive robotic arm are taken to evaluate the performance of our CCD, which show an improved accuracy with a small reduction of speed in comparison with PCD. So, it is a promising tool in robot motion planning.

Assistive technology implementation with brain-machine interface

As the development of the Brain Computer Interface, it can be a possible way for human to control the external devices by the mind. This can be good news for the handicapped people whose life qualities are greatly decreased. In this paper, designed an assistive robot arm for those disabled. The assistive robot arm mainly contains a filter, a microcontroller and a motor drive system. The filter can filter out the noises while the microcontroller judging the brain signals and turn it into an actual control signal. The actual control signal will finally drive the motor and for the sake of safety and stability, a PID circuit was add in the motor drive system. The mechanic arm designed can turn what people think to the actual robot behaviour and this will obviously increase the life quality of the disabled people. Its also a light weight system with a wide range of versatility and will bring benefits to all the handicapped.

AI for simplifying the use of an assistive robotic arm for people with severe body impairments

AI for simplifying the use of an assistive robotic arm for people with severe body impairments

Adaptive control of a wheelchair mounted robotic arm with neuromorphically integrated velocity readings and online-learning.

Wheelchair-mounted robotic arms support people with upper extremity disabilities with various activities of daily living (ADL). However, the associated cost and the power consumption of responsive and adaptive assistive robotic arms contribute to the fact that such systems are in limited use. Neuromorphic spiking neural networks can be used for a real-time machine learning-driven control of robots, providing an energy efficient framework for adaptive control. In this work, we demonstrate a neuromorphic adaptive control of a wheelchair-mounted robotic arm deployed on Intel’s Loihi chip. Our algorithm design uses neuromorphically represented and integrated velocity readings to derive the arm’s current state. The proposed controller provides the robotic arm with adaptive signals, guiding its motion while accounting for kinematic changes in real-time. We pilot-tested the device with an able-bodied participant to evaluate its accuracy while performing ADL-related trajectories. We further demonstrated the capacity of the controller to compensate for unexpected inertia-generating payloads using online learning. Videotaped recordings of ADL tasks performed by the robot were viewed by caregivers; data summarizing their feedback on the user experience and the potential benefit of the system is reported.

Wearable Auxiliary Robotic Forearm

For individuals who have some type of physical mobility problem, they need rehabilitation tools that replace their limb capabilities or assistive devices that enhance their limb functions. While for most people, assistive robotic arms can be used as additional limbs to help them perform operations that cannot be conducted with one hand, extending the functional range of the human arm, such as grasping objects at heights. In this paper, a wearable forearm assistive robotic arm with multiple degrees of freedom is studied and constructed with various types of lightweight materials and is mainly composed of three parts: a mechanical arm telescoping part, a mechanical claw part and a wearing part. The interaction is carried out by button through arduino design program. The performance of this robotic arm is investigated through experiments as well as controls in the grasping and assisted handover scenarios.

Learning-Based Approach for a Soft Assistive Robotic Arm to Achieve Simultaneous Position and Force Control

Soft robotics have demonstrated great advantages in assisting elderly/disabled people during daily tasks, owing to their highly dexterous motions and safe human-robot interactions. However, simultaneously controlling the position and force of soft robots is still a challenging task due to soft actuators’ nonlinearity, system uncertainty, and high-dimensional control space. Classical control methods are usually based on first-principle/analytical models that are difficult to derive for soft robots without making significant simplifications. To overcome such control challenges, the central concept of this work is to introduce a learning-based data-driven approach. The approach employs a probabilistic model to explicitly capture system nonlinearity and uncertainty. Besides, nonparametric local learning methods are investigated to deal with redundant high-dimensional control space. The approach is applied in a soft robotic arm interacting with a manikin to simulate the bathing task. Experimental results demonstrate that the soft robotic arm could be well controlled to track the desired position and force simultaneously (maximum error of position and force is 6 mm and 0.037 N). Meanwhile, our method outperforms another typical data-driven approach (maximum error of position and force is 10 mm and 0.058 N). The results indicate that our approach is helpful for soft robots because of the physical interactions needed in assistive tasks.

A Brain-Computer Interface for Teleoperation of a Semiautonomous Mobile Robotic Assistive System Using SLAM

The proposed assistive hybrid brain-computer interface (BCI) semiautonomous mobile robotic arm demonstrates a design that is (1) adaptable by observing environmental changes with sensors and deploying alternate solutions and (2) versatile by receiving commands from the user’s brainwave signals through a noninvasive electroencephalogram cap. Composed of three integrated subsystems, a hybrid BCI controller, an omnidirectional mobile base, and a robotic arm, the proposed robot has commands mapped to the user’s brainwaves related to a set of specific physical or mental tasks. The implementation of sensors and the camera systems enable both the mobile base and the arm to be semiautonomous. The mobile base’s SLAM algorithm has obstacle avoidance capability and path planning to assist the robot maneuver safely. The robot arm calculates and deploys the necessary joint movement to pick up or drop off a desired object selected by the user via a brainwave controlled cursor on a camera feed. Validation, testing, and implementation of the subsystems were conducted using Gazebo. Communication between the BCI controller and the subsystems is tested independently. A loop of prerecorded brainwave data related to each specific task is used to ensure that the mobile base command is executed; the same prerecorded file is used to move the robot arm cursor and initiate a pick-up or drop-off action. A final system test is conducted where the BCI controller input moves the cursor and selects a goal point. Successful virtual demonstrations of the assistive robotic arm show the feasibility of restoring movement capability and autonomy for a disabled user.

Adapt or Perish? Exploring the Effectiveness of Adaptive DoF Control Interaction Methods for Assistive Robot Arms

Robot arms are one of many assistive technologies used by people with motor impairments. Assistive robot arms can allow people to perform activities of daily living (ADL) involving grasping and manipulating objects in their environment without the assistance of caregivers. Suitable input devices (e.g., joysticks) mostly have two Degrees of Freedom (DoF), while most assistive robot arms have six or more. This results in time-consuming and cognitively demanding mode switches to change the mapping of DoFs to control the robot. One option to decrease the difficulty of controlling a high-DoF assistive robot arm using a low-DoF input device is to assign different combinations of movement-DoFs to the device’s input DoFs depending on the current situation (adaptive control). To explore this method of control, we designed two adaptive control methods for a realistic virtual 3D environment. We evaluated our methods against a commonly used non-adaptive control method that requires the user to switch controls manually. This was conducted in a simulated remote study that used Virtual Reality and involved 39 non-disabled participants. Our results show that the number of mode switches necessary to complete a simple pick-and-place task decreases significantly when using an adaptive control type. In contrast, the task completion time and workload stay the same. A thematic analysis of qualitative feedback of our participants suggests that a longer period of training could further improve the performance of adaptive control methods.

Controlling an effector with eye movements: The effect of entangled sensory and motor responsibilities.

Restoring arm and hand function has been indicated by individuals with tetraplegia as one of the most important factors for regaining independence. The overall goal of our research is to develop assistive technologies that allow individuals with tetraplegia to control functional reaching movements. This study served as an initial step toward our overall goal by assessing the feasibility of using eye movements to control the motion of an effector in an experimental environment. We aimed to understand how additional motor requirements placed on the eyes affected eye-hand coordination during functional reaching. We were particularly interested in how eye fixation error was affected when the sensory and motor functions of the eyes were entangled due to the additional motor responsibility. We recorded participants' eye and hand movements while they reached for targets on a monitor. We presented a cursor at the participant's point of gaze position which can be thought of as being similar to the control of an assistive robot arm. To measure eye fixation error, we used an offline filter to extract eye fixations from the raw eye movement data. We compared the fixations to the locations of the targets presented on the monitor. The results show that not only are humans able to use eye movements to direct the cursor to a desired location (1.04 ± 0.15 cm), but they can do so with error similar to that of the hand (0.84 ± 0.05 cm). In other words, despite the additional motor responsibility placed on the eyes during direct eye-movement control of an effector, the ability to coordinate functional reaching movements was unaffected. The outcomes of this study support the efficacy of using the eyes as a direct command input for controlling movement.

Mix Frame Visual Servo Control Framework for Autonomous Assistive Robotic Arms.

Assistive robotic arms (ARAs) that provide care to the elderly and people with disabilities, are a significant part of Human-Robot Interaction (HRI). Presently available ARAs provide non-intuitive interfaces such as joysticks for control and thus, lacks the autonomy to perform daily activities. This study proposes that, for inducing autonomous behavior in ARAs, visual sensors integration is vital, and visual servoing in the direct Cartesian control mode is the preferred method. Generally, ARAs are designed in a configuration where its end-effector’s position is defined in the fixed base frame while orientation is expressed in the end-effector frame. We denoted this configuration as ‘mixed frame robotic arms’. Consequently, conventional visual servo controllers which operate in a single frame of reference are incompatible with mixed frame ARAs. Therefore, we propose a mixed-frame visual servo control framework for ARAs. Moreover, we enlightened the task space kinematics of a mixed frame ARAs, which led us to the development of a novel “mixed frame Jacobian matrix”. The proposed framework was validated on a mixed frame JACO-2 7 DoF ARA using an adaptive proportional derivative controller for achieving image-based visual servoing (IBVS), which showed a significant increase of 31% in the convergence rate, outperforming conventional IBVS joint controllers, especially in the outstretched arm positions and near the base frame. Our Results determine the need for the mixed frame controller for deploying visual servo control on modern ARAs, that can inherently cater to the robotic arm’s joint limits, singularities, and self-collision problems.

Design and development of 5-axis robot

Robotics has been developed to ease the works of human beings using assistive robots and robotic arms. This paper is provided with the design and development of such robotic arm. Since the robotic arm is an analogy of human arm, it consists of several rigid links connected and articulated using rotational joints. In this paper, the design and analysis of a simple 5-axis industrial robotic arm is brought forth. The foremost information on the actuators and other electrical components used in this robotic arm is presented. The angular velocity, angular acceleration, and torque at different joints have been calculated and analyzed with the help of the multibody dynamics simulation software ADAMS (Automated Dynamic Analysis of Mechanical Systems). This simulation assisted in determining the stability of the robot and in choosing the appropriate actuators.

IMU-Based Hand Gesture Interface Implementing a Sequence-Matching Algorithm for the Control of Assistive Technologies

Assistive technologies (ATs) often have a high-dimensionality of possible movements (e.g., assistive robot with several degrees of freedom or a computer), but the users have to control them with low-dimensionality sensors and interfaces (e.g., switches). This paper presents the development of an open-source interface based on a sequence-matching algorithm for the control of ATs. Sequence matching allows the user to input several different commands with low-dimensionality sensors by not only recognizing their output, but also their sequential pattern through time, similarly to Morse code. In this paper, the algorithm is applied to the recognition of hand gestures, inputted using an inertial measurement unit worn by the user. An SVM-based algorithm, that is aimed to be robust, with small training sets (e.g., five examples per class) is developed to recognize gestures in real-time. Finally, the interface is applied to control a computer’s mouse and keyboard. The interface was compared against (and combined with) the head movement-based AssystMouse software. The hand gesture interface showed encouraging results for this application but could also be used with other body parts (e.g., head and feet) and could control various ATs (e.g., assistive robotic arm and prosthesis).

Communicating Inferred Goals With Passive Augmented Reality and Active Haptic Feedback

Robots learn as they interact with humans. Consider a human teleoperating an assistive robot arm: as the human guides and corrects the arm's motion, the robot gathers information about the human's desired task. But how does the human know what their robot has inferred? Today's approaches often focus on conveying intent: for instance, using legible motions or gestures to indicate what the robot is planning. However, closing the loop on robot inference requires more than just revealing the robot's current policy: the robot should also display the alternatives it thinks are likely, and prompt the human teacher when additional guidance is necessary. In this letter we propose a multimodal approach for communicating robot inference that combines both passive and active feedback. Specifically, we leverage information-rich augmented reality to passively visualize what the robot has inferred, and attention-grabbing haptic wristbands to actively prompt and direct the human's teaching. We apply our system to shared autonomy tasks where the robot must infer the human's goal in real-time. Within this context, we integrate passive and active modalities into a single algorithmic framework that determines when and which type of feedback to provide. Combining both passive and active feedback experimentally outperforms single modality baselines; during an in-person user study, we demonstrate that our integrated approach increases how efficiently humans teach the robot while simultaneously decreasing the amount of time humans spend interacting with the robot. Videos here: https://youtu.be/swq_u4iIP-g

Nose Tracking Assistive Robot Control for the People with Motor Dysfunctions

<h3>Research Objectives</h3> To Implement Human Computer Interface (HCI) with digital image processing for a new approach to control assistive robots. <h3>Design</h3> Experimental Study. <h3>Setting</h3> In UWM BioRobotics Lab. <h3>Participants</h3> Healthy participants. <h3>Interventions</h3> Assistive robot control for daily living activities(Eating, Picking/Placing Objects), Nose tracking cursor control, User interface. <h3>Main Outcome Measures</h3> #Control the computer cursor with nose movements. #App Interface is developed to control the robotic arm. #Control the interface using nose and teeth (showing teeth triggered the click events) to control the assistive robotic arm. <h3>Results</h3> Result contains the demonstration of the control of a robotic arm with nose tracking control through an user interface. External webcam was used to get the real-time video for tracking the nose for cursor movement. We conducted experiments on healthy subjects. The feedbacks are recorded in Tables. We compared the results with other two techniques which are eye tracking and camera mouse for cursor movement and clicking events in terms of error pixels and other factors. <h3>Conclusions</h3> The nose tracking cursor control is mainly for the disabled people who are struggling with or unable to do their basic day to day tasks for their motor dysfunctions. This method will help them to access and control over a computer as well as a robotic assistive arm to ease their lives. <h3>Author(s) Disclosures</h3> No Conflict