Human-in-the-loop control of an assistive robotic arm in unstructured environments for spinal cord injured users

A wheelchair-mounted robotic arm (WMRA) was designed and built to meet the needs of mobility-impaired persons, and to exceed the capabilities of current devices of this type. Combining the wheelchair control and the arm control through the augmentation of the Jacobian to include representations of both resulted in a control system that effectively and simultaneously controls both devices at once. The control system was designed for coordinated Cartesian control with singularity robustness and task-optimized combined mobility and manipulation. Weighted Least Norm solution was implemented to prioritize the motion between different arm joints and the wheelchair. Modularity in both the hardware and software levels allowed multiple input devices to be used to control the system, including the Brain-Computer Interface (BCI). The ability to communicate a chosen character from the BCI to the controller of the WMRA was presented, and the user was able to control the motion of WMRA system by focusing attention on a specific character on the screen. Further testing of different types of displays (e.g. commands, picture of objects, and a menu display with objects, tasks and locations) is planned to facilitate communication, mobility and manipulation for people with severe

Wheelchair-mounted robotic arms support people with upper extremity disabilities with various activities of daily living (ADL). However, the associated cost and the power consumption of responsive and adaptive assistive robotic arms contribute to the fact that such systems are in limited use. Neuromorphic spiking neural networks can be used for a real-time machine learning-driven control of robots, providing an energy efficient framework for adaptive control. In this work, we demonstrate a neuromorphic adaptive control of a wheelchair-mounted robotic arm deployed on Intel’s Loihi chip. Our algorithm design uses neuromorphically represented and integrated velocity readings to derive the arm’s current state. The proposed controller provides the robotic arm with adaptive signals, guiding its motion while accounting for kinematic changes in real-time. We pilot-tested the device with an able-bodied participant to evaluate its accuracy while performing ADL-related trajectories. We further demonstrated the capacity of the controller to compensate for unexpected inertia-generating payloads using online learning. Videotaped recordings of ADL tasks performed by the robot were viewed by caregivers; data summarizing their feedback on the user experience and the potential benefit of the system is reported.

In this paper, we present a novel vision based interface for selecting an object using a Brain Computer Interface (BCI), and grasping it using a robotic arm mounted to a powered wheelchair. As issuing commands through BCI is slow, this system was designed to allow a user to perform a complete task using the robotic system via the BCI issuing as few commands as possible, without losing concentration on the stimuli or the task. A scene image is captured by a camera mounted on the wheelchair, from which a dynamically sized non-uniform stimulus grid is created using edge information. Dynamically sized grids improve object selection efficiency. Oddball paradigm and P300 event related potentials (ERP) are used to select stimuli, the stimuli being each cell in the grid. Once selected, object segmentation and matching is used to identify the object. Then the user, using BCI, chooses an action to be performed on the object via the wheelchair mounted robotic arm (WMRA). Tests on 6 healthy human subjects validated the functionality of the system. An average accuracy of 85.56% was achieved for stimuli selection over all subjects. With the proposed system, it took the users an average of 5 commands to grasp an object. The system will eventually be useful for completely paralyzed or locked-in patients for performing activities of daily living (ADL) tasks.

People suffering from severe disability like spinal cord injury dysfunctions are unable to control a wheelchair via a common joystick interface. For this target group we have developed a robotic wheelchair able to perform autonomous navigation and operate on different driving modes depending on the disability type of its user. Moreover, a 7-DoF wheelchair - mounted robotic arm (WMRA) was added to the 2-DoF nonholonomic wheelchair platform. In this paper, an image - based visual servoing (IBVS) approach is described with a Speeded Up Robust local Features detection (SURF) using eye-in-hand and eye-to-hand camera configuration for combined control of mobility and manipulation of the 9-DoF WMRA system to execute activities of daily living (ADL) autonomously. Selecting a control mode is done either by voice or by the touchscreen. Experiments with human users highlight advantages of augmentation in wheelchairs.

The redundant kinematic structure of the human upper body provides sufficient dexterity in performing daily living activities, such as drinking, eating, and other manipulation tasks. Most of these Activities of Daily Living (ADL) tasks can be performed using various trajectories; however, healthy humans employ some possible trajectories to accomplish these tasks. Assistive robots can explore these Cartesian trajectories and joint angles’ trajectories from human demonstrations and utilize artificial intelligence techniques to recognize, learn, and perform ADLs using robotic arms. This paper aims to develop a platform for Assistive Robots to identify and perform ADLs through human demonstration. In this paper, two Mask Regional Convolutional Neural Networks (Mask R-CNN) were developed to predict human intention for a given task, and the Reinforcement Learning approach was implemented for optimum trajectory planning. The First Mask R-CNN network, which was used for the detection of an object, was trained on a novel dataset. This dataset comprises images captured by the research team as well as gathered images from the internet. The first Mask R-CNN is expected to achieve two main objectives. The first objective is related to structure learning, which detects an object in a cluttered environment. The second objective is to estimate parameters that maximize the weight and detection rate of diverse types of similar objects. The first Mask R-CNN tries to detect the object with its location in the image and provides a soft mask on it. The second Mask R-CNN, which tries to detect human posture, was trained on a pre-existing dataset comprising human posture images. The output of these two Mask R-CNN’s then feed-forward to recognize human intensions that can be achieved through human-pose key points. Prediction of human intention was performed if the detected object and human pose match a range of pre-stored values specific to ADLs. Once the human intention has been identified via Mask R-CNN networks, the system starts planning for an optimal trajectory. For the trajectory planning, the Q-learning approach was implemented, which uses the concept of reward and penalty while exploring the unstructured environment in order to find an optimal trajectory. Experiments were conducted to determine the level of accuracy and reliability provided by our assistive platform in predicting human actions and performing trajectory planning. Results show that our assistive platform successfully recognized human actions such as water pouring, cereal making, and opening the door, as well as finding optimized trajectories to perform those actions.

In this paper, we describe an empirical study with healthy subjects with simulated ADL tasks using UCF-ARM - a 6-DOF assistive robot that is visually guided through a calibrated stereo camera system fitted in the gripper of Exact Dynamics' Manus ARM. The goal of the research is to reduce time to task completion and cognitive burden for users interacting with an unstructured environment via a Wheelchair Mounted Robotic Arm (WMRA). Our WMRA, UCF-ARM, provides access to a multimodal user-customizable human computer interface and accomplishes visual servoing through an in-hand stereo rig as well as adaptive grip force application through force sensing resistors embedded in the fingers of the gripper. Choice of user interface is dependent on the user's functional level and injury. Two level object grasping tasks are used to assess the time to task completion and cognitive burden for users evaluating the system under different tasks, control modes, and interface modalities. Results of the statistical analysis are provided to compare the advantages and disadvantages of the evaluated options.

Background: Wheelchair mounted robotic arm is a typical assistive robot, which is widely used to help the elders and the disabled to complete the activities of daily life. But limited by the restrictions of the users’ athletic ability and cognitive ability, how to flexibly manipulate such robot is still a problem in front of them. The human-computer interaction technology is the core technology of the robot. Its performance directly affects the user's acceptability, satisfaction and promotion of intelligent wheelchairs. Objective: The study aims to give a general summary of recent human-robot interface of wheelchair mounted robotic arm and introduce their respective characteristics. Methods: Based on various patents and research developments about the human-robot interface of the assistive robot at home and abroad, this paper puts forward the basic principle of designing the humanrobot interaction mode, divides the man-robot interface into two categories based on the perspective of user control robot arm, and describes in detail, the typical human-robot interface and its related characteristics contained in each classification. Results: The development trends of the human-robot interface in future are predicted, so as to provide some research reference for the related scientific researchers. Conclusion: Wheelchair mounted robotic arm has important practical significance. Further improvements are needed in the design of the human-robot interface. It can effectively improve the operation performance of the WMRA, and take full advantage of the user’s existing movement ability to meet the requirement of dominating the control process. Furthermore, these improvements in the human-robot interface will allow more and more users to accept the WMRA, manipulate the WMRA, and enjoy improvements in the quality of their life for these assistive robots.

A wheelchair-mounted robotic arm (WMRA) system was designed and built to meet the needs of mobility-impaired persons with limitations of upper extremities, and to exceed the capabilities of current devices of this type. The control of this 9-DoF system expands on the conventional control methods and combines the 7-DoF robotic arm control with the 2-DoF power wheelchair control. The 3-degrees of redundancy are optimized to effectively perform activities of daily living (ADLs) and overcome singularities, joint limits and some workspace limitations. The control system is designed for teleoperated or autonomous coordinated Cartesian control, and it offers expandability for future research, such as voice or sip and puff control operations and sensor assist functions.

A wheelchair-mounted robotic arm (WMRA) system was designed and built to meet the needs of mobility-impaired persons with limitations of upper extremities, and to exceed the capabilities of current devices of this type. The control of this 9-DoF system expands on the conventional control methods and combines the 7-DoF robotic arm control with the 2-DoF power wheelchair control. The 3-degrees of redundancy are optimized to effectively perform activities of daily living (ADLs) and overcome singularities, joint limits and some workspace limitations. The control system is designed for teleoperated or autonomous coordinated Cartesian control, and it offers expandability for future research, such as voice or sip and puff control operations and sensor assist functions.

A wheelchair-mounted robotic arm (WMRA) system was designed and built to meet the needs of mobility-impaired persons with limitations of upper extremities, and to exceed the capabilities of current devices of this type. The control of this 9-DoF system expands on the conventional control methods and combines the 7-DoF robotic arm control with the 2-DoF power wheelchair control. The 3-degrees of redundancy are optimized to effectively perform activities of daily living (ADLs) and overcome singularities, joint limits and some workspace limitations. The control system is designed for teleoperated or autonomous coordinated Cartesian control, and it offers expandability for future research. Different user-interface devices, including a Brain Computer Interface (BCI) were integrated to the control of the WMRA system. Testing and data collection were performed on human subjects. Various optimized control methods and test results are presented in this paper.

A wheelchair-mounted robotic arm (WMRA) system was designed and built to meet the needs of mobility-impaired persons with limitations of upper extremities, and to exceed the capabilities of current devices of this type. The control of this 9-DoF system combines the 7-DoF robotic arm control with the 2-DoF power wheelchair control. The 3-degrees of redundancy can be optimized to effectively perform activities of daily living (ADLs) and overcome some workspace limitations. The control system is designed for teleoperated or autonomous coordinated Cartesian control, and it offers expandability for future research, such as voice or sip and puff control operations and sensor assist functions

A wheelchair-mounted robotic arm (WMRA) system was designed and built to meet the needs of mobility-impaired persons with limitations of upper extremities, and to exceed the capabilities of current devices of this type. The control of this 9-DoF system expands on the conventional control methods and combines the 7-DoF robotic arm control with the 2-DoF power wheelchair control. The 3-degrees of redundancy are optimized to effectively perform activities of daily living (ADLs) and overcome singularities, joint limits and some workspace limitations. The control system is designed for teleoperated or autonomous coordinated Cartesian control, and it offers expandability for future research, such as voice or sip and puff control operations and sensor assist functions.

There has been significant progress in bringing commercially-viable wheelchair mounted robotic arms (WMRA) into the marketplace in the past 30 years. This paper focuses on kinematic analysis and evaluation of such robotic arms. It addresses the kinematics of the WMRA with respect to its ability to reach common positions while performing activities of daily living (ADL). A procedure is developed for the kinematic analysis and evaluation of a wheelchair mounted robotic arm. In addition to developing the analytical procedure, the manipulator is evaluated, and design recommendations and insights are obtained. Current commercially-available wheelchair mountable robotic manipulators have been designed specifically for use in rehabilitation robotics. In an effort to evaluate two commercial manipulators, the procedure for kinematic analysis is applied to each manipulator. Design recommendations with regard to each device are obtained. This method will benefit the researchers by providing a standardized procedure for kinematic analysis of WMRAs that is capable of evaluating independent designs.

In order to let the wheelchair mounted robotic arm (WMRA) be more intelligent and adaptable, aiming to offer simple and convenient assistance for the elders and disabilities to cope with the complex tasks in the daily life, we present a learning method based on robot learning from demonstration to solve the problem. This method adopts the Beta Process Autoregressive Hidden Markov Model to segment the demonstrations of related task, acquire the contained skills and recognize the repeated skills. After that, it uses the Dynamic Movement Primitives to adjust the related skill according to the given goal position, so as to replay the demonstrated task in a new environment. This learning framework was validated on a six-degree-of-freedom JACO robotic arm, performing the task of drinking water from the bottle through a straw.

Brain-computer interface (BCI) provides a novel technology for patients and healthy human subjects to control a robotic arm. Currently, BCI control of a robotic arm to complete the reaching and grasping tasks in an unstructured environment is still challenging because the current BCI technology does not meet the requirement of manipulating a multi-degree robotic arm accurately and robustly. BCI based on steady-state visual evoked potential (SSVEP) could output a high information transfer rate; however, the conventional SSVEP paradigm failed to control a robotic arm to move continuously and accurately because the users have to switch their gaze between the flickering stimuli and the target frequently. This study proposed a novel SSVEP paradigm in which the flickering stimuli were attached to the robotic arm's gripper and moved with it. First, an offline experiment was designed to investigate the effects of moving flickering stimuli on the SSVEP's responses and decoding accuracy. After that, contrast experiments were conducted, and twelve subjects were recruited to participate in a robotic arm control experiment using both the paradigm one (P1, with moving flickering stimuli) and the paradigm two (P2, conventional fixed flickering stimuli) using a block randomization design to balance their sequences. Double blinks were used to trigger the grasping action asynchronously whenever the subjects were confident that the position of the robotic arm's gripper was accurate enough. Experimental results showed that the paradigm P1 with moving flickering stimuli provided a much better control performance than the conventional paradigm P2 in completing a reaching and grasping task in an unstructured environment. Subjects' subjective feedback scored by a NASA-TLX mental workload scale also corroborated the BCI control performance. The results of this study suggest that the proposed control interface based on SSVEP BCI provides a better solution for robotic arm control to complete the accurate reaching and grasping tasks.