Recent Advances on Human-Robot Interface of Wheelchair-Mounted Robotic Arm

A wheelchair-mounted robotic arm (WMRA) was designed and built to meet the needs of mobility-impaired persons, and to exceed the capabilities of current devices of this type. Combining the wheelchair control and the arm control through the augmentation of the Jacobian to include representations of both resulted in a control system that effectively and simultaneously controls both devices at once. The control system was designed for coordinated Cartesian control with singularity robustness and task-optimized combined mobility and manipulation. Weighted Least Norm solution was implemented to prioritize the motion between different arm joints and the wheelchair. Modularity in both the hardware and software levels allowed multiple input devices to be used to control the system, including the Brain-Computer Interface (BCI). The ability to communicate a chosen character from the BCI to the controller of the WMRA was presented, and the user was able to control the motion of WMRA system by focusing attention on a specific character on the screen. Further testing of different types of displays (e.g. commands, picture of objects, and a menu display with objects, tasks and locations) is planned to facilitate communication, mobility and manipulation for people with severe

In this paper, we present a novel vision based interface for selecting an object using a Brain Computer Interface (BCI), and grasping it using a robotic arm mounted to a powered wheelchair. As issuing commands through BCI is slow, this system was designed to allow a user to perform a complete task using the robotic system via the BCI issuing as few commands as possible, without losing concentration on the stimuli or the task. A scene image is captured by a camera mounted on the wheelchair, from which a dynamically sized non-uniform stimulus grid is created using edge information. Dynamically sized grids improve object selection efficiency. Oddball paradigm and P300 event related potentials (ERP) are used to select stimuli, the stimuli being each cell in the grid. Once selected, object segmentation and matching is used to identify the object. Then the user, using BCI, chooses an action to be performed on the object via the wheelchair mounted robotic arm (WMRA). Tests on 6 healthy human subjects validated the functionality of the system. An average accuracy of 85.56% was achieved for stimuli selection over all subjects. With the proposed system, it took the users an average of 5 commands to grasp an object. The system will eventually be useful for completely paralyzed or locked-in patients for performing activities of daily living (ADL) tasks.

People suffering from severe disability like spinal cord injury dysfunctions are unable to control a wheelchair via a common joystick interface. For this target group we have developed a robotic wheelchair able to perform autonomous navigation and operate on different driving modes depending on the disability type of its user. Moreover, a 7-DoF wheelchair - mounted robotic arm (WMRA) was added to the 2-DoF nonholonomic wheelchair platform. In this paper, an image - based visual servoing (IBVS) approach is described with a Speeded Up Robust local Features detection (SURF) using eye-in-hand and eye-to-hand camera configuration for combined control of mobility and manipulation of the 9-DoF WMRA system to execute activities of daily living (ADL) autonomously. Selecting a control mode is done either by voice or by the touchscreen. Experiments with human users highlight advantages of augmentation in wheelchairs.

Wheelchair-mounted robotic arms support people with upper extremity disabilities with various activities of daily living (ADL). However, the associated cost and the power consumption of responsive and adaptive assistive robotic arms contribute to the fact that such systems are in limited use. Neuromorphic spiking neural networks can be used for a real-time machine learning-driven control of robots, providing an energy efficient framework for adaptive control. In this work, we demonstrate a neuromorphic adaptive control of a wheelchair-mounted robotic arm deployed on Intel’s Loihi chip. Our algorithm design uses neuromorphically represented and integrated velocity readings to derive the arm’s current state. The proposed controller provides the robotic arm with adaptive signals, guiding its motion while accounting for kinematic changes in real-time. We pilot-tested the device with an able-bodied participant to evaluate its accuracy while performing ADL-related trajectories. We further demonstrated the capacity of the controller to compensate for unexpected inertia-generating payloads using online learning. Videotaped recordings of ADL tasks performed by the robot were viewed by caregivers; data summarizing their feedback on the user experience and the potential benefit of the system is reported.

A wheelchair-mounted robotic arm (WMRA) system was designed and built to meet the needs of mobilityimpaired persons with limitations of upper extremities, and to exceed the capabilities of current devices of this type. The control of this 9-degree-of-freedom system expands upon conventional control methods and combines the 7-DoF robotic arm control with the 2-degree-of-freedom power wheelchair control. The 3- degrees of redundancy are optimized to effectively perform activities of daily living and overcome singularities, joint limits and some workspace limitations. The control system is designed for teleoperated or autonomous coordinated Cartesian control, which offers expandability for future research. A P300 Brain Computer Interface (BCI), the BCI2000, was implemented to control the WMRA system. The control is done by recording and analysing the brain activity through an electrode cap while providing visual stimulation to the user via a visual matrix. The visual matrix contains a symbolic or an alphabetic array corresponding to the motion of the WMRA. By recognizing online and in real-time, which element in the matrix elicited a P300, the BCI system can identify which element the user chose to communicate. The chosen element is then communicated to the controller of the WMRA system. The speed and accuracy of the BCI system was tested. This paper gives details of the WMRA's integration with the BCI2000 and documents the experimental results of the BCI and the WMRA in simulation.

In this paper, we describe an empirical study with healthy subjects with simulated ADL tasks using UCF-ARM - a 6-DOF assistive robot that is visually guided through a calibrated stereo camera system fitted in the gripper of Exact Dynamics' Manus ARM. The goal of the research is to reduce time to task completion and cognitive burden for users interacting with an unstructured environment via a Wheelchair Mounted Robotic Arm (WMRA). Our WMRA, UCF-ARM, provides access to a multimodal user-customizable human computer interface and accomplishes visual servoing through an in-hand stereo rig as well as adaptive grip force application through force sensing resistors embedded in the fingers of the gripper. Choice of user interface is dependent on the user's functional level and injury. Two level object grasping tasks are used to assess the time to task completion and cognitive burden for users evaluating the system under different tasks, control modes, and interface modalities. Results of the statistical analysis are provided to compare the advantages and disadvantages of the evaluated options.

In a household setting, a wheelchair-mounted robotic arm (WMRA) can be useful for assisting elderly and disabled individuals. However, the current WMRA can only perform movement and grasping tasks through joystick remote control. This method results in low efficiency due to poor coordination between the mobile platform and the robotic arm as well as the numerous operational steps required. To improve the efficiency and success rate of the robot in task execution, this paper proposes a parking location optimization method that combines the occupied grid map (OGM) and the inverse reachability map (IRM). Firstly, the SLAM algorithm is used to collect environment information, which is then stored in the form of an occupied grid map. The robotic arm workspace is then gridded, and the inverse reachability map is calculated based on the grasping pose of the target object. Finally, the optimal position of the mobile platform is obtained by comparing the optimal location point in the inverse reachability map and the obstacle information in the occupied grid map. This process achieves base placement optimization based on the grasping pose. The experimental results demonstrate that this method reduces the user operation time by 97.31% and overall task completion time by 40.57% when executing household environment tasks compared with the joystick control, increasing the range of executable tasks compared with the algorithm of the EL-E robot and reducing task completion time by 23.48% for the same task. This paper presents a parking location optimization method that can improve the grasping efficiency of the robotic arm and achieve parking location position selection for the WMRA in a household environment.

In order to let the wheelchair mounted robotic arm (WMRA) be more intelligent and adaptable, aiming to offer simple and convenient assistance for the elders and disabilities to cope with the complex tasks in the daily life, we present a learning method based on robot learning from demonstration to solve the problem. This method adopts the Beta Process Autoregressive Hidden Markov Model to segment the demonstrations of related task, acquire the contained skills and recognize the repeated skills. After that, it uses the Dynamic Movement Primitives to adjust the related skill according to the given goal position, so as to replay the demonstrated task in a new environment. This learning framework was validated on a six-degree-of-freedom JACO robotic arm, performing the task of drinking water from the bottle through a straw.

Majority of wheelchair users with neuromuscular diseases experience upperbody muscular weakness, which limit their ability to perform common activities of daily living. A Wheelchair Mounted Robotic Arm (WMRA) can assist these individuals to perform various tasks independently. This research paper is focused on designing a control system and interfacing mechanism to a newly designed WMRA. The proposed interfacing mechanism is based on virtual image and voice activated signals. The paper outlined the implementation through three dimensional imagining and a virtual workspace model of the real world for the arm to navigate through. The user speaks a simple command through a microphone and then be run through a control algorithm to drive the robotic arm. Inverse dynamics based control scheme is implemented to command the active joints of the robot, based on an input from the virtual CAD environment. The preliminary simulation results show that the virtual model joint trajectories are well tracked with minimal error. The paper also outlined the overall implementation of the system using Robot Operating System based interfacing system.

Article 14: Effect of a wheelchair-mounted robotic arm on functional outcomes in persons with spinal cord injury

We describe the progress in implementing a vision based robotic assist device to facilitate Activities of Daily Living (ADL) tasks for a class of users with motor disabilities. The goal of the research is to reduce time to task completion and cognitive burden for users interacting with an unstructured environment via a Wheelchair Mounted Robotic Arm (WMRA). A developed robot system is tested with five healthy subjects to assess its usefulness.

A new and light wheelchair-mounted robotic arm (WMRA-II) was designed and built to meet the needs of mobility-impaired persons with limitations of upper extremities, and to exceed the capabilities of current devices of this type. The structure of the new robotic arm utilizes carbon-fiber and polycarbonate tubes to reduce the arm weight. The mechanical design incorporates DC servo drive, with actuator hardware at each individual joint, allowing reconfigurable link lengths. It has seven degrees of freedom and uses a side mount on a power wheelchair. A single control board that is capable of controlling eight motors was used for coordinated Cartesian control of the robotic arm. This paper discusses the current state of the art in WMRAs; describes the design goals and user requirements for this device; explains the component selection process; discusses details of the mechanical design and the controller design; and describes the testing of the completed arm. Further improvements are also suggested.

Picking up an object with a wheelchair mounted robotic arm can be decomposed into a wheelchair navigation task designed to position the robotic arm such that the object is “easy to reach”, and the actual grasp performed by the robotic arm. A convenient definition of the notion of ease of reach can be given by creating a score (ERS) that relies on the number of distinct ways the object can be picked up from a given location. Unfortunately, the accurate calculation of ERS must rely on repeating the path planning process for every candidate position and grasp type, in the presence of obstacles. In this paper we use the bootstrap aggregation over hand-crafted, domain specific features to learn a model for the estimation of ERS. In a simulation study, we show that the estimated ERS closely matches the actual value and the speed of estimation is fast enough for real-time operation, even in the presence of a large number of obstacles in the scene.

Throughout the last decade, many assistive robots for people with disabilities have been developed; however, researchers have not fully utilized these robotic technologies to entirely create independent living conditions for people with disabilities, particularly in relation to activities of daily living (ADLs). An assistive system can help satisfy the demands of regular ADLs for people with disabilities. With an increasing shortage of caregivers and a growing number of individuals with impairments and the elderly, assistive robots can help meet future healthcare demands. One of the critical aspects of designing these assistive devices is to improve functional independence while providing an excellent human–machine interface. People with limited upper limb function due to stroke, spinal cord injury, cerebral palsy, amyotrophic lateral sclerosis, and other conditions find the controls of assistive devices such as power wheelchairs difficult to use. Thus, the objective of this research was to design a multimodal control method for robotic self-assistance that could assist individuals with disabilities in performing self-care tasks on a daily basis. In this research, a control framework for two interchangeable operating modes with a finger joystick and a chin joystick is developed where joysticks seamlessly control a wheelchair and a wheelchair-mounted robotic arm. Custom circuitry was developed to complete the control architecture. A user study was conducted to test the robotic system. Ten healthy individuals agreed to perform three tasks using both (chin and finger) joysticks for a total of six tasks with 10 repetitions each. The control method has been tested rigorously, maneuvering the robot at different velocities and under varying payload (1–3.5 lb) conditions. The absolute position accuracy was experimentally found to be approximately 5 mm. The round-trip delay we observed between the commands while controlling the xArm was 4 ms. Tests performed showed that the proposed control system allowed individuals to perform some ADLs such as picking up and placing items with a completion time of less than 1 min for each task and 100% success.

Majority of wheelchair users experience upper-body muscular weakness, resulting from neuromuscular diseases, which limit their ability to perform common activities of daily living. A Wheelchair Mounted Robotic Arm (WMRA) will assist these individuals to eat, drink, and move objects as needed. This paper presents the design of a new WMRA as well as the analysis of its function. The design is side-mounted onto either a normal or power wheelchair, and incorporates a slim profile to allow ease of passage through doorways and be otherwise unobtrusive. The arm is easily removable, with assistance, for storage or travel. The mechanical design utilizes a belt and pulley system for remote actuation of each joint, driven by DC Gearmotors located in the base of the arm. This helps to shift the weight closer to the wheelchair and to maintain the required speed, torque and inertia while actively driving each joint of the robot. The end-effector is a unique design, intended to have the adaptability to securely lift a large variety of objects. Grasping simulations were performed on several standard objects which might be encountered daily. Structural, kinematic and workspace analyses are conducted, and results confirm that the designed WMRA is rated to lift a 4 kg payload, while also having a reach of 1.3 meters long radius.